Started by user Ritesh Agarwal [EnvInject] - Loading node environment variables. Building remotely on magma-sc1403-32g-24c (magma_slave) in workspace /data/workspace/magma_temp_job4 [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Wiping out workspace first. Cloning the remote Git repository Cloning repository https://github.com/couchbaselabs/TAF.git > /usr/bin/git init /data/workspace/magma_temp_job4 # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 015817b28390742d4cc5dd5624e630175c8c7202 (origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 015817b28390742d4cc5dd5624e630175c8c7202 > /usr/bin/git rev-list 015817b28390742d4cc5dd5624e630175c8c7202 # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #1296 jenkins-magma_temp_job4-1296 # timeout=10 No emails were triggered. [magma_temp_job4] $ /bin/bash /tmp/jenkins7656801766802325205.sh ############################################## Installing dependencies from requirements.txt ############################################## Requirement already satisfied (use --upgrade to upgrade): futures==3.3.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): requests==2.24.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): urllib3==1.25.10 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): ruamel.yaml==0.16.12 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 4)) Requirement already satisfied (use --upgrade to upgrade): six==1.15.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 5)) Cleaning up... #################### Done #################### [magma_temp_job4] $ /bin/sh -xe /tmp/jenkins4447722698933062379.sh + echo version=7.1.0-2255 version=7.1.0-2255 + mkdir tr_for_install + cd tr_for_install + git clone https://github.com/couchbase/testrunner.git Cloning into 'testrunner'... + cd testrunner + git checkout master Already on 'master' + git pull origin master From https://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. + py_executable=python3 + python3 scripts/new_install.py -i /tmp/magma_temp_job4.ini -p timeout=1200,get-cbcollect-info=True,version=7.1.0-2255,product=cb,debug_logs=False,url=,install_debug_info=true 2022-02-07 21:06:23,433 - root - WARNING - URL: is not valid, will use version to locate build 2022-02-07 21:06:23,434 - root - INFO - SSH Connecting to 172.23.120.170 with username:root, attempt#1 of 5 2022-02-07 21:06:23,551 - root - INFO - SSH Connected to 172.23.120.170 as root 2022-02-07 21:06:23,886 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:24,186 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:24,188 - root - INFO - SSH Connecting to 172.23.121.115 with username:root, attempt#1 of 5 2022-02-07 21:06:24,292 - root - INFO - SSH Connected to 172.23.121.115 as root 2022-02-07 21:06:24,580 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:24,877 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:24,879 - root - INFO - SSH Connecting to 172.23.121.116 with username:root, attempt#1 of 5 2022-02-07 21:06:24,982 - root - INFO - SSH Connected to 172.23.121.116 as root 2022-02-07 21:06:25,271 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:25,570 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:25,572 - root - INFO - SSH Connecting to 172.23.121.123 with username:root, attempt#1 of 5 2022-02-07 21:06:25,674 - root - INFO - SSH Connected to 172.23.121.123 as root 2022-02-07 21:06:25,986 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:26,287 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:26,289 - root - INFO - SSH Connecting to 172.23.121.124 with username:root, attempt#1 of 5 2022-02-07 21:06:26,393 - root - INFO - SSH Connected to 172.23.121.124 as root 2022-02-07 21:06:26,682 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:26,992 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:26,994 - root - INFO - SSH Connecting to 172.23.121.126 with username:root, attempt#1 of 5 2022-02-07 21:06:27,107 - root - INFO - SSH Connected to 172.23.121.126 as root 2022-02-07 21:06:27,423 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:27,743 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:27,745 - root - INFO - SSH Connecting to 172.23.121.127 with username:root, attempt#1 of 5 2022-02-07 21:06:27,847 - root - INFO - SSH Connected to 172.23.121.127 as root 2022-02-07 21:06:28,119 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:28,414 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:28,415 - root - INFO - SSH Connecting to 172.23.121.128 with username:root, attempt#1 of 5 2022-02-07 21:06:28,738 - root - INFO - SSH Connected to 172.23.121.128 as root 2022-02-07 21:06:29,037 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:29,345 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:29,348 - root - INFO - SSH Connecting to 172.23.121.129 with username:root, attempt#1 of 5 2022-02-07 21:06:29,452 - root - INFO - SSH Connected to 172.23.121.129 as root 2022-02-07 21:06:29,744 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:30,045 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:30,048 - root - INFO - SSH Connecting to 172.23.121.130 with username:root, attempt#1 of 5 2022-02-07 21:06:30,152 - root - INFO - SSH Connected to 172.23.121.130 as root 2022-02-07 21:06:30,442 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:30,738 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:30,740 - root - INFO - SSH Connecting to 172.23.121.131 with username:root, attempt#1 of 5 2022-02-07 21:06:30,844 - root - INFO - SSH Connected to 172.23.121.131 as root 2022-02-07 21:06:31,132 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:31,427 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:31,429 - root - INFO - SSH Connecting to 172.23.121.132 with username:root, attempt#1 of 5 2022-02-07 21:06:31,531 - root - INFO - SSH Connected to 172.23.121.132 as root 2022-02-07 21:06:31,822 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:32,119 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:32,120 - root - INFO - SSH Connecting to 172.23.121.133 with username:root, attempt#1 of 5 2022-02-07 21:06:32,222 - root - INFO - SSH Connected to 172.23.121.133 as root 2022-02-07 21:06:32,505 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:32,797 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:32,799 - root - INFO - SSH Connecting to 172.23.121.134 with username:root, attempt#1 of 5 2022-02-07 21:06:32,902 - root - INFO - SSH Connected to 172.23.121.134 as root 2022-02-07 21:06:33,184 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:33,491 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:33,493 - root - INFO - SSH Connecting to 172.23.121.135 with username:root, attempt#1 of 5 2022-02-07 21:06:33,600 - root - INFO - SSH Connected to 172.23.121.135 as root 2022-02-07 21:06:33,883 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:34,178 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:34,180 - root - INFO - SSH Connecting to 172.23.121.136 with username:root, attempt#1 of 5 2022-02-07 21:06:34,285 - root - INFO - SSH Connected to 172.23.121.136 as root 2022-02-07 21:06:34,570 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:34,864 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:34,866 - root - INFO - SSH Connecting to 172.23.120.170 with username:root, attempt#1 of 5 2022-02-07 21:06:34,965 - root - INFO - SSH Connected to 172.23.120.170 as root 2022-02-07 21:06:35,250 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:35,544 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:35,545 - root - INFO - SSH Connecting to 172.23.121.115 with username:root, attempt#1 of 5 2022-02-07 21:06:35,645 - root - INFO - SSH Connected to 172.23.121.115 as root 2022-02-07 21:06:35,924 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:36,221 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:36,223 - root - INFO - SSH Connecting to 172.23.121.116 with username:root, attempt#1 of 5 2022-02-07 21:06:36,323 - root - INFO - SSH Connected to 172.23.121.116 as root 2022-02-07 21:06:36,597 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:36,893 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:36,895 - root - INFO - SSH Connecting to 172.23.121.123 with username:root, attempt#1 of 5 2022-02-07 21:06:36,997 - root - INFO - SSH Connected to 172.23.121.123 as root 2022-02-07 21:06:37,279 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:37,575 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:37,576 - root - INFO - SSH Connecting to 172.23.121.124 with username:root, attempt#1 of 5 2022-02-07 21:06:37,676 - root - INFO - SSH Connected to 172.23.121.124 as root 2022-02-07 21:06:37,964 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:38,272 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:38,274 - root - INFO - SSH Connecting to 172.23.121.126 with username:root, attempt#1 of 5 2022-02-07 21:06:38,374 - root - INFO - SSH Connected to 172.23.121.126 as root 2022-02-07 21:06:38,653 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:38,952 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:38,954 - root - INFO - SSH Connecting to 172.23.121.127 with username:root, attempt#1 of 5 2022-02-07 21:06:39,054 - root - INFO - SSH Connected to 172.23.121.127 as root 2022-02-07 21:06:39,321 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:39,621 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:39,624 - root - INFO - SSH Connecting to 172.23.121.128 with username:root, attempt#1 of 5 2022-02-07 21:06:39,722 - root - INFO - SSH Connected to 172.23.121.128 as root 2022-02-07 21:06:39,983 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:40,277 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:40,279 - root - INFO - SSH Connecting to 172.23.121.129 with username:root, attempt#1 of 5 2022-02-07 21:06:40,380 - root - INFO - SSH Connected to 172.23.121.129 as root 2022-02-07 21:06:40,651 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:40,945 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:40,947 - root - INFO - SSH Connecting to 172.23.121.130 with username:root, attempt#1 of 5 2022-02-07 21:06:41,049 - root - INFO - SSH Connected to 172.23.121.130 as root 2022-02-07 21:06:41,321 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:41,616 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:41,618 - root - INFO - SSH Connecting to 172.23.121.131 with username:root, attempt#1 of 5 2022-02-07 21:06:41,718 - root - INFO - SSH Connected to 172.23.121.131 as root 2022-02-07 21:06:41,994 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:42,292 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:42,293 - root - INFO - SSH Connecting to 172.23.121.132 with username:root, attempt#1 of 5 2022-02-07 21:06:42,392 - root - INFO - SSH Connected to 172.23.121.132 as root 2022-02-07 21:06:42,668 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:42,958 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:42,960 - root - INFO - SSH Connecting to 172.23.121.133 with username:root, attempt#1 of 5 2022-02-07 21:06:43,061 - root - INFO - SSH Connected to 172.23.121.133 as root 2022-02-07 21:06:43,339 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:43,636 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:43,637 - root - INFO - SSH Connecting to 172.23.121.134 with username:root, attempt#1 of 5 2022-02-07 21:06:43,735 - root - INFO - SSH Connected to 172.23.121.134 as root 2022-02-07 21:06:44,011 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:44,307 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:44,310 - root - INFO - SSH Connecting to 172.23.121.135 with username:root, attempt#1 of 5 2022-02-07 21:06:44,410 - root - INFO - SSH Connected to 172.23.121.135 as root 2022-02-07 21:06:44,685 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:44,979 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:44,981 - root - INFO - SSH Connecting to 172.23.121.136 with username:root, attempt#1 of 5 2022-02-07 21:06:45,083 - root - INFO - SSH Connected to 172.23.121.136 as root 2022-02-07 21:06:45,371 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:06:45,671 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-07 21:06:45,672 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,676 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,676 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,694 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,694 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,695 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,695 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,697 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,697 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,698 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,698 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,699 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,699 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,701 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,701 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,702 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,702 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,703 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,703 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,704 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,704 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,706 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,706 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,707 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,707 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,708 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,708 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,709 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,709 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,710 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,710 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,712 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,712 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,713 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,713 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,714 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,714 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,715 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,715 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,716 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,717 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,718 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,718 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,719 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,719 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,720 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,720 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,721 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,721 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,722 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,722 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,723 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,724 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,725 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,725 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,726 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,726 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,727 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,727 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,728 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,728 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,729 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:45,729 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:06:45,731 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2255/couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm is live 2022-02-07 21:06:55,629 - root - INFO - Done with uninstall on 172.23.121.134. 2022-02-07 21:06:55,638 - root - INFO - Done with uninstall on 172.23.121.132. 2022-02-07 21:06:59,113 - root - INFO - Done with uninstall on 172.23.121.131. 2022-02-07 21:07:01,587 - root - INFO - Done with uninstall on 172.23.121.130. 2022-02-07 21:07:04,023 - root - INFO - Done with uninstall on 172.23.121.133. 2022-02-07 21:07:58,525 - root - INFO - Done with uninstall on 172.23.121.136. 2022-02-07 21:08:02,894 - root - INFO - Done with uninstall on 172.23.121.116. 2022-02-07 21:08:04,369 - root - INFO - Done with uninstall on 172.23.121.135. 2022-02-07 21:08:43,963 - root - INFO - Done with uninstall on 172.23.121.124. 2022-02-07 21:08:52,283 - root - INFO - Done with uninstall on 172.23.121.126. 2022-02-07 21:08:52,511 - root - INFO - Done with uninstall on 172.23.121.127. 2022-02-07 21:08:56,087 - root - INFO - Done with uninstall on 172.23.121.123. 2022-02-07 21:09:02,395 - root - INFO - Done with uninstall on 172.23.121.128. 2022-02-07 21:09:12,358 - root - INFO - Done with uninstall on 172.23.121.115. 2022-02-07 21:09:14,816 - root - INFO - Done with uninstall on 172.23.121.129. 2022-02-07 21:09:18,096 - root - INFO - Done with uninstall on 172.23.120.170. 2022-02-07 21:09:21,850 - root - INFO - running command.raw on 172.23.120.170: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:21,881 - root - INFO - command executed successfully with root 2022-02-07 21:09:23,442 - root - INFO - running command.raw on 172.23.120.170: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:23,457 - root - INFO - command executed successfully with root 2022-02-07 21:09:24,626 - root - INFO - running command.raw on 172.23.121.115: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:24,648 - root - INFO - command executed successfully with root 2022-02-07 21:09:25,734 - root - INFO - running command.raw on 172.23.121.115: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:25,748 - root - INFO - command executed successfully with root 2022-02-07 21:09:26,879 - root - INFO - running command.raw on 172.23.121.116: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:26,898 - root - INFO - command executed successfully with root 2022-02-07 21:09:27,920 - root - INFO - running command.raw on 172.23.121.116: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:27,935 - root - INFO - command executed successfully with root 2022-02-07 21:09:28,973 - root - INFO - running command.raw on 172.23.121.123: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:28,998 - root - INFO - command executed successfully with root 2022-02-07 21:09:30,144 - root - INFO - running command.raw on 172.23.121.123: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:30,157 - root - INFO - command executed successfully with root 2022-02-07 21:09:31,277 - root - INFO - running command.raw on 172.23.121.124: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:31,293 - root - INFO - command executed successfully with root 2022-02-07 21:09:32,361 - root - INFO - running command.raw on 172.23.121.124: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:32,376 - root - INFO - command executed successfully with root 2022-02-07 21:09:33,491 - root - INFO - running command.raw on 172.23.121.126: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:33,505 - root - INFO - command executed successfully with root 2022-02-07 21:09:34,689 - root - INFO - running command.raw on 172.23.121.126: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:34,703 - root - INFO - command executed successfully with root 2022-02-07 21:09:35,819 - root - INFO - running command.raw on 172.23.121.127: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:35,833 - root - INFO - command executed successfully with root 2022-02-07 21:09:36,941 - root - INFO - running command.raw on 172.23.121.127: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:36,954 - root - INFO - command executed successfully with root 2022-02-07 21:09:38,011 - root - INFO - running command.raw on 172.23.121.128: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:38,025 - root - INFO - command executed successfully with root 2022-02-07 21:09:39,062 - root - INFO - running command.raw on 172.23.121.128: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:39,077 - root - INFO - command executed successfully with root 2022-02-07 21:09:40,159 - root - INFO - running command.raw on 172.23.121.129: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:40,174 - root - INFO - command executed successfully with root 2022-02-07 21:09:41,201 - root - INFO - running command.raw on 172.23.121.129: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:41,216 - root - INFO - command executed successfully with root 2022-02-07 21:09:42,276 - root - INFO - running command.raw on 172.23.121.130: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:42,292 - root - INFO - command executed successfully with root 2022-02-07 21:09:43,361 - root - INFO - running command.raw on 172.23.121.130: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:43,374 - root - INFO - command executed successfully with root 2022-02-07 21:09:44,333 - root - INFO - running command.raw on 172.23.121.131: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:44,351 - root - INFO - command executed successfully with root 2022-02-07 21:09:45,439 - root - INFO - running command.raw on 172.23.121.131: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:45,453 - root - INFO - command executed successfully with root 2022-02-07 21:09:46,464 - root - INFO - running command.raw on 172.23.121.132: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:46,478 - root - INFO - command executed successfully with root 2022-02-07 21:09:47,691 - root - INFO - running command.raw on 172.23.121.132: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:47,704 - root - INFO - command executed successfully with root 2022-02-07 21:09:48,738 - root - INFO - running command.raw on 172.23.121.133: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:48,751 - root - INFO - command executed successfully with root 2022-02-07 21:09:49,880 - root - INFO - running command.raw on 172.23.121.133: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:49,895 - root - INFO - command executed successfully with root 2022-02-07 21:09:50,942 - root - INFO - running command.raw on 172.23.121.134: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:50,956 - root - INFO - command executed successfully with root 2022-02-07 21:09:52,123 - root - INFO - running command.raw on 172.23.121.134: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:52,136 - root - INFO - command executed successfully with root 2022-02-07 21:09:53,223 - root - INFO - running command.raw on 172.23.121.135: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:53,266 - root - INFO - command executed successfully with root 2022-02-07 21:09:54,404 - root - INFO - running command.raw on 172.23.121.135: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:54,418 - root - INFO - command executed successfully with root 2022-02-07 21:09:55,498 - root - INFO - running command.raw on 172.23.121.136: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:55,513 - root - INFO - command executed successfully with root 2022-02-07 21:09:56,537 - root - INFO - running command.raw on 172.23.121.136: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-2255-centos7.x86_64.rpm 2022-02-07 21:09:56,551 - root - INFO - command executed successfully with root 2022-02-07 21:13:50,631 - root - INFO - Done with install on 172.23.121.130. 2022-02-07 21:13:50,631 - root - INFO - Waiting for couchbase to be reachable 2022-02-07 21:13:50,639 - root - INFO - running command.raw on 172.23.121.130: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.121.130 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-07 21:13:51,492 - root - INFO - command executed successfully with root 2022-02-07 21:13:51,496 - root - INFO - running command.raw on 172.23.121.130: rm -rf /data/* 2022-02-07 21:13:51,637 - root - INFO - command executed successfully with root 2022-02-07 21:13:51,637 - root - INFO - running command.raw on 172.23.121.130: chown -R couchbase:couchbase /data 2022-02-07 21:13:51,690 - root - INFO - command executed successfully with root 2022-02-07 21:13:51,691 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-07 21:13:56,097 - root - INFO - Setting data_path: /data: status True 2022-02-07 21:13:56,521 - root - INFO - Done with install on 172.23.121.131. 2022-02-07 21:13:56,521 - root - INFO - Waiting for couchbase to be reachable 2022-02-07 21:13:56,528 - root - INFO - running command.raw on 172.23.121.131: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.121.131 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-07 21:13:56,707 - root - INFO - Done with install on 172.23.121.133. 2022-02-07 21:13:56,707 - root - INFO - Waiting for couchbase to be reachable 2022-02-07 21:13:56,709 - root - INFO - Done with install on 172.23.121.134. 2022-02-07 21:13:56,710 - root - INFO - Waiting for couchbase to be reachable 2022-02-07 21:13:56,714 - root - INFO - running command.raw on 172.23.121.133: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.121.133 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-07 21:13:56,716 - root - INFO - running command.raw on 172.23.121.134: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.121.134 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-07 21:13:57,103 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.121.130 2022-02-07 21:13:57,103 - root - INFO - pools/default params : memoryQuota=7855 2022-02-07 21:13:57,107 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-07 21:13:57,107 - root - INFO - /node/controller/setupServices params on 172.23.121.130: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-07 21:13:57,165 - root - INFO - command executed successfully with root 2022-02-07 21:13:57,169 - root - INFO - running command.raw on 172.23.121.131: rm -rf /data/* 2022-02-07 21:13:57,422 - root - INFO - command executed successfully with root 2022-02-07 21:13:57,424 - root - INFO - command executed successfully with root 2022-02-07 21:13:57,427 - root - INFO - running command.raw on 172.23.121.134: rm -rf /data/* 2022-02-07 21:13:57,429 - root - INFO - running command.raw on 172.23.121.133: rm -rf /data/* 2022-02-07 21:13:57,481 - root - INFO - command executed successfully with root 2022-02-07 21:13:57,482 - root - INFO - running command.raw on 172.23.121.134: chown -R couchbase:couchbase /data 2022-02-07 21:13:57,533 - root - INFO - command executed successfully with root 2022-02-07 21:13:57,533 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-07 21:13:57,852 - root - INFO - command executed successfully with root 2022-02-07 21:13:57,853 - root - INFO - running command.raw on 172.23.121.131: chown -R couchbase:couchbase /data 2022-02-07 21:13:58,038 - root - INFO - command executed successfully with root 2022-02-07 21:13:58,039 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-07 21:13:58,293 - root - INFO - command executed successfully with root 2022-02-07 21:13:58,293 - root - INFO - running command.raw on 172.23.121.133: chown -R couchbase:couchbase /data 2022-02-07 21:13:58,308 - root - INFO - command executed successfully with root 2022-02-07 21:13:58,308 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-07 21:14:01,488 - root - INFO - Setting data_path: /data: status True 2022-02-07 21:14:02,356 - root - INFO - Done with install on 172.23.121.132. 2022-02-07 21:14:02,356 - root - INFO - Waiting for couchbase to be reachable 2022-02-07 21:14:02,366 - root - INFO - running command.raw on 172.23.121.132: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.121.132 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-07 21:14:02,393 - root - INFO - Setting data_path: /data: status True 2022-02-07 21:14:02,493 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.121.133 2022-02-07 21:14:02,493 - root - INFO - pools/default params : memoryQuota=7855 2022-02-07 21:14:02,497 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-07 21:14:02,497 - root - INFO - /node/controller/setupServices params on 172.23.121.133: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-07 21:14:02,613 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-07 21:14:02,613 - root - INFO - settings/web params on 172.23.121.133:8091:port=8091&username=Administrator&password=password 2022-02-07 21:14:02,672 - root - INFO - --> status:True 2022-02-07 21:14:02,673 - root - INFO - Done with init on 172.23.121.133. 2022-02-07 21:14:02,808 - root - INFO - command executed successfully with root 2022-02-07 21:14:02,812 - root - INFO - running command.raw on 172.23.121.132: rm -rf /data/* 2022-02-07 21:14:02,868 - root - INFO - command executed successfully with root 2022-02-07 21:14:02,868 - root - INFO - running command.raw on 172.23.121.132: chown -R couchbase:couchbase /data 2022-02-07 21:14:02,890 - root - INFO - Done with cleanup on 172.23.121.133. 2022-02-07 21:14:02,923 - root - INFO - command executed successfully with root 2022-02-07 21:14:02,924 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-07 21:14:03,190 - root - INFO - Done with install on 172.23.121.135. 2022-02-07 21:14:03,190 - root - INFO - Waiting for couchbase to be reachable 2022-02-07 21:14:03,196 - root - INFO - running command.raw on 172.23.121.135: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.121.135 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-07 21:14:03,205 - root - INFO - Done with install on 172.23.121.116. 2022-02-07 21:14:03,205 - root - INFO - Waiting for couchbase to be reachable 2022-02-07 21:14:03,212 - root - INFO - running command.raw on 172.23.121.116: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.121.116 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-07 21:14:03,399 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.121.131 2022-02-07 21:14:03,399 - root - INFO - pools/default params : memoryQuota=7855 2022-02-07 21:14:03,403 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-07 21:14:03,403 - root - INFO - /node/controller/setupServices params on 172.23.121.131: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-07 21:14:03,507 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-07 21:14:03,507 - root - INFO - settings/web params on 172.23.121.131:8091:port=8091&username=Administrator&password=password 2022-02-07 21:14:03,567 - root - INFO - --> status:True 2022-02-07 21:14:03,567 - root - INFO - Done with init on 172.23.121.131. 2022-02-07 21:14:03,694 - root - INFO - command executed successfully with root 2022-02-07 21:14:03,698 - root - INFO - running command.raw on 172.23.121.135: rm -rf /data/* 2022-02-07 21:14:03,790 - root - INFO - Done with cleanup on 172.23.121.131. 2022-02-07 21:14:04,026 - root - INFO - command executed successfully with root 2022-02-07 21:14:04,029 - root - INFO - running command.raw on 172.23.121.116: rm -rf /data/* 2022-02-07 21:14:04,944 - root - INFO - Done with install on 172.23.121.136. 2022-02-07 21:14:04,944 - root - INFO - Waiting for couchbase to be reachable 2022-02-07 21:14:04,951 - root - INFO - running command.raw on 172.23.121.136: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.121.136 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-07 21:14:05,414 - root - INFO - command executed successfully with root 2022-02-07 21:14:05,417 - root - INFO - Setting data_path: /data: status True 2022-02-07 21:14:05,418 - root - INFO - running command.raw on 172.23.121.136: rm -rf /data/* 2022-02-07 21:14:05,761 - root - INFO - command executed successfully with root 2022-02-07 21:14:05,762 - root - INFO - running command.raw on 172.23.121.135: chown -R couchbase:couchbase /data 2022-02-07 21:14:06,030 - root - INFO - command executed successfully with root 2022-02-07 21:14:06,031 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-07 21:14:06,057 - root - INFO - command executed successfully with root 2022-02-07 21:14:06,058 - root - INFO - running command.raw on 172.23.121.116: chown -R couchbase:couchbase /data 2022-02-07 21:14:06,369 - root - INFO - command executed successfully with root 2022-02-07 21:14:06,370 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-07 21:14:06,421 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.121.132 2022-02-07 21:14:06,421 - root - INFO - pools/default params : memoryQuota=7855 2022-02-07 21:14:06,425 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-07 21:14:06,425 - root - INFO - /node/controller/setupServices params on 172.23.121.132: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-07 21:14:06,429 - root - INFO - Setting data_path: /data: status True 2022-02-07 21:14:06,455 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-07 21:14:06,455 - root - INFO - settings/web params on 172.23.121.132:8091:port=8091&username=Administrator&password=password 2022-02-07 21:14:06,515 - root - INFO - --> status:True 2022-02-07 21:14:06,515 - root - INFO - Done with init on 172.23.121.132. 2022-02-07 21:14:06,717 - root - INFO - Done with cleanup on 172.23.121.132. 2022-02-07 21:14:07,433 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.121.134 2022-02-07 21:14:07,434 - root - INFO - pools/default params : memoryQuota=7855 2022-02-07 21:14:07,437 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-07 21:14:07,437 - root - INFO - /node/controller/setupServices params on 172.23.121.134: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-07 21:14:07,488 - root - INFO - command executed successfully with root 2022-02-07 21:14:07,488 - root - INFO - running command.raw on 172.23.121.136: chown -R couchbase:couchbase /data 2022-02-07 21:14:07,818 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-07 21:14:07,818 - root - INFO - settings/web params on 172.23.121.134:8091:port=8091&username=Administrator&password=password 2022-02-07 21:14:07,878 - root - INFO - --> status:True 2022-02-07 21:14:07,879 - root - INFO - Done with init on 172.23.121.134. 2022-02-07 21:14:08,082 - root - INFO - Done with cleanup on 172.23.121.134. 2022-02-07 21:14:09,271 - root - INFO - Setting data_path: /data: status True 2022-02-07 21:14:09,817 - root - INFO - command executed successfully with root 2022-02-07 21:14:09,817 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-07 21:14:10,275 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.121.135 2022-02-07 21:14:10,275 - root - INFO - pools/default params : memoryQuota=7855 2022-02-07 21:14:10,279 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-07 21:14:10,279 - root - INFO - /node/controller/setupServices params on 172.23.121.135: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-07 21:14:10,373 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-07 21:14:10,373 - root - INFO - settings/web params on 172.23.121.135:8091:port=8091&username=Administrator&password=password 2022-02-07 21:14:10,432 - root - INFO - --> status:True 2022-02-07 21:14:10,432 - root - INFO - Done with init on 172.23.121.135. 2022-02-07 21:14:10,450 - root - INFO - Done with cleanup on 172.23.121.135. 2022-02-07 21:14:12,114 - root - ERROR - POST http://172.23.121.130:8091//node/controller/setupServices body: hostname=None&user=Administrator&password=password&services=kv headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'["Unexpected server error, request logged."]' auth: Administrator:password 2022-02-07 21:14:12,124 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-07 21:14:12,125 - root - INFO - settings/web params on 172.23.121.130:8091:port=8091&username=Administrator&password=password 2022-02-07 21:14:12,228 - root - INFO - Setting data_path: /data: status True 2022-02-07 21:14:13,233 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.121.136 2022-02-07 21:14:13,233 - root - INFO - pools/default params : memoryQuota=7855 2022-02-07 21:14:13,252 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-07 21:14:13,252 - root - INFO - /node/controller/setupServices params on 172.23.121.136: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-07 21:14:13,288 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-07 21:14:13,289 - root - INFO - settings/web params on 172.23.121.136:8091:port=8091&username=Administrator&password=password 2022-02-07 21:14:13,349 - root - INFO - --> status:True 2022-02-07 21:14:13,350 - root - INFO - Done with init on 172.23.121.136. 2022-02-07 21:14:13,367 - root - INFO - Done with cleanup on 172.23.121.136. 2022-02-07 21:14:15,149 - root - INFO - Setting data_path: /data: status True 2022-02-07 21:14:16,159 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.121.116 2022-02-07 21:14:16,159 - root - INFO - pools/default params : memoryQuota=7855 2022-02-07 21:14:16,163 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-07 21:14:16,163 - root - INFO - /node/controller/setupServices params on 172.23.121.116: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-07 21:14:16,193 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-07 21:14:16,193 - root - INFO - settings/web params on 172.23.121.116:8091:port=8091&username=Administrator&password=password 2022-02-07 21:14:16,252 - root - INFO - --> status:True 2022-02-07 21:14:16,252 - root - INFO - Done with init on 172.23.121.116. 2022-02-07 21:14:16,270 - root - INFO - Done with cleanup on 172.23.121.116. 2022-02-07 21:14:20,464 - root - INFO - --> status:True 2022-02-07 21:14:20,464 - root - INFO - Done with init on 172.23.121.130. 2022-02-07 21:14:20,671 - root - INFO - Done with cleanup on 172.23.121.130. 2022-02-07 21:14:28,264 - root - INFO - Done with install on 172.23.120.170. 2022-02-07 21:14:28,264 - root - INFO - Waiting for couchbase to be reachable 2022-02-07 21:14:28,271 - root - INFO - running command.raw on 172.23.120.170: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.120.170 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-07 21:14:28,748 - root - INFO - command executed successfully with root 2022-02-07 21:14:28,751 - root - INFO - running command.raw on 172.23.120.170: rm -rf /data/* 2022-02-07 21:14:29,499 - root - INFO - Done with install on 172.23.121.123. 2022-02-07 21:14:29,499 - root - INFO - Waiting for couchbase to be reachable 2022-02-07 21:14:29,505 - root - INFO - running command.raw on 172.23.121.123: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.121.123 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-07 21:14:29,751 - root - INFO - Done with install on 172.23.121.124. 2022-02-07 21:14:29,752 - root - INFO - Waiting for couchbase to be reachable 2022-02-07 21:14:29,758 - root - INFO - running command.raw on 172.23.121.124: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.121.124 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-07 21:14:29,971 - root - INFO - command executed successfully with root 2022-02-07 21:14:29,974 - root - INFO - running command.raw on 172.23.121.123: rm -rf /data/* 2022-02-07 21:14:30,474 - root - INFO - command executed successfully with root 2022-02-07 21:14:30,479 - root - INFO - running command.raw on 172.23.121.124: rm -rf /data/* 2022-02-07 21:14:30,518 - root - INFO - command executed successfully with root 2022-02-07 21:14:30,518 - root - INFO - running command.raw on 172.23.121.124: chown -R couchbase:couchbase /data 2022-02-07 21:14:30,650 - root - INFO - command executed successfully with root 2022-02-07 21:14:30,651 - root - INFO - running command.raw on 172.23.120.170: chown -R couchbase:couchbase /data 2022-02-07 21:14:31,358 - root - INFO - command executed successfully with root 2022-02-07 21:14:31,359 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-07 21:14:31,684 - root - INFO - command executed successfully with root 2022-02-07 21:14:31,684 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-07 21:14:31,810 - root - INFO - Done with install on 172.23.121.129. 2022-02-07 21:14:31,810 - root - INFO - Waiting for couchbase to be reachable 2022-02-07 21:14:31,816 - root - INFO - running command.raw on 172.23.121.129: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.121.129 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-07 21:14:31,871 - root - INFO - command executed successfully with root 2022-02-07 21:14:31,871 - root - INFO - running command.raw on 172.23.121.123: chown -R couchbase:couchbase /data 2022-02-07 21:14:32,306 - root - INFO - command executed successfully with root 2022-02-07 21:14:32,316 - root - INFO - running command.raw on 172.23.121.129: rm -rf /data/* 2022-02-07 21:14:32,668 - root - INFO - command executed successfully with root 2022-02-07 21:14:32,668 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-07 21:14:33,599 - root - INFO - command executed successfully with root 2022-02-07 21:14:33,600 - root - INFO - running command.raw on 172.23.121.129: chown -R couchbase:couchbase /data 2022-02-07 21:14:34,176 - root - INFO - Done with install on 172.23.121.115. 2022-02-07 21:14:34,176 - root - INFO - Waiting for couchbase to be reachable 2022-02-07 21:14:34,182 - root - INFO - running command.raw on 172.23.121.115: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.121.115 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-07 21:14:34,557 - root - INFO - command executed successfully with root 2022-02-07 21:14:34,557 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-07 21:14:35,001 - root - INFO - command executed successfully with root 2022-02-07 21:14:35,004 - root - INFO - running command.raw on 172.23.121.115: rm -rf /data/* 2022-02-07 21:14:36,231 - root - INFO - Setting data_path: /data: status True 2022-02-07 21:14:36,278 - root - INFO - command executed successfully with root 2022-02-07 21:14:36,278 - root - INFO - running command.raw on 172.23.121.115: chown -R couchbase:couchbase /data 2022-02-07 21:14:37,237 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.120.170 2022-02-07 21:14:37,237 - root - INFO - pools/default params : memoryQuota=7855 2022-02-07 21:14:37,241 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-07 21:14:37,241 - root - INFO - /node/controller/setupServices params on 172.23.120.170: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-07 21:14:37,446 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-07 21:14:37,446 - root - INFO - settings/web params on 172.23.120.170:8091:port=8091&username=Administrator&password=password 2022-02-07 21:14:37,504 - root - INFO - --> status:True 2022-02-07 21:14:37,504 - root - INFO - Done with init on 172.23.120.170. 2022-02-07 21:14:37,523 - root - INFO - Done with cleanup on 172.23.120.170. 2022-02-07 21:14:39,718 - root - INFO - command executed successfully with root 2022-02-07 21:14:39,719 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-07 21:14:41,364 - root - ERROR - POST http://172.23.121.124:8091//nodes/self/controller/settings body: path=%2Fdata headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'["Unexpected server error, request logged."]' auth: Administrator:password 2022-02-07 21:14:41,364 - root - ERROR - Unable to set data_path /data : b'["Unexpected server error, request logged."]' 2022-02-07 21:14:41,365 - root - ERROR - socket error while connecting to http://172.23.121.124:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:14:42,200 - root - INFO - Setting data_path: /data: status True 2022-02-07 21:14:43,205 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.121.123 2022-02-07 21:14:43,205 - root - INFO - pools/default params : memoryQuota=7855 2022-02-07 21:14:43,208 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-07 21:14:43,208 - root - INFO - /node/controller/setupServices params on 172.23.121.123: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-07 21:14:44,253 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-07 21:14:44,253 - root - INFO - settings/web params on 172.23.121.123:8091:port=8091&username=Administrator&password=password 2022-02-07 21:14:44,312 - root - INFO - --> status:True 2022-02-07 21:14:44,312 - root - INFO - Done with init on 172.23.121.123. 2022-02-07 21:14:44,330 - root - INFO - Done with cleanup on 172.23.121.123. 2022-02-07 21:14:44,369 - root - ERROR - socket error while connecting to http://172.23.121.124:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:14:44,562 - root - ERROR - POST http://172.23.121.129:8091//nodes/self/controller/settings body: path=%2Fdata headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'["Unexpected server error, request logged."]' auth: Administrator:password 2022-02-07 21:14:44,563 - root - ERROR - Unable to set data_path /data : b'["Unexpected server error, request logged."]' 2022-02-07 21:14:44,564 - root - ERROR - socket error while connecting to http://172.23.121.129:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:14:45,047 - root - INFO - Done with install on 172.23.121.126. 2022-02-07 21:14:45,047 - root - INFO - Waiting for couchbase to be reachable 2022-02-07 21:14:45,054 - root - INFO - running command.raw on 172.23.121.126: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.121.126 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-07 21:14:46,120 - root - INFO - command executed successfully with root 2022-02-07 21:14:46,124 - root - INFO - running command.raw on 172.23.121.126: rm -rf /data/* 2022-02-07 21:14:47,566 - root - ERROR - socket error while connecting to http://172.23.121.129:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:14:48,082 - root - INFO - command executed successfully with root 2022-02-07 21:14:48,083 - root - INFO - running command.raw on 172.23.121.126: chown -R couchbase:couchbase /data 2022-02-07 21:14:48,568 - root - INFO - command executed successfully with root 2022-02-07 21:14:48,568 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-07 21:14:49,722 - root - ERROR - POST http://172.23.121.115:8091//nodes/self/controller/settings body: path=%2Fdata headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'["Unexpected server error, request logged."]' auth: Administrator:password 2022-02-07 21:14:49,723 - root - ERROR - Unable to set data_path /data : b'["Unexpected server error, request logged."]' 2022-02-07 21:14:49,724 - root - ERROR - socket error while connecting to http://172.23.121.115:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:14:51,377 - root - INFO - Setting KV memory quota as 6283 MB on 172.23.121.124 2022-02-07 21:14:51,377 - root - INFO - pools/default params : memoryQuota=6283 2022-02-07 21:14:51,383 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-07 21:14:51,383 - root - INFO - /node/controller/setupServices params on 172.23.121.124: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-07 21:14:52,216 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-07 21:14:52,216 - root - INFO - settings/web params on 172.23.121.124:8091:port=8091&username=Administrator&password=password 2022-02-07 21:14:52,276 - root - INFO - --> status:True 2022-02-07 21:14:52,276 - root - INFO - Done with init on 172.23.121.124. 2022-02-07 21:14:52,296 - root - INFO - Done with cleanup on 172.23.121.124. 2022-02-07 21:14:52,729 - root - ERROR - socket error while connecting to http://172.23.121.115:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:14:53,573 - root - ERROR - socket error while connecting to http://172.23.121.129:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:14:57,694 - root - INFO - Done with install on 172.23.121.127. 2022-02-07 21:14:57,694 - root - INFO - Waiting for couchbase to be reachable 2022-02-07 21:14:57,701 - root - INFO - running command.raw on 172.23.121.127: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.121.127 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-07 21:14:58,572 - root - ERROR - POST http://172.23.121.126:8091//nodes/self/controller/settings body: path=%2Fdata headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'["Unexpected server error, request logged."]' auth: Administrator:password 2022-02-07 21:14:58,572 - root - ERROR - Unable to set data_path /data : b'["Unexpected server error, request logged."]' 2022-02-07 21:14:58,573 - root - ERROR - socket error while connecting to http://172.23.121.126:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:14:58,737 - root - ERROR - socket error while connecting to http://172.23.121.115:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:14:59,857 - root - INFO - command executed successfully with root 2022-02-07 21:14:59,861 - root - INFO - running command.raw on 172.23.121.127: rm -rf /data/* 2022-02-07 21:15:00,403 - root - INFO - Done with install on 172.23.121.128. 2022-02-07 21:15:00,404 - root - INFO - Waiting for couchbase to be reachable 2022-02-07 21:15:00,410 - root - INFO - running command.raw on 172.23.121.128: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.121.128 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-07 21:15:01,012 - root - INFO - command executed successfully with root 2022-02-07 21:15:01,015 - root - INFO - running command.raw on 172.23.121.128: rm -rf /data/* 2022-02-07 21:15:01,583 - root - ERROR - socket error while connecting to http://172.23.121.126:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:15:01,757 - root - INFO - command executed successfully with root 2022-02-07 21:15:01,757 - root - INFO - running command.raw on 172.23.121.127: chown -R couchbase:couchbase /data 2022-02-07 21:15:02,349 - root - INFO - command executed successfully with root 2022-02-07 21:15:02,349 - root - INFO - running command.raw on 172.23.121.128: chown -R couchbase:couchbase /data 2022-02-07 21:15:02,583 - root - INFO - command executed successfully with root 2022-02-07 21:15:02,584 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-07 21:15:02,594 - root - INFO - command executed successfully with root 2022-02-07 21:15:02,594 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-07 21:15:06,590 - root - INFO - Setting KV memory quota as 6283 MB on 172.23.121.129 2022-02-07 21:15:06,590 - root - INFO - pools/default params : memoryQuota=6283 2022-02-07 21:15:06,601 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-07 21:15:06,601 - root - INFO - /node/controller/setupServices params on 172.23.121.129: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-07 21:15:06,702 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-07 21:15:06,703 - root - INFO - settings/web params on 172.23.121.129:8091:port=8091&username=Administrator&password=password 2022-02-07 21:15:06,766 - root - INFO - --> status:True 2022-02-07 21:15:06,766 - root - INFO - Done with init on 172.23.121.129. 2022-02-07 21:15:06,784 - root - INFO - Done with cleanup on 172.23.121.129. 2022-02-07 21:15:06,838 - root - INFO - Setting data_path: /data: status True 2022-02-07 21:15:07,842 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.121.127 2022-02-07 21:15:07,843 - root - INFO - pools/default params : memoryQuota=7855 2022-02-07 21:15:07,847 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-07 21:15:07,847 - root - INFO - /node/controller/setupServices params on 172.23.121.127: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-07 21:15:07,944 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-07 21:15:07,944 - root - INFO - settings/web params on 172.23.121.127:8091:port=8091&username=Administrator&password=password 2022-02-07 21:15:08,003 - root - INFO - --> status:True 2022-02-07 21:15:08,003 - root - INFO - Done with init on 172.23.121.127. 2022-02-07 21:15:08,021 - root - INFO - Done with cleanup on 172.23.121.127. 2022-02-07 21:15:08,059 - root - INFO - Setting data_path: /data: status True 2022-02-07 21:15:08,593 - root - INFO - Setting KV memory quota as 6283 MB on 172.23.121.126 2022-02-07 21:15:08,594 - root - INFO - pools/default params : memoryQuota=6283 2022-02-07 21:15:08,600 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-07 21:15:08,601 - root - INFO - /node/controller/setupServices params on 172.23.121.126: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-07 21:15:09,020 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-07 21:15:09,021 - root - INFO - settings/web params on 172.23.121.126:8091:port=8091&username=Administrator&password=password 2022-02-07 21:15:09,064 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.121.128 2022-02-07 21:15:09,064 - root - INFO - pools/default params : memoryQuota=7855 2022-02-07 21:15:09,078 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-07 21:15:09,078 - root - INFO - /node/controller/setupServices params on 172.23.121.128: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-07 21:15:09,079 - root - INFO - --> status:True 2022-02-07 21:15:09,080 - root - INFO - Done with init on 172.23.121.126. 2022-02-07 21:15:09,101 - root - INFO - Done with cleanup on 172.23.121.126. 2022-02-07 21:15:09,106 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-07 21:15:09,107 - root - INFO - settings/web params on 172.23.121.128:8091:port=8091&username=Administrator&password=password 2022-02-07 21:15:09,166 - root - INFO - --> status:True 2022-02-07 21:15:09,167 - root - INFO - Done with init on 172.23.121.128. 2022-02-07 21:15:09,186 - root - INFO - Done with cleanup on 172.23.121.128. 2022-02-07 21:15:11,755 - root - INFO - Setting KV memory quota as 6283 MB on 172.23.121.115 2022-02-07 21:15:11,755 - root - INFO - pools/default params : memoryQuota=6283 2022-02-07 21:15:11,761 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-07 21:15:11,761 - root - INFO - /node/controller/setupServices params on 172.23.121.115: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-07 21:15:11,791 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-07 21:15:11,791 - root - INFO - settings/web params on 172.23.121.115:8091:port=8091&username=Administrator&password=password 2022-02-07 21:15:11,849 - root - INFO - --> status:True 2022-02-07 21:15:11,849 - root - INFO - Done with init on 172.23.121.115. 2022-02-07 21:15:11,866 - root - INFO - Done with cleanup on 172.23.121.115. 2022-02-07 21:15:11,890 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-07 21:15:11,958 - root - INFO - cluster:C1 node:172.23.120.170:8091 version:7.1.0-2255-enterprise aFamily:inet services:['kv'] 2022-02-07 21:15:11,959 - root - INFO - cluster:C2 node:172.23.121.115:8091 version:7.1.0-2255-enterprise aFamily:inet services:['kv'] 2022-02-07 21:15:11,959 - root - INFO - cluster:C3 node:172.23.121.116:8091 version:7.1.0-2255-enterprise aFamily:inet services:['kv'] 2022-02-07 21:15:11,959 - root - INFO - cluster:C4 node:172.23.121.123:8091 version:7.1.0-2255-enterprise aFamily:inet services:['kv'] 2022-02-07 21:15:11,959 - root - INFO - cluster:C5 node:172.23.121.124:8091 version:7.1.0-2255-enterprise aFamily:inet services:['kv'] 2022-02-07 21:15:11,959 - root - INFO - cluster:C6 node:172.23.121.126:8091 version:7.1.0-2255-enterprise aFamily:inet services:['kv'] 2022-02-07 21:15:11,959 - root - INFO - cluster:C7 node:172.23.121.127:8091 version:7.1.0-2255-enterprise aFamily:inet services:['kv'] 2022-02-07 21:15:11,959 - root - INFO - cluster:C8 node:172.23.121.128:8091 version:7.1.0-2255-enterprise aFamily:inet services:['kv'] 2022-02-07 21:15:11,959 - root - INFO - cluster:C9 node:172.23.121.129:8091 version:7.1.0-2255-enterprise aFamily:inet services:['kv'] 2022-02-07 21:15:11,959 - root - INFO - cluster:C10 node:172.23.121.130:8091 version:7.1.0-2255-enterprise aFamily:inet services:['kv'] 2022-02-07 21:15:11,959 - root - INFO - cluster:C11 node:172.23.121.131:8091 version:7.1.0-2255-enterprise aFamily:inet services:['kv'] 2022-02-07 21:15:11,959 - root - INFO - cluster:C12 node:172.23.121.132:8091 version:7.1.0-2255-enterprise aFamily:inet services:['kv'] 2022-02-07 21:15:11,959 - root - INFO - cluster:C13 node:172.23.121.133:8091 version:7.1.0-2255-enterprise aFamily:inet services:['kv'] 2022-02-07 21:15:11,959 - root - INFO - cluster:C14 node:172.23.121.134:8091 version:7.1.0-2255-enterprise aFamily:inet services:['kv'] 2022-02-07 21:15:11,960 - root - INFO - cluster:C15 node:172.23.121.135:8091 version:7.1.0-2255-enterprise aFamily:inet services:['kv'] 2022-02-07 21:15:11,960 - root - INFO - cluster:C16 node:172.23.121.136:8091 version:7.1.0-2255-enterprise aFamily:inet services:['kv'] 2022-02-07 21:15:11,960 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-07 21:15:11,960 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-07 21:15:11,960 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-07 21:15:11,960 - root - INFO - INSTALL COMPLETED ON: 172.23.120.170 2022-02-07 21:15:11,960 - root - INFO - INSTALL COMPLETED ON: 172.23.121.115 2022-02-07 21:15:11,960 - root - INFO - INSTALL COMPLETED ON: 172.23.121.116 2022-02-07 21:15:11,960 - root - INFO - INSTALL COMPLETED ON: 172.23.121.123 2022-02-07 21:15:11,960 - root - INFO - INSTALL COMPLETED ON: 172.23.121.124 2022-02-07 21:15:11,960 - root - INFO - INSTALL COMPLETED ON: 172.23.121.126 2022-02-07 21:15:11,960 - root - INFO - INSTALL COMPLETED ON: 172.23.121.127 2022-02-07 21:15:11,960 - root - INFO - INSTALL COMPLETED ON: 172.23.121.128 2022-02-07 21:15:11,960 - root - INFO - INSTALL COMPLETED ON: 172.23.121.129 2022-02-07 21:15:11,960 - root - INFO - INSTALL COMPLETED ON: 172.23.121.130 2022-02-07 21:15:11,960 - root - INFO - INSTALL COMPLETED ON: 172.23.121.131 2022-02-07 21:15:11,961 - root - INFO - INSTALL COMPLETED ON: 172.23.121.132 2022-02-07 21:15:11,961 - root - INFO - INSTALL COMPLETED ON: 172.23.121.133 2022-02-07 21:15:11,961 - root - INFO - INSTALL COMPLETED ON: 172.23.121.134 2022-02-07 21:15:11,961 - root - INFO - INSTALL COMPLETED ON: 172.23.121.135 2022-02-07 21:15:11,961 - root - INFO - INSTALL COMPLETED ON: 172.23.121.136 2022-02-07 21:15:11,961 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-07 21:15:11,961 - root - INFO - TOTAL INSTALL TIME = 529 seconds + status=0 + cd ../.. + rm -rf tr_for_install + git clone https://github.com/gradle/guides.git Cloning into 'guides'... + nodes='172.23.120.170 172.23.121.115 172.23.121.116 172.23.121.123 172.23.121.124 172.23.121.126 172.23.121.127 172.23.121.128 172.23.121.129 172.23.121.130 172.23.121.131 172.23.121.132 172.23.121.133 172.23.121.134 172.23.121.135 172.23.121.136' [magma_temp_job4] $ /bin/sh -xe /tmp/jenkins8820672608897874681.sh + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job4.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False,iterations=2,sdk_timeout=60,log_level=debug,infra_log_level=debug,skip_cleanup=True -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=9,graceful=True,skip_cleanup=True,num_items=1000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,key_size=18,assert_crashes_on_load=True,num_collections=20,maxttl=10,num_indexes=5,pc=10,index_nodes=5,query_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=60000,doc_ops=create:update:delete:read,durability=MAJORITY,crashes=20,bucket_purge_interval=0.04 -m rest' Starting a Gradle Daemon, 21 busy Daemons could not be reused, use --status for details > Configure project : Using Transaction_client :: 1.2.1 Using Java_client :: 3.2.2 Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job4/build/classes/java/main:/data/workspace/magma_temp_job4/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar scripts/ssh.py -i /tmp/magma_temp_job4.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False,iterations=2,sdk_timeout=60,log_level=debug,infra_log_level=debug,skip_cleanup=True -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=9,graceful=True,skip_cleanup=True,num_items=1000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,key_size=18,assert_crashes_on_load=True,num_collections=20,maxttl=10,num_indexes=5,pc=10,index_nodes=5,query_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=60000,doc_ops=create:update:delete:read,durability=MAJORITY,crashes=20,bucket_purge_interval=0.04 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job4/build/classes/java/main:/data/workspace/magma_temp_job4/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar scripts/eagles_all_around.py -i /tmp/magma_temp_job4.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False,iterations=2,sdk_timeout=60,log_level=debug,infra_log_level=debug,skip_cleanup=True -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=9,graceful=True,skip_cleanup=True,num_items=1000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,key_size=18,assert_crashes_on_load=True,num_collections=20,maxttl=10,num_indexes=5,pc=10,index_nodes=5,query_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=60000,doc_ops=create:update:delete:read,durability=MAJORITY,crashes=20,bucket_purge_interval=0.04 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job4/build/classes/java/main:/data/workspace/magma_temp_job4/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar:build/classes/java/main scripts/install.py -i /tmp/magma_temp_job4.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False,iterations=2,sdk_timeout=60,log_level=debug,infra_log_level=debug,skip_cleanup=True -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=9,graceful=True,skip_cleanup=True,num_items=1000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,key_size=18,assert_crashes_on_load=True,num_collections=20,maxttl=10,num_indexes=5,pc=10,index_nodes=5,query_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=60000,doc_ops=create:update:delete:read,durability=MAJORITY,crashes=20,bucket_purge_interval=0.04 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job4/build/classes/java/main:/data/workspace/magma_temp_job4/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/magma_temp_job4.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False,iterations=2,sdk_timeout=60,log_level=debug,infra_log_level=debug,skip_cleanup=True -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=9,graceful=True,skip_cleanup=True,num_items=1000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,key_size=18,assert_crashes_on_load=True,num_collections=20,maxttl=10,num_indexes=5,pc=10,index_nodes=5,query_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=60000,doc_ops=create:update:delete:read,durability=MAJORITY,crashes=20,bucket_purge_interval=0.04 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job4/build/classes/java/main:/data/workspace/magma_temp_job4/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/magma_temp_job4.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False,iterations=2,sdk_timeout=60,log_level=debug,infra_log_level=debug,skip_cleanup=True -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=9,graceful=True,skip_cleanup=True,num_items=1000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,key_size=18,assert_crashes_on_load=True,num_collections=20,maxttl=10,num_indexes=5,pc=10,index_nodes=5,query_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=60000,doc_ops=create:update:delete:read,durability=MAJORITY,crashes=20,bucket_purge_interval=0.04 -m rest > Task :compileJava warning: unknown enum constant javax.annotation.meta.When.MAYBE reason: class file for javax.annotation.meta.When not found Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. 1 warning > Task :testrunner Global Test input params: {'bucket_eviction_policy': 'fullEviction', 'bucket_storage': 'magma', 'cluster_name': 'magma_temp_job4', 'infra_log_level': 'debug', 'ini': '/tmp/magma_temp_job4.ini', 'iterations': '2', 'log_level': 'debug', 'num_nodes': 16, 'rerun': 'False', 'sdk_timeout': '60', 'skip_cleanup': 'True'} Logs will be stored at /data/workspace/magma_temp_job4/logs/testrunner-22-Feb-07_21-15-37/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job4.ini bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False,iterations=2,sdk_timeout=60,log_level=debug,infra_log_level=debug,skip_cleanup=True -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=9,graceful=True,skip_cleanup=True,num_items=1000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,key_size=18,assert_crashes_on_load=True,num_collections=20,maxttl=10,num_indexes=5,pc=10,index_nodes=5,query_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=60000,doc_ops=create:update:delete:read,durability=MAJORITY,crashes=20,bucket_purge_interval=0.04' Test Input params: {'fts_nodes': '0', 'doc_size': '1024', 'query_nodes': '0', 'ops_rate': '60000', 'maxttl': '10', 'num_collections': '20', 'crashes': '20', 'graceful': 'True', 'rerun': 'False', 'num_nodes': 16, 'doc_ops': 'create:update:delete:read', 'sdk_timeout': '60', 'bucket_purge_interval': '0.04', 'case_number': 1, 'num_indexes': '5', 'cluster_name': 'magma_temp_job4', 'ini': '/tmp/magma_temp_job4.ini', 'num_buckets': '1', 'durability': 'MAJORITY', 'bucket_storage': 'magma', 'log_level': 'debug', 'bucket_eviction_policy': 'fullEviction', 'iterations': '2', 'logs_folder': '/data/workspace/magma_temp_job4/logs/testrunner-22-Feb-07_21-15-37/test_1', 'skip_cleanup': 'True', 'nodes_init': '9', 'pc': '10', 'num_items': '1000000', 'cbas_nodes': '0', 'assert_crashes_on_load': 'True', 'bucket_names': 'GleamBook', 'index_nodes': '5', 'key_size': '18', 'infra_log_level': 'debug'} SystemTestMagma (aGoodDoctor.Hospital.Murphy) ... 2022-02-07 21:15:50,194 | infra | INFO | MainThread | [task_manager:__init__:13] Initiating TaskManager with 30 threads 2022-02-07 21:15:50,197 | infra | DEBUG | MainThread | [cluster:__init__:29] Initiating ServerTasks 2022-02-07 21:15:50,198 | test | INFO | MainThread | [basetestcase:log_setup_status:684] ========= Murphy setup started for test #1 SystemTestMagma ========= 2022-02-07 21:15:50,415 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 21:15:50,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:15:50,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 21:15:50,717 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:15:50,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 21:15:50,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 21:15:50,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 21:15:50,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 21:15:50,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 21:15:50,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 21:15:50,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 21:15:50,815 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 21:15:50,816 | test | INFO | MainThread | [basetestcase:log_setup_status:684] ========= BaseTestCase setup started for test #1 SystemTestMagma ========= 2022-02-07 21:15:50,818 | test | DEBUG | MainThread | [basetestcase:setUp:329] Cleaning up cluster 2022-02-07 21:15:51,513 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 120secs 2022-02-07 21:15:51,566 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.120.170:8091 2022-02-07 21:15:51,568 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2022-02-07 21:15:51,582 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.120.170:8091 is running 2022-02-07 21:15:51,585 | test | WARNING | MainThread | [basetestcase:tearDownEverything:630] CLEANUP WAS SKIPPED 2022-02-07 21:15:51,585 | infra | INFO | MainThread | [basetestcase:tearDownEverything:652] ========== tasks in thread pool ========== 2022-02-07 21:15:51,586 | infra | INFO | MainThread | [basetestcase:tearDownEverything:654] ========================================== 2022-02-07 21:15:51,586 | test | INFO | MainThread | [basetestcase:initialize_cluster:433] Initializing cluster : C1 2022-02-07 21:15:51,591 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 21:15:51,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:15:51,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 21:15:51,940 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:15:51,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 21:15:51,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 21:15:51,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 21:15:51,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 21:15:51,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 21:15:52,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 21:15:52,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 21:15:52,075 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 21:15:52,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:15:52,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 21:15:52,407 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:15:52,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 21:15:52,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 21:15:52,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 21:15:52,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 21:15:52,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 21:15:52,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 21:15:52,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 21:15:52,499 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.120.170 - Checking if couchbase-server is installed 2022-02-07 21:15:52,517 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.120.170 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:15:52,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /opt/couchbase/VERSION.txt 2022-02-07 21:15:52,566 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3718] 172.23.120.170 - Running systemd command 2022-02-07 21:15:52,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: systemctl stop couchbase-server.service 2022-02-07 21:15:54,272 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.120.170 port:8091 ssh_username:root 2022-02-07 21:15:54,273 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 21:15:54,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: rm -rf /data/* 2022-02-07 21:15:54,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:15:54,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2022-02-07 21:15:54,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:15:54,345 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 21:15:54,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:15:54,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 21:15:54,686 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:15:54,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 21:15:54,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 21:15:54,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 21:15:54,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 21:15:54,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 21:15:54,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 21:15:54,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 21:15:54,779 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.120.170 - Checking if couchbase-server is installed 2022-02-07 21:15:54,798 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.120.170 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:15:54,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /opt/couchbase/VERSION.txt 2022-02-07 21:15:54,845 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3818] 172.23.120.170 - Running systemd command 2022-02-07 21:15:54,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: systemctl start couchbase-server.service 2022-02-07 21:15:54,868 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.120.170 port:8091 ssh_username:root 2022-02-07 21:15:54,868 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 Got this failure java.net.ConnectException: Connection refused: /172.23.120.170:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0xdf85cef0, 0.0.0.0/0.0.0.0:36242] timeout=300.0>) 2022-02-07 21:15:54,878 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.120.170:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:15:55,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:15:55,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.120.170:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0xa475095f, 0.0.0.0/0.0.0.0:60454] timeout=300.0>) 2022-02-07 21:15:57,408 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.120.170:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:15:57,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:15:57,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.120.170:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0xd81662ea, 0.0.0.0/0.0.0.0:33091] timeout=300.0>) 2022-02-07 21:15:59,440 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.120.170:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:15:59,459 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:15:59,459 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.120.170:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0x5450c167, 0.0.0.0/0.0.0.0:48765] timeout=300.0>) 2022-02-07 21:16:01,463 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.120.170:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:01,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:01,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-07 21:16:03,503 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2022-02-07 21:16:03,517 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 21:16:03,525 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 21:16:03,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:16:03,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 21:16:03,872 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:16:03,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 21:16:03,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 21:16:03,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 21:16:03,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 21:16:03,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 21:16:03,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 21:16:03,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 21:16:04,000 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 21:16:04,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:16:04,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 21:16:04,336 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:16:04,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 21:16:04,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 21:16:04,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 21:16:04,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 21:16:04,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 21:16:04,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 21:16:04,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 21:16:04,428 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.115 - Checking if couchbase-server is installed 2022-02-07 21:16:04,447 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.115 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:16:04,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /opt/couchbase/VERSION.txt 2022-02-07 21:16:04,494 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3718] 172.23.121.115 - Running systemd command 2022-02-07 21:16:04,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: systemctl stop couchbase-server.service 2022-02-07 21:16:06,490 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.121.115 port:8091 ssh_username:root 2022-02-07 21:16:06,490 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 21:16:06,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: rm -rf /data/* 2022-02-07 21:16:06,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:16:06,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2022-02-07 21:16:06,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:16:06,561 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 21:16:06,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:16:06,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 21:16:06,904 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:16:06,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 21:16:06,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 21:16:06,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 21:16:06,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 21:16:06,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 21:16:06,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 21:16:06,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 21:16:07,000 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.115 - Checking if couchbase-server is installed 2022-02-07 21:16:07,020 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.115 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:16:07,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /opt/couchbase/VERSION.txt 2022-02-07 21:16:07,069 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3818] 172.23.121.115 - Running systemd command 2022-02-07 21:16:07,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: systemctl start couchbase-server.service 2022-02-07 21:16:07,092 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.121.115 port:8091 ssh_username:root 2022-02-07 21:16:07,094 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 Got this failure java.net.ConnectException: Connection refused: /172.23.121.115:8091 during connect (<_realsocket at 0x12 type=client open_count=1 channel=[id: 0x15fd9d04, 0.0.0.0/0.0.0.0:60454] timeout=300.0>) 2022-02-07 21:16:07,098 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.115:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:07,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:07,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.115:8091 during connect (<_realsocket at 0x13 type=client open_count=1 channel=[id: 0xa1b0f875, 0.0.0.0/0.0.0.0:51978] timeout=300.0>) 2022-02-07 21:16:09,117 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.115:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:09,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:09,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.115:8091 during connect (<_realsocket at 0x14 type=client open_count=1 channel=[id: 0x860ebccc, 0.0.0.0/0.0.0.0:50271] timeout=300.0>) 2022-02-07 21:16:11,134 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.115:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:11,145 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:11,147 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.115:8091 during connect (<_realsocket at 0x15 type=client open_count=1 channel=[id: 0x0d1d5cd0, 0.0.0.0/0.0.0.0:60318] timeout=300.0>) 2022-02-07 21:16:13,151 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.115:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:13,163 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:13,163 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-07 21:16:15,177 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2022-02-07 21:16:15,188 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 21:16:15,193 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 21:16:15,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:16:15,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 21:16:15,529 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:16:15,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 21:16:15,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 21:16:15,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 21:16:15,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 21:16:15,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 21:16:15,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 21:16:15,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 21:16:15,655 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 21:16:15,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:16:15,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 21:16:15,983 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:16:15,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 21:16:15,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 21:16:16,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 21:16:16,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 21:16:16,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 21:16:16,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 21:16:16,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 21:16:16,075 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.116 - Checking if couchbase-server is installed 2022-02-07 21:16:16,092 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.116 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:16:16,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /opt/couchbase/VERSION.txt 2022-02-07 21:16:16,142 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3718] 172.23.121.116 - Running systemd command 2022-02-07 21:16:16,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: systemctl stop couchbase-server.service 2022-02-07 21:16:18,085 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.121.116 port:8091 ssh_username:root 2022-02-07 21:16:18,085 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 21:16:18,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: rm -rf /data/* 2022-02-07 21:16:18,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:16:18,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2022-02-07 21:16:18,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:16:18,144 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 21:16:18,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:16:18,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 21:16:18,497 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:16:18,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 21:16:18,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 21:16:18,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 21:16:18,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 21:16:18,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 21:16:18,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 21:16:18,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 21:16:18,591 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.116 - Checking if couchbase-server is installed 2022-02-07 21:16:18,608 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.116 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:16:18,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /opt/couchbase/VERSION.txt 2022-02-07 21:16:18,654 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3818] 172.23.121.116 - Running systemd command 2022-02-07 21:16:18,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: systemctl start couchbase-server.service 2022-02-07 21:16:18,674 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.121.116 port:8091 ssh_username:root 2022-02-07 21:16:18,676 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 Got this failure java.net.ConnectException: Connection refused: /172.23.121.116:8091 during connect (<_realsocket at 0x16 type=client open_count=1 channel=[id: 0x736c4d91, 0.0.0.0/0.0.0.0:42546] timeout=300.0>) 2022-02-07 21:16:18,680 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.116:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:18,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:18,696 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.116:8091 during connect (<_realsocket at 0x17 type=client open_count=1 channel=[id: 0x4b027e97, 0.0.0.0/0.0.0.0:36516] timeout=300.0>) 2022-02-07 21:16:20,698 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.116:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:20,711 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:20,711 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.116:8091 during connect (<_realsocket at 0x18 type=client open_count=1 channel=[id: 0x8e3daf9e, 0.0.0.0/0.0.0.0:60850] timeout=300.0>) 2022-02-07 21:16:22,717 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.116:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:22,729 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:22,729 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.116:8091 during connect (<_realsocket at 0x19 type=client open_count=1 channel=[id: 0x2123f2ca, 0.0.0.0/0.0.0.0:60476] timeout=300.0>) 2022-02-07 21:16:24,742 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.116:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:24,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:24,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-07 21:16:26,776 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2022-02-07 21:16:26,792 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 21:16:26,796 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 21:16:26,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:16:27,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 21:16:27,151 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:16:27,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 21:16:27,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 21:16:27,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 21:16:27,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 21:16:27,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 21:16:27,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 21:16:27,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 21:16:27,306 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 21:16:27,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:16:27,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 21:16:27,651 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:16:27,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 21:16:27,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 21:16:27,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 21:16:27,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 21:16:27,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 21:16:27,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 21:16:27,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 21:16:27,740 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.123 - Checking if couchbase-server is installed 2022-02-07 21:16:27,759 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.123 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:16:27,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /opt/couchbase/VERSION.txt 2022-02-07 21:16:27,806 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3718] 172.23.121.123 - Running systemd command 2022-02-07 21:16:27,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: systemctl stop couchbase-server.service 2022-02-07 21:16:29,684 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.121.123 port:8091 ssh_username:root 2022-02-07 21:16:29,684 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 21:16:29,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: rm -rf /data/* 2022-02-07 21:16:29,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:16:29,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2022-02-07 21:16:29,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:16:29,747 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 21:16:29,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:16:30,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 21:16:30,091 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:16:30,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 21:16:30,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 21:16:30,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 21:16:30,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 21:16:30,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 21:16:30,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 21:16:30,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 21:16:30,223 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.123 - Checking if couchbase-server is installed 2022-02-07 21:16:30,242 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.123 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:16:30,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /opt/couchbase/VERSION.txt 2022-02-07 21:16:30,292 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3818] 172.23.121.123 - Running systemd command 2022-02-07 21:16:30,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: systemctl start couchbase-server.service 2022-02-07 21:16:30,315 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.121.123 port:8091 ssh_username:root 2022-02-07 21:16:30,315 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 Got this failure java.net.ConnectException: Connection refused: /172.23.121.123:8091 during connect (<_realsocket at 0x1a type=client open_count=1 channel=[id: 0xd1777a33, 0.0.0.0/0.0.0.0:55857] timeout=300.0>) 2022-02-07 21:16:30,322 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.123:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:30,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:30,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.123:8091 during connect (<_realsocket at 0x1b type=client open_count=1 channel=[id: 0x5e786e5e, 0.0.0.0/0.0.0.0:55566] timeout=300.0>) 2022-02-07 21:16:32,342 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.123:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:32,359 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:32,359 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.123:8091 during connect (<_realsocket at 0x1c type=client open_count=1 channel=[id: 0x4b0bd08f, 0.0.0.0/0.0.0.0:35864] timeout=300.0>) 2022-02-07 21:16:34,388 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.123:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:34,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:34,401 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.123:8091 during connect (<_realsocket at 0x1d type=client open_count=1 channel=[id: 0xedbb3bbe, 0.0.0.0/0.0.0.0:39697] timeout=300.0>) 2022-02-07 21:16:36,405 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.123:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:36,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:36,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-07 21:16:38,437 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2022-02-07 21:16:38,453 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 21:16:38,457 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 21:16:38,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:16:38,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 21:16:38,763 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:16:38,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 21:16:38,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 21:16:38,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 21:16:38,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 21:16:38,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 21:16:38,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 21:16:38,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 21:16:38,904 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 21:16:38,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:16:39,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 21:16:39,253 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:16:39,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 21:16:39,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 21:16:39,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 21:16:39,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 21:16:39,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 21:16:39,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 21:16:39,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 21:16:39,362 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.124 - Checking if couchbase-server is installed 2022-02-07 21:16:39,382 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.124 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:16:39,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /opt/couchbase/VERSION.txt 2022-02-07 21:16:39,433 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3718] 172.23.121.124 - Running systemd command 2022-02-07 21:16:39,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: systemctl stop couchbase-server.service 2022-02-07 21:16:41,766 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.121.124 port:8091 ssh_username:root 2022-02-07 21:16:41,766 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 21:16:41,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: rm -rf /data/* 2022-02-07 21:16:41,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:16:41,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2022-02-07 21:16:41,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:16:41,828 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 21:16:41,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:16:42,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 21:16:42,158 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:16:42,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 21:16:42,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 21:16:42,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 21:16:42,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 21:16:42,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 21:16:42,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 21:16:42,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 21:16:42,259 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.124 - Checking if couchbase-server is installed 2022-02-07 21:16:42,279 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.124 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:16:42,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /opt/couchbase/VERSION.txt 2022-02-07 21:16:42,332 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3818] 172.23.121.124 - Running systemd command 2022-02-07 21:16:42,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: systemctl start couchbase-server.service 2022-02-07 21:16:42,357 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.121.124 port:8091 ssh_username:root 2022-02-07 21:16:42,358 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 Got this failure java.net.ConnectException: Connection refused: /172.23.121.124:8091 during connect (<_realsocket at 0x1e type=client open_count=1 channel=[id: 0x9e800a06, 0.0.0.0/0.0.0.0:43316] timeout=300.0>) 2022-02-07 21:16:42,362 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.124:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:42,374 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:42,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.124:8091 during connect (<_realsocket at 0x1f type=client open_count=1 channel=[id: 0x37ded131, 0.0.0.0/0.0.0.0:33482] timeout=300.0>) 2022-02-07 21:16:44,381 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.124:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:44,392 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:44,392 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.124:8091 during connect (<_realsocket at 0x20 type=client open_count=1 channel=[id: 0x30f09f7d, 0.0.0.0/0.0.0.0:47513] timeout=300.0>) 2022-02-07 21:16:46,397 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.124:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:46,405 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:46,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.124:8091 during connect (<_realsocket at 0x21 type=client open_count=1 channel=[id: 0x8c1e77c2, 0.0.0.0/0.0.0.0:51975] timeout=300.0>) 2022-02-07 21:16:48,410 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.124:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:48,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:48,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-07 21:16:50,436 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2022-02-07 21:16:50,447 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 21:16:50,451 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 21:16:50,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:16:50,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 21:16:50,788 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:16:50,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 21:16:50,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 21:16:50,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 21:16:50,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 21:16:50,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 21:16:50,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 21:16:50,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 21:16:50,924 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 21:16:50,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:16:51,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 21:16:51,246 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:16:51,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 21:16:51,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 21:16:51,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 21:16:51,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 21:16:51,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 21:16:51,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 21:16:51,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 21:16:51,335 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.126 - Checking if couchbase-server is installed 2022-02-07 21:16:51,354 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.126 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:16:51,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /opt/couchbase/VERSION.txt 2022-02-07 21:16:51,404 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3718] 172.23.121.126 - Running systemd command 2022-02-07 21:16:51,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: systemctl stop couchbase-server.service 2022-02-07 21:16:53,375 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.121.126 port:8091 ssh_username:root 2022-02-07 21:16:53,377 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 21:16:53,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: rm -rf /data/* 2022-02-07 21:16:53,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:16:53,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2022-02-07 21:16:53,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:16:53,443 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 21:16:53,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:16:53,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 21:16:53,762 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:16:53,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 21:16:53,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 21:16:53,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 21:16:53,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 21:16:53,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 21:16:53,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 21:16:53,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 21:16:53,852 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.126 - Checking if couchbase-server is installed 2022-02-07 21:16:53,871 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.126 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:16:53,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /opt/couchbase/VERSION.txt 2022-02-07 21:16:53,918 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3818] 172.23.121.126 - Running systemd command 2022-02-07 21:16:53,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: systemctl start couchbase-server.service 2022-02-07 21:16:53,938 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.121.126 port:8091 ssh_username:root 2022-02-07 21:16:53,940 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 Got this failure java.net.ConnectException: Connection refused: /172.23.121.126:8091 during connect (<_realsocket at 0x22 type=client open_count=1 channel=[id: 0x8c01746c, 0.0.0.0/0.0.0.0:39726] timeout=300.0>) 2022-02-07 21:16:53,944 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.126:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:53,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:53,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.126:8091 during connect (<_realsocket at 0x23 type=client open_count=1 channel=[id: 0x38b93167, 0.0.0.0/0.0.0.0:60352] timeout=300.0>) 2022-02-07 21:16:55,963 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.126:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:55,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:55,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.126:8091 during connect (<_realsocket at 0x24 type=client open_count=1 channel=[id: 0x8c2c783c, 0.0.0.0/0.0.0.0:52047] timeout=300.0>) 2022-02-07 21:16:57,980 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.126:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:16:57,990 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:16:57,990 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.126:8091 during connect (<_realsocket at 0x25 type=client open_count=1 channel=[id: 0xe92a6012, 0.0.0.0/0.0.0.0:55475] timeout=300.0>) 2022-02-07 21:16:59,993 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.126:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:00,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:00,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-07 21:17:02,019 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2022-02-07 21:17:02,029 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 21:17:02,033 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 21:17:02,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:17:02,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 21:17:02,352 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:17:02,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 21:17:02,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 21:17:02,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 21:17:02,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 21:17:02,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 21:17:02,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 21:17:02,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 21:17:02,476 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 21:17:02,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:17:02,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 21:17:02,786 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:17:02,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 21:17:02,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 21:17:02,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 21:17:02,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 21:17:02,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 21:17:02,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 21:17:02,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 21:17:02,869 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.127 - Checking if couchbase-server is installed 2022-02-07 21:17:02,887 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.127 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:17:02,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /opt/couchbase/VERSION.txt 2022-02-07 21:17:02,931 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3718] 172.23.121.127 - Running systemd command 2022-02-07 21:17:02,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: systemctl stop couchbase-server.service 2022-02-07 21:17:05,065 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.121.127 port:8091 ssh_username:root 2022-02-07 21:17:05,066 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 21:17:05,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: rm -rf /data/* 2022-02-07 21:17:05,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:17:05,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2022-02-07 21:17:05,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:17:05,121 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 21:17:05,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:17:05,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 21:17:05,454 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:17:05,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 21:17:05,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 21:17:05,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 21:17:05,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 21:17:05,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 21:17:05,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 21:17:05,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 21:17:05,542 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.127 - Checking if couchbase-server is installed 2022-02-07 21:17:05,559 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.127 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:17:05,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /opt/couchbase/VERSION.txt 2022-02-07 21:17:05,607 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3818] 172.23.121.127 - Running systemd command 2022-02-07 21:17:05,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: systemctl start couchbase-server.service 2022-02-07 21:17:05,628 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.121.127 port:8091 ssh_username:root 2022-02-07 21:17:05,628 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 Got this failure java.net.ConnectException: Connection refused: /172.23.121.127:8091 during connect (<_realsocket at 0x26 type=client open_count=1 channel=[id: 0x6f6d6342, 0.0.0.0/0.0.0.0:60784] timeout=300.0>) 2022-02-07 21:17:05,631 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.127:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:05,645 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:05,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.127:8091 during connect (<_realsocket at 0x27 type=client open_count=1 channel=[id: 0x8ad9f6df, 0.0.0.0/0.0.0.0:46256] timeout=300.0>) 2022-02-07 21:17:07,651 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.127:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:07,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:07,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.127:8091 during connect (<_realsocket at 0x28 type=client open_count=1 channel=[id: 0x1887549f, 0.0.0.0/0.0.0.0:38950] timeout=300.0>) 2022-02-07 21:17:09,667 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.127:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:09,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:09,681 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.127:8091 during connect (<_realsocket at 0x29 type=client open_count=1 channel=[id: 0x011ac4c6, 0.0.0.0/0.0.0.0:33709] timeout=300.0>) 2022-02-07 21:17:11,686 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.127:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:11,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:11,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-07 21:17:13,711 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2022-02-07 21:17:13,721 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 21:17:13,726 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 21:17:13,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:17:13,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 21:17:14,051 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:17:14,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 21:17:14,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 21:17:14,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 21:17:14,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 21:17:14,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 21:17:14,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 21:17:14,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 21:17:14,174 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 21:17:14,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:17:14,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 21:17:14,487 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:17:14,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 21:17:14,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 21:17:14,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 21:17:14,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 21:17:14,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 21:17:14,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 21:17:14,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 21:17:14,575 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.128 - Checking if couchbase-server is installed 2022-02-07 21:17:14,594 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.128 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:17:14,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /opt/couchbase/VERSION.txt 2022-02-07 21:17:14,638 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3718] 172.23.121.128 - Running systemd command 2022-02-07 21:17:14,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: systemctl stop couchbase-server.service 2022-02-07 21:17:16,944 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.121.128 port:8091 ssh_username:root 2022-02-07 21:17:16,944 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 21:17:16,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: rm -rf /data/* 2022-02-07 21:17:16,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:17:16,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2022-02-07 21:17:16,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:17:16,999 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 21:17:17,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:17:17,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 21:17:17,306 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:17:17,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 21:17:17,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 21:17:17,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 21:17:17,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 21:17:17,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 21:17:17,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 21:17:17,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 21:17:17,391 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.128 - Checking if couchbase-server is installed 2022-02-07 21:17:17,407 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.128 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:17:17,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /opt/couchbase/VERSION.txt 2022-02-07 21:17:17,450 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3818] 172.23.121.128 - Running systemd command 2022-02-07 21:17:17,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: systemctl start couchbase-server.service 2022-02-07 21:17:17,470 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.121.128 port:8091 ssh_username:root 2022-02-07 21:17:17,470 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 Got this failure java.net.ConnectException: Connection refused: /172.23.121.128:8091 during connect (<_realsocket at 0x2a type=client open_count=1 channel=[id: 0x9871dcb5, 0.0.0.0/0.0.0.0:48119] timeout=300.0>) 2022-02-07 21:17:17,473 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.128:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:17,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:17,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.128:8091 during connect (<_realsocket at 0x2b type=client open_count=1 channel=[id: 0xb9d82b85, 0.0.0.0/0.0.0.0:53308] timeout=300.0>) 2022-02-07 21:17:19,486 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.128:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:19,499 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:19,500 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.128:8091 during connect (<_realsocket at 0x2c type=client open_count=1 channel=[id: 0xaa37c3ef, 0.0.0.0/0.0.0.0:48594] timeout=300.0>) 2022-02-07 21:17:21,506 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.128:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:21,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:21,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.128:8091 during connect (<_realsocket at 0x2d type=client open_count=1 channel=[id: 0xa9bcfccc, 0.0.0.0/0.0.0.0:58011] timeout=300.0>) 2022-02-07 21:17:23,519 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.128:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:23,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:23,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-07 21:17:25,628 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2022-02-07 21:17:25,638 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 21:17:25,642 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 21:17:25,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:17:25,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 21:17:25,990 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:17:25,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 21:17:26,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 21:17:26,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 21:17:26,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 21:17:26,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 21:17:26,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 21:17:26,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 21:17:26,118 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 21:17:26,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:17:26,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 21:17:26,433 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:17:26,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 21:17:26,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 21:17:26,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 21:17:26,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 21:17:26,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 21:17:26,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 21:17:26,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 21:17:26,519 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.129 - Checking if couchbase-server is installed 2022-02-07 21:17:26,538 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.129 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:17:26,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /opt/couchbase/VERSION.txt 2022-02-07 21:17:26,585 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3718] 172.23.121.129 - Running systemd command 2022-02-07 21:17:26,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: systemctl stop couchbase-server.service 2022-02-07 21:17:28,073 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.121.129 port:8091 ssh_username:root 2022-02-07 21:17:28,073 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 21:17:28,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: rm -rf /data/* 2022-02-07 21:17:28,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:17:28,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2022-02-07 21:17:28,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:17:28,128 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 21:17:28,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:17:28,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 21:17:28,467 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:17:28,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 21:17:28,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 21:17:28,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 21:17:28,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 21:17:28,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 21:17:28,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 21:17:28,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 21:17:28,552 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.129 - Checking if couchbase-server is installed 2022-02-07 21:17:28,569 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.129 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:17:28,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /opt/couchbase/VERSION.txt 2022-02-07 21:17:28,615 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3818] 172.23.121.129 - Running systemd command 2022-02-07 21:17:28,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: systemctl start couchbase-server.service 2022-02-07 21:17:28,635 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.121.129 port:8091 ssh_username:root 2022-02-07 21:17:28,635 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 Got this failure java.net.ConnectException: Connection refused: /172.23.121.129:8091 during connect (<_realsocket at 0x2e type=client open_count=1 channel=[id: 0x6d8bbc59, 0.0.0.0/0.0.0.0:41359] timeout=300.0>) 2022-02-07 21:17:28,640 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.129:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:28,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:28,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.129:8091 during connect (<_realsocket at 0x2f type=client open_count=1 channel=[id: 0x14c9cea0, 0.0.0.0/0.0.0.0:44235] timeout=300.0>) 2022-02-07 21:17:30,661 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.129:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:30,670 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:30,670 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.129:8091 during connect (<_realsocket at 0x30 type=client open_count=1 channel=[id: 0x135bcff2, 0.0.0.0/0.0.0.0:56256] timeout=300.0>) 2022-02-07 21:17:32,674 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.129:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:32,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:32,687 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.129:8091 during connect (<_realsocket at 0x31 type=client open_count=1 channel=[id: 0x7da3f372, 0.0.0.0/0.0.0.0:35847] timeout=300.0>) 2022-02-07 21:17:34,690 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.129:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:34,704 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:34,704 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-07 21:17:36,721 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2022-02-07 21:17:36,733 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 21:17:36,736 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-07 21:17:36,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:17:37,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-07 21:17:37,075 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:17:37,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-07 21:17:37,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-07 21:17:37,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-07 21:17:37,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-07 21:17:37,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-07 21:17:37,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-07 21:17:37,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-07 21:17:37,210 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-07 21:17:37,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:17:37,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-07 21:17:37,562 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:17:37,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-07 21:17:37,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-07 21:17:37,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-07 21:17:37,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-07 21:17:37,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-07 21:17:37,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-07 21:17:37,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-07 21:17:37,653 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.130 - Checking if couchbase-server is installed 2022-02-07 21:17:37,671 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.130 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:17:37,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /opt/couchbase/VERSION.txt 2022-02-07 21:17:37,720 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3718] 172.23.121.130 - Running systemd command 2022-02-07 21:17:37,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: systemctl stop couchbase-server.service 2022-02-07 21:17:39,487 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.121.130 port:8091 ssh_username:root 2022-02-07 21:17:39,487 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-07 21:17:39,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: rm -rf /data/* 2022-02-07 21:17:39,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:17:39,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2022-02-07 21:17:39,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:17:39,546 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-07 21:17:39,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:17:39,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-07 21:17:39,885 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:17:39,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-07 21:17:39,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-07 21:17:39,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-07 21:17:39,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-07 21:17:39,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-07 21:17:39,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-07 21:17:39,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-07 21:17:39,976 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.130 - Checking if couchbase-server is installed 2022-02-07 21:17:39,994 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.130 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:17:40,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /opt/couchbase/VERSION.txt 2022-02-07 21:17:40,043 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3818] 172.23.121.130 - Running systemd command 2022-02-07 21:17:40,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: systemctl start couchbase-server.service 2022-02-07 21:17:40,065 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.121.130 port:8091 ssh_username:root 2022-02-07 21:17:40,066 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 Got this failure java.net.ConnectException: Connection refused: /172.23.121.130:8091 during connect (<_realsocket at 0x32 type=client open_count=1 channel=[id: 0xf8c2e434, 0.0.0.0/0.0.0.0:40927] timeout=300.0>) 2022-02-07 21:17:40,069 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.130:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:40,079 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:40,079 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.130:8091 during connect (<_realsocket at 0x33 type=client open_count=1 channel=[id: 0x152a80d2, 0.0.0.0/0.0.0.0:50340] timeout=300.0>) 2022-02-07 21:17:42,085 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.130:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:42,096 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:42,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.130:8091 during connect (<_realsocket at 0x34 type=client open_count=1 channel=[id: 0xb1f9fcb9, 0.0.0.0/0.0.0.0:57905] timeout=300.0>) 2022-02-07 21:17:44,101 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.130:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:44,112 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:44,112 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.130:8091 during connect (<_realsocket at 0x35 type=client open_count=1 channel=[id: 0x691b3602, 0.0.0.0/0.0.0.0:48218] timeout=300.0>) 2022-02-07 21:17:46,117 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.130:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:46,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:46,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-07 21:17:48,158 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2022-02-07 21:17:48,171 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-07 21:17:48,174 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-07 21:17:48,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:17:48,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-07 21:17:48,517 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:17:48,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-07 21:17:48,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-07 21:17:48,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-07 21:17:48,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-07 21:17:48,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-07 21:17:48,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-07 21:17:48,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-07 21:17:48,641 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-07 21:17:48,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:17:48,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-07 21:17:48,957 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:17:48,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-07 21:17:48,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-07 21:17:48,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-07 21:17:48,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-07 21:17:49,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-07 21:17:49,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-07 21:17:49,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-07 21:17:49,048 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.131 - Checking if couchbase-server is installed 2022-02-07 21:17:49,065 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.131 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:17:49,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /opt/couchbase/VERSION.txt 2022-02-07 21:17:49,111 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3718] 172.23.121.131 - Running systemd command 2022-02-07 21:17:49,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: systemctl stop couchbase-server.service 2022-02-07 21:17:51,378 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.121.131 port:8091 ssh_username:root 2022-02-07 21:17:51,378 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-07 21:17:51,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: rm -rf /data/* 2022-02-07 21:17:51,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:17:51,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2022-02-07 21:17:51,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:17:51,434 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-07 21:17:51,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:17:51,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-07 21:17:51,765 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:17:51,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-07 21:17:51,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-07 21:17:51,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-07 21:17:51,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-07 21:17:51,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-07 21:17:51,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-07 21:17:51,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-07 21:17:51,855 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.131 - Checking if couchbase-server is installed 2022-02-07 21:17:51,872 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.131 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:17:51,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /opt/couchbase/VERSION.txt 2022-02-07 21:17:51,924 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3818] 172.23.121.131 - Running systemd command 2022-02-07 21:17:51,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: systemctl start couchbase-server.service 2022-02-07 21:17:51,946 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.121.131 port:8091 ssh_username:root 2022-02-07 21:17:51,946 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 Got this failure java.net.ConnectException: Connection refused: /172.23.121.131:8091 during connect (<_realsocket at 0x36 type=client open_count=1 channel=[id: 0x369e81d1, 0.0.0.0/0.0.0.0:60475] timeout=300.0>) 2022-02-07 21:17:51,948 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.131:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:51,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:51,959 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.131:8091 during connect (<_realsocket at 0x37 type=client open_count=1 channel=[id: 0xa56a983a, 0.0.0.0/0.0.0.0:43475] timeout=300.0>) 2022-02-07 21:17:53,961 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.131:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:53,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:53,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.131:8091 during connect (<_realsocket at 0x38 type=client open_count=1 channel=[id: 0x770577c8, 0.0.0.0/0.0.0.0:37937] timeout=300.0>) 2022-02-07 21:17:55,974 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.131:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:55,987 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:55,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.131:8091 during connect (<_realsocket at 0x39 type=client open_count=1 channel=[id: 0x7d21bd3c, 0.0.0.0/0.0.0.0:51440] timeout=300.0>) 2022-02-07 21:17:57,992 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.131:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:17:58,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:17:58,002 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-07 21:18:00,016 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2022-02-07 21:18:00,029 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-07 21:18:00,033 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-07 21:18:00,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:18:00,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-07 21:18:00,384 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:18:00,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-07 21:18:00,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-07 21:18:00,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-07 21:18:00,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-07 21:18:00,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-07 21:18:00,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-07 21:18:00,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-07 21:18:00,503 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-07 21:18:00,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:18:00,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-07 21:18:00,828 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:18:00,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-07 21:18:00,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-07 21:18:00,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-07 21:18:00,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-07 21:18:00,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-07 21:18:00,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-07 21:18:00,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-07 21:18:00,915 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.132 - Checking if couchbase-server is installed 2022-02-07 21:18:00,933 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.132 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:18:00,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /opt/couchbase/VERSION.txt 2022-02-07 21:18:00,979 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3718] 172.23.121.132 - Running systemd command 2022-02-07 21:18:00,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: systemctl stop couchbase-server.service 2022-02-07 21:18:03,167 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.121.132 port:8091 ssh_username:root 2022-02-07 21:18:03,167 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-07 21:18:03,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: rm -rf /data/* 2022-02-07 21:18:03,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:18:03,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2022-02-07 21:18:03,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:18:03,224 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-07 21:18:03,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:18:03,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-07 21:18:03,566 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:18:03,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-07 21:18:03,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-07 21:18:03,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-07 21:18:03,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-07 21:18:03,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-07 21:18:03,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-07 21:18:03,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-07 21:18:03,654 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.132 - Checking if couchbase-server is installed 2022-02-07 21:18:03,671 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.132 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:18:03,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /opt/couchbase/VERSION.txt 2022-02-07 21:18:03,717 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3818] 172.23.121.132 - Running systemd command 2022-02-07 21:18:03,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: systemctl start couchbase-server.service 2022-02-07 21:18:03,737 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.121.132 port:8091 ssh_username:root 2022-02-07 21:18:03,737 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 Got this failure java.net.ConnectException: Connection refused: /172.23.121.132:8091 during connect (<_realsocket at 0x3a type=client open_count=1 channel=[id: 0x858fd8a3, 0.0.0.0/0.0.0.0:47792] timeout=300.0>) 2022-02-07 21:18:03,740 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.132:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:03,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:03,752 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.132:8091 during connect (<_realsocket at 0x3b type=client open_count=1 channel=[id: 0x7a2fe31d, 0.0.0.0/0.0.0.0:45344] timeout=300.0>) 2022-02-07 21:18:05,755 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.132:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:05,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:05,765 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.132:8091 during connect (<_realsocket at 0x3c type=client open_count=1 channel=[id: 0xc4f4b454, 0.0.0.0/0.0.0.0:33565] timeout=300.0>) 2022-02-07 21:18:07,769 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.132:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:07,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:07,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.132:8091 during connect (<_realsocket at 0x3d type=client open_count=1 channel=[id: 0x349699b6, 0.0.0.0/0.0.0.0:41023] timeout=300.0>) 2022-02-07 21:18:09,782 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.132:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:09,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:09,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-07 21:18:11,805 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2022-02-07 21:18:11,815 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-07 21:18:11,819 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-07 21:18:11,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:18:12,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-07 21:18:12,145 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:18:12,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-07 21:18:12,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-07 21:18:12,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-07 21:18:12,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-07 21:18:12,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-07 21:18:12,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-07 21:18:12,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-07 21:18:12,273 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-07 21:18:12,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:18:12,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-07 21:18:12,596 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:18:12,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-07 21:18:12,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-07 21:18:12,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-07 21:18:12,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-07 21:18:12,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-07 21:18:12,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-07 21:18:12,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-07 21:18:12,683 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.133 - Checking if couchbase-server is installed 2022-02-07 21:18:12,700 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.133 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:18:12,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /opt/couchbase/VERSION.txt 2022-02-07 21:18:12,746 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3718] 172.23.121.133 - Running systemd command 2022-02-07 21:18:12,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: systemctl stop couchbase-server.service 2022-02-07 21:18:14,881 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.121.133 port:8091 ssh_username:root 2022-02-07 21:18:14,881 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-07 21:18:14,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: rm -rf /data/* 2022-02-07 21:18:14,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:18:14,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2022-02-07 21:18:14,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:18:14,937 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-07 21:18:15,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:18:15,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-07 21:18:15,266 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:18:15,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-07 21:18:15,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-07 21:18:15,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-07 21:18:15,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-07 21:18:15,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-07 21:18:15,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-07 21:18:15,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-07 21:18:15,355 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.133 - Checking if couchbase-server is installed 2022-02-07 21:18:15,372 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.133 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:18:15,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /opt/couchbase/VERSION.txt 2022-02-07 21:18:15,423 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3818] 172.23.121.133 - Running systemd command 2022-02-07 21:18:15,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: systemctl start couchbase-server.service 2022-02-07 21:18:15,444 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.121.133 port:8091 ssh_username:root 2022-02-07 21:18:15,444 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 Got this failure java.net.ConnectException: Connection refused: /172.23.121.133:8091 during connect (<_realsocket at 0x3e type=client open_count=1 channel=[id: 0x335b8a36, 0.0.0.0/0.0.0.0:43329] timeout=300.0>) 2022-02-07 21:18:15,448 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.133:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:15,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:15,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.133:8091 during connect (<_realsocket at 0x3f type=client open_count=1 channel=[id: 0x9dc258ee, 0.0.0.0/0.0.0.0:56487] timeout=300.0>) 2022-02-07 21:18:17,460 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.133:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:17,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:17,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.133:8091 during connect (<_realsocket at 0x40 type=client open_count=1 channel=[id: 0x608c6db0, 0.0.0.0/0.0.0.0:49190] timeout=300.0>) 2022-02-07 21:18:19,473 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.133:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:19,482 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:19,482 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.133:8091 during connect (<_realsocket at 0x41 type=client open_count=1 channel=[id: 0x8ec0f766, 0.0.0.0/0.0.0.0:54913] timeout=300.0>) 2022-02-07 21:18:21,486 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.133:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:21,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:21,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-07 21:18:23,513 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2022-02-07 21:18:23,523 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-07 21:18:23,526 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-07 21:18:23,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:18:23,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-07 21:18:23,868 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:18:23,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-07 21:18:23,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-07 21:18:23,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-07 21:18:23,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-07 21:18:23,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-07 21:18:23,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-07 21:18:23,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-07 21:18:23,997 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-07 21:18:24,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:18:24,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-07 21:18:24,332 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:18:24,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-07 21:18:24,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-07 21:18:24,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-07 21:18:24,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-07 21:18:24,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-07 21:18:24,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-07 21:18:24,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-07 21:18:24,448 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.134 - Checking if couchbase-server is installed 2022-02-07 21:18:24,466 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.134 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:18:24,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /opt/couchbase/VERSION.txt 2022-02-07 21:18:24,513 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3718] 172.23.121.134 - Running systemd command 2022-02-07 21:18:24,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: systemctl stop couchbase-server.service 2022-02-07 21:18:26,279 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.121.134 port:8091 ssh_username:root 2022-02-07 21:18:26,280 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-07 21:18:26,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: rm -rf /data/* 2022-02-07 21:18:26,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:18:26,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2022-02-07 21:18:26,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:18:26,336 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-07 21:18:26,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:18:26,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-07 21:18:26,687 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:18:26,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-07 21:18:26,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-07 21:18:26,711 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-07 21:18:26,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-07 21:18:26,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-07 21:18:26,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-07 21:18:26,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-07 21:18:26,773 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.134 - Checking if couchbase-server is installed 2022-02-07 21:18:26,789 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.134 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:18:26,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /opt/couchbase/VERSION.txt 2022-02-07 21:18:26,835 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3818] 172.23.121.134 - Running systemd command 2022-02-07 21:18:26,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: systemctl start couchbase-server.service 2022-02-07 21:18:26,855 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.121.134 port:8091 ssh_username:root 2022-02-07 21:18:26,855 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 Got this failure java.net.ConnectException: Connection refused: /172.23.121.134:8091 during connect (<_realsocket at 0x42 type=client open_count=1 channel=[id: 0x01ab1e07, 0.0.0.0/0.0.0.0:36565] timeout=300.0>) 2022-02-07 21:18:26,858 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.134:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:26,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:26,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.134:8091 during connect (<_realsocket at 0x43 type=client open_count=1 channel=[id: 0x2c5704c2, 0.0.0.0/0.0.0.0:36981] timeout=300.0>) 2022-02-07 21:18:28,871 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.134:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:28,880 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:28,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.134:8091 during connect (<_realsocket at 0x44 type=client open_count=1 channel=[id: 0xa4ee2619, 0.0.0.0/0.0.0.0:39457] timeout=300.0>) 2022-02-07 21:18:30,891 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.134:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:30,900 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:30,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.134:8091 during connect (<_realsocket at 0x45 type=client open_count=1 channel=[id: 0xb0426be8, 0.0.0.0/0.0.0.0:52721] timeout=300.0>) 2022-02-07 21:18:32,904 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.134:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:32,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:32,915 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-07 21:18:34,930 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2022-02-07 21:18:34,940 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-07 21:18:34,944 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-07 21:18:35,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:18:35,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-07 21:18:35,285 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:18:35,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-07 21:18:35,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-07 21:18:35,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-07 21:18:35,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-07 21:18:35,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-07 21:18:35,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-07 21:18:35,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-07 21:18:35,515 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-07 21:18:35,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:18:35,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-07 21:18:35,846 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:18:35,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-07 21:18:35,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-07 21:18:35,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-07 21:18:35,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-07 21:18:35,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-07 21:18:35,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-07 21:18:35,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-07 21:18:35,937 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.135 - Checking if couchbase-server is installed 2022-02-07 21:18:35,954 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.135 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:18:35,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /opt/couchbase/VERSION.txt 2022-02-07 21:18:36,002 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3718] 172.23.121.135 - Running systemd command 2022-02-07 21:18:36,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: systemctl stop couchbase-server.service 2022-02-07 21:18:37,885 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.121.135 port:8091 ssh_username:root 2022-02-07 21:18:37,885 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-07 21:18:37,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: rm -rf /data/* 2022-02-07 21:18:37,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:18:37,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2022-02-07 21:18:37,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:18:37,943 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-07 21:18:38,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:18:38,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-07 21:18:38,301 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:18:38,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-07 21:18:38,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-07 21:18:38,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-07 21:18:38,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-07 21:18:38,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-07 21:18:38,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-07 21:18:38,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-07 21:18:38,398 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.135 - Checking if couchbase-server is installed 2022-02-07 21:18:38,417 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.135 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:18:38,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /opt/couchbase/VERSION.txt 2022-02-07 21:18:38,463 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3818] 172.23.121.135 - Running systemd command 2022-02-07 21:18:38,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: systemctl start couchbase-server.service 2022-02-07 21:18:38,484 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.121.135 port:8091 ssh_username:root 2022-02-07 21:18:38,486 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 Got this failure java.net.ConnectException: Connection refused: /172.23.121.135:8091 during connect (<_realsocket at 0x46 type=client open_count=1 channel=[id: 0xb141e5f2, 0.0.0.0/0.0.0.0:37384] timeout=300.0>) 2022-02-07 21:18:38,490 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.135:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:38,499 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:38,499 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.135:8091 during connect (<_realsocket at 0x47 type=client open_count=1 channel=[id: 0xecda6168, 0.0.0.0/0.0.0.0:43551] timeout=300.0>) 2022-02-07 21:18:40,502 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.135:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:40,512 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:40,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.135:8091 during connect (<_realsocket at 0x48 type=client open_count=1 channel=[id: 0x673d0787, 0.0.0.0/0.0.0.0:58359] timeout=300.0>) 2022-02-07 21:18:42,517 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.135:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:42,526 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:42,526 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.135:8091 during connect (<_realsocket at 0x49 type=client open_count=1 channel=[id: 0x686b0435, 0.0.0.0/0.0.0.0:42903] timeout=300.0>) 2022-02-07 21:18:44,532 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.135:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:44,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:44,548 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-07 21:18:46,565 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2022-02-07 21:18:46,575 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-07 21:18:46,578 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-07 21:18:46,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:18:46,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-07 21:18:46,923 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:18:46,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-07 21:18:46,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-07 21:18:46,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-07 21:18:46,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-07 21:18:46,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-07 21:18:46,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-07 21:18:47,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-07 21:18:47,049 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-07 21:18:47,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:18:47,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-07 21:18:47,381 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:18:47,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-07 21:18:47,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-07 21:18:47,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-07 21:18:47,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-07 21:18:47,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-07 21:18:47,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-07 21:18:47,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-07 21:18:47,473 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.136 - Checking if couchbase-server is installed 2022-02-07 21:18:47,490 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.136 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:18:47,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /opt/couchbase/VERSION.txt 2022-02-07 21:18:47,538 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3718] 172.23.121.136 - Running systemd command 2022-02-07 21:18:47,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: systemctl stop couchbase-server.service 2022-02-07 21:18:49,684 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.121.136 port:8091 ssh_username:root 2022-02-07 21:18:49,684 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-07 21:18:49,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: rm -rf /data/* 2022-02-07 21:18:49,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:18:49,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2022-02-07 21:18:49,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2022-02-07 21:18:49,743 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-07 21:18:49,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:18:50,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-07 21:18:50,068 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:18:50,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-07 21:18:50,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-07 21:18:50,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-07 21:18:50,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-07 21:18:50,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-07 21:18:50,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-07 21:18:50,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-07 21:18:50,155 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:709] 172.23.121.136 - Checking if couchbase-server is installed 2022-02-07 21:18:50,174 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:742] 172.23.121.136 - The version file /opt/couchbase/ VERSION.txt exists 2022-02-07 21:18:50,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /opt/couchbase/VERSION.txt 2022-02-07 21:18:50,221 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3818] 172.23.121.136 - Running systemd command 2022-02-07 21:18:50,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: systemctl start couchbase-server.service 2022-02-07 21:18:50,243 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.121.136 port:8091 ssh_username:root 2022-02-07 21:18:50,243 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 Got this failure java.net.ConnectException: Connection refused: /172.23.121.136:8091 during connect (<_realsocket at 0x4a type=client open_count=1 channel=[id: 0xead519a1, 0.0.0.0/0.0.0.0:38858] timeout=300.0>) 2022-02-07 21:18:50,249 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.136:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:50,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:50,260 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.136:8091 during connect (<_realsocket at 0x4b type=client open_count=1 channel=[id: 0xeeb17ebd, 0.0.0.0/0.0.0.0:39551] timeout=300.0>) 2022-02-07 21:18:52,263 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.136:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:52,275 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:52,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.136:8091 during connect (<_realsocket at 0x4c type=client open_count=1 channel=[id: 0x3f375d30, 0.0.0.0/0.0.0.0:34069] timeout=300.0>) 2022-02-07 21:18:54,279 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.136:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:54,289 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:54,289 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.121.136:8091 during connect (<_realsocket at 0x4d type=client open_count=1 channel=[id: 0xd7b759ec, 0.0.0.0/0.0.0.0:33170] timeout=300.0>) 2022-02-07 21:18:56,292 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.121.136:8091/nodes/self error [Errno 111] Connection refused 2022-02-07 21:18:56,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L848 2022-02-07 21:18:56,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-07 21:18:58,319 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2022-02-07 21:18:58,328 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-07 21:18:58,336 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> reset_cluster():L562 2022-02-07 21:18:58,336 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-07 21:19:08,341 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 21:19:08,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:08,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 21:19:08,691 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:08,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 21:19:08,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 21:19:08,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 21:19:08,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 21:19:08,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 21:19:08,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 21:19:08,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 21:19:08,789 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 21:19:08,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:09,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 21:19:09,144 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:09,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 21:19:09,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 21:19:09,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 21:19:09,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 21:19:09,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 21:19:09,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 21:19:09,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 21:19:09,240 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 21:19:09,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:09,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 21:19:09,582 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:09,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 21:19:09,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 21:19:09,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 21:19:09,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 21:19:09,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 21:19:09,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 21:19:09,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 21:19:09,680 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 21:19:09,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:09,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 21:19:10,056 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:10,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 21:19:10,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 21:19:10,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 21:19:10,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 21:19:10,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 21:19:10,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 21:19:10,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 21:19:10,161 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 21:19:10,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:10,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 21:19:10,513 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:10,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 21:19:10,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 21:19:10,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 21:19:10,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 21:19:10,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 21:19:10,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 21:19:10,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 21:19:10,617 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 21:19:10,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:10,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 21:19:10,959 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:10,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 21:19:10,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 21:19:10,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 21:19:10,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 21:19:11,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 21:19:11,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 21:19:11,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 21:19:11,055 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 21:19:11,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:11,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 21:19:11,394 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:11,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 21:19:11,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 21:19:11,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 21:19:11,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 21:19:11,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 21:19:11,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 21:19:11,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 21:19:11,490 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 21:19:11,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:11,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 21:19:11,832 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:11,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 21:19:11,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 21:19:11,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 21:19:11,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 21:19:11,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 21:19:11,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 21:19:11,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 21:19:11,961 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 21:19:12,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:12,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 21:19:12,292 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:12,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 21:19:12,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 21:19:12,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 21:19:12,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 21:19:12,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 21:19:12,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 21:19:12,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 21:19:12,387 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-07 21:19:12,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:12,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-07 21:19:12,726 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:12,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-07 21:19:12,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-07 21:19:12,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-07 21:19:12,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-07 21:19:12,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-07 21:19:12,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-07 21:19:12,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-07 21:19:12,822 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-07 21:19:12,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:13,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-07 21:19:13,167 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:13,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-07 21:19:13,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-07 21:19:13,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-07 21:19:13,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-07 21:19:13,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-07 21:19:13,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-07 21:19:13,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-07 21:19:13,263 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-07 21:19:13,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:13,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-07 21:19:13,589 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:13,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-07 21:19:13,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-07 21:19:13,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-07 21:19:13,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-07 21:19:13,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-07 21:19:13,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-07 21:19:13,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-07 21:19:13,681 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-07 21:19:13,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:13,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-07 21:19:14,026 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:14,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-07 21:19:14,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-07 21:19:14,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-07 21:19:14,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-07 21:19:14,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-07 21:19:14,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-07 21:19:14,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-07 21:19:14,125 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-07 21:19:14,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:14,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-07 21:19:14,466 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:14,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-07 21:19:14,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-07 21:19:14,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-07 21:19:14,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-07 21:19:14,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-07 21:19:14,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-07 21:19:14,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-07 21:19:14,561 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-07 21:19:14,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:14,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-07 21:19:14,923 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:14,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-07 21:19:14,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-07 21:19:14,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-07 21:19:14,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-07 21:19:14,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-07 21:19:14,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-07 21:19:15,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-07 21:19:15,019 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-07 21:19:15,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:15,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-07 21:19:15,375 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:15,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-07 21:19:15,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-07 21:19:15,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-07 21:19:15,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-07 21:19:15,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-07 21:19:15,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-07 21:19:15,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-07 21:19:15,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: rm -rf /data/* 2022-02-07 21:19:15,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: chown -R couchbase:couchbase /data 2022-02-07 21:19:15,525 | test | DEBUG | MainThread | [rest_client:set_data_path:2273] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2022-02-07 21:19:17,783 | test | DEBUG | MainThread | [rest_client:set_data_path:2276] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2022-02-07 21:19:17,786 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.120.170_8091 2022-02-07 21:19:17,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: rm -rf /data/* 2022-02-07 21:19:17,790 | infra | INFO | pool-3-thread-1 | [task:start_task:91] Thread 'node_init_task_172.23.120.170_8091' started 2022-02-07 21:19:17,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: chown -R couchbase:couchbase /data 2022-02-07 21:19:17,822 | infra | DEBUG | pool-3-thread-1 | [task:wait_until:135] Wait for expected condition to get satisfied 2022-02-07 21:19:17,834 | test | DEBUG | MainThread | [rest_client:set_data_path:2273] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2022-02-07 21:19:17,835 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:17,838 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2022-02-07 21:19:17,862 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:17,862 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2022-02-07 21:19:17,897 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:17,898 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2022-02-07 21:19:17,956 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:17,956 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2022-02-07 21:19:18,055 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:18,055 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2022-02-07 21:19:18,233 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:18,233 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2022-02-07 21:19:18,569 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:18,569 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2022-02-07 21:19:19,226 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:19,226 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2022-02-07 21:19:20,346 | test | DEBUG | MainThread | [rest_client:set_data_path:2276] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2022-02-07 21:19:20,348 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.121.115_8091 2022-02-07 21:19:20,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: rm -rf /data/* 2022-02-07 21:19:20,349 | infra | INFO | pool-3-thread-2 | [task:start_task:91] Thread 'node_init_task_172.23.121.115_8091' started 2022-02-07 21:19:20,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: chown -R couchbase:couchbase /data 2022-02-07 21:19:20,375 | infra | DEBUG | pool-3-thread-2 | [task:wait_until:135] Wait for expected condition to get satisfied 2022-02-07 21:19:20,381 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:20,381 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2022-02-07 21:19:20,398 | test | DEBUG | MainThread | [rest_client:set_data_path:2273] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2022-02-07 21:19:20,411 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:20,411 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2022-02-07 21:19:20,446 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:20,446 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2022-02-07 21:19:20,499 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:20,500 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2022-02-07 21:19:20,520 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:20,520 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2022-02-07 21:19:20,592 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:20,592 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2022-02-07 21:19:20,769 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:20,770 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2022-02-07 21:19:21,105 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:21,105 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2022-02-07 21:19:21,757 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:21,759 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2022-02-07 21:19:22,900 | test | DEBUG | MainThread | [rest_client:set_data_path:2276] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2022-02-07 21:19:22,901 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.121.116_8091 2022-02-07 21:19:22,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: rm -rf /data/* 2022-02-07 21:19:22,910 | infra | INFO | pool-3-thread-3 | [task:start_task:91] Thread 'node_init_task_172.23.121.116_8091' started 2022-02-07 21:19:22,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: chown -R couchbase:couchbase /data 2022-02-07 21:19:22,936 | infra | DEBUG | pool-3-thread-3 | [task:wait_until:135] Wait for expected condition to get satisfied 2022-02-07 21:19:22,940 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:22,940 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2022-02-07 21:19:22,943 | test | DEBUG | MainThread | [rest_client:set_data_path:2273] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2022-02-07 21:19:22,964 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:22,964 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2022-02-07 21:19:22,999 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:23,000 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2022-02-07 21:19:23,058 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:23,059 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2022-02-07 21:19:23,062 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:23,062 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2022-02-07 21:19:23,098 | test | DEBUG | pool-3-thread-1 | [task:call:6151] server: ip:172.23.120.170 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11424227328L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.120.170:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 9379, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.120.170', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.1.0-2255-enterprise', 'indexMemoryQuota': 512, 'uptime': u'205', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294115328L, 'memoryQuota': 6368, 'status': u'healthy'} 2022-02-07 21:19:23,099 | test | DEBUG | pool-3-thread-1 | [rest_client:set_service_mem_quota:945] Setting RAM quotas: {'memoryQuota': 8351} 2022-02-07 21:19:23,115 | test | DEBUG | pool-3-thread-1 | [rest_client:set_indexer_storage_mode:969] settings/indexes params: storageMode=plasma 2022-02-07 21:19:23,130 | test | DEBUG | pool-3-thread-1 | [rest_client:init_cluster:856] settings/web params on 172.23.120.170:8091:password=password&port=8091&username=Administrator 2022-02-07 21:19:23,154 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:23,155 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2022-02-07 21:19:23,328 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:23,328 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2022-02-07 21:19:23,664 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:23,664 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2022-02-07 21:19:24,319 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:24,321 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2022-02-07 21:19:25,441 | test | DEBUG | MainThread | [rest_client:set_data_path:2276] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2022-02-07 21:19:25,443 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.121.123_8091 2022-02-07 21:19:25,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: rm -rf /data/* 2022-02-07 21:19:25,444 | infra | INFO | pool-3-thread-4 | [task:start_task:91] Thread 'node_init_task_172.23.121.123_8091' started 2022-02-07 21:19:25,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: chown -R couchbase:couchbase /data 2022-02-07 21:19:25,470 | infra | DEBUG | pool-3-thread-4 | [task:wait_until:135] Wait for expected condition to get satisfied 2022-02-07 21:19:25,473 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:25,473 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2022-02-07 21:19:25,489 | test | DEBUG | MainThread | [rest_client:set_data_path:2273] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2022-02-07 21:19:25,496 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:25,497 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2022-02-07 21:19:25,532 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:25,532 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2022-02-07 21:19:25,588 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:25,588 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2022-02-07 21:19:25,615 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:25,615 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2022-02-07 21:19:25,632 | test | DEBUG | pool-3-thread-2 | [task:call:6151] server: ip:172.23.121.115 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11523461120L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.121.115:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 9379, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.121.115', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.1.0-2255-enterprise', 'indexMemoryQuota': 512, 'uptime': u'196', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294119424L, 'memoryQuota': 6368, 'status': u'healthy'} 2022-02-07 21:19:25,634 | test | DEBUG | pool-3-thread-2 | [rest_client:set_service_mem_quota:945] Setting RAM quotas: {'memoryQuota': 8351} 2022-02-07 21:19:25,645 | test | DEBUG | pool-3-thread-2 | [rest_client:set_indexer_storage_mode:969] settings/indexes params: storageMode=plasma 2022-02-07 21:19:25,661 | test | DEBUG | pool-3-thread-2 | [rest_client:init_cluster:856] settings/web params on 172.23.121.115:8091:password=password&port=8091&username=Administrator 2022-02-07 21:19:25,686 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:25,686 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2022-02-07 21:19:25,859 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:25,859 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2022-02-07 21:19:26,193 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:26,193 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2022-02-07 21:19:26,846 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:26,846 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2022-02-07 21:19:27,709 | test | DEBUG | MainThread | [rest_client:set_data_path:2276] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2022-02-07 21:19:27,710 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.121.124_8091 2022-02-07 21:19:27,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: rm -rf /data/* 2022-02-07 21:19:27,713 | infra | INFO | pool-3-thread-5 | [task:start_task:91] Thread 'node_init_task_172.23.121.124_8091' started 2022-02-07 21:19:27,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: chown -R couchbase:couchbase /data 2022-02-07 21:19:27,739 | infra | DEBUG | pool-3-thread-5 | [task:wait_until:135] Wait for expected condition to get satisfied 2022-02-07 21:19:27,742 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:27,742 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2022-02-07 21:19:27,767 | test | DEBUG | MainThread | [rest_client:set_data_path:2273] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2022-02-07 21:19:27,773 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:27,773 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2022-02-07 21:19:27,811 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:27,812 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2022-02-07 21:19:27,868 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:27,868 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2022-02-07 21:19:27,963 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:27,963 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2022-02-07 21:19:28,144 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:28,144 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2022-02-07 21:19:28,144 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:28,144 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2022-02-07 21:19:28,187 | test | DEBUG | pool-3-thread-3 | [task:call:6151] server: ip:172.23.121.116 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11539525632L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.121.116:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 9379, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.121.116', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.1.0-2255-enterprise', 'indexMemoryQuota': 512, 'uptime': u'187', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294119424L, 'memoryQuota': 6368, 'status': u'healthy'} 2022-02-07 21:19:28,190 | test | DEBUG | pool-3-thread-3 | [rest_client:set_service_mem_quota:945] Setting RAM quotas: {'memoryQuota': 8351} 2022-02-07 21:19:28,204 | test | DEBUG | pool-3-thread-3 | [rest_client:set_indexer_storage_mode:969] settings/indexes params: storageMode=plasma 2022-02-07 21:19:28,223 | test | DEBUG | pool-3-thread-3 | [rest_client:init_cluster:856] settings/web params on 172.23.121.116:8091:password=password&port=8091&username=Administrator 2022-02-07 21:19:28,482 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:28,482 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2022-02-07 21:19:29,140 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:29,141 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2022-02-07 21:19:30,236 | test | DEBUG | MainThread | [rest_client:set_data_path:2276] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2022-02-07 21:19:30,237 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.121.126_8091 2022-02-07 21:19:30,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: rm -rf /data/* 2022-02-07 21:19:30,242 | infra | INFO | pool-3-thread-6 | [task:start_task:91] Thread 'node_init_task_172.23.121.126_8091' started 2022-02-07 21:19:30,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: chown -R couchbase:couchbase /data 2022-02-07 21:19:30,266 | infra | DEBUG | pool-3-thread-6 | [task:wait_until:135] Wait for expected condition to get satisfied 2022-02-07 21:19:30,267 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:30,269 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2022-02-07 21:19:30,283 | test | DEBUG | MainThread | [rest_client:set_data_path:2273] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2022-02-07 21:19:30,295 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:30,295 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2022-02-07 21:19:30,331 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:30,332 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2022-02-07 21:19:30,398 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:30,398 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2022-02-07 21:19:30,438 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:30,438 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2022-02-07 21:19:30,493 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:30,493 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2022-02-07 21:19:30,670 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:30,670 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2022-02-07 21:19:30,716 | test | DEBUG | pool-3-thread-4 | [task:call:6151] server: ip:172.23.121.123 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11527360512L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.121.123:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 9379, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.121.123', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.1.0-2255-enterprise', 'indexMemoryQuota': 512, 'uptime': u'178', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294119424L, 'memoryQuota': 6368, 'status': u'healthy'} 2022-02-07 21:19:30,717 | test | DEBUG | pool-3-thread-4 | [rest_client:set_service_mem_quota:945] Setting RAM quotas: {'memoryQuota': 8351} 2022-02-07 21:19:30,729 | test | DEBUG | pool-3-thread-4 | [rest_client:set_indexer_storage_mode:969] settings/indexes params: storageMode=plasma 2022-02-07 21:19:30,743 | test | DEBUG | pool-3-thread-4 | [rest_client:init_cluster:856] settings/web params on 172.23.121.123:8091:password=password&port=8091&username=Administrator 2022-02-07 21:19:31,005 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:31,005 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2022-02-07 21:19:31,658 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:31,660 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2022-02-07 21:19:32,549 | test | DEBUG | MainThread | [rest_client:set_data_path:2276] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2022-02-07 21:19:32,551 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.121.127_8091 2022-02-07 21:19:32,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: rm -rf /data/* 2022-02-07 21:19:32,552 | infra | INFO | pool-3-thread-7 | [task:start_task:91] Thread 'node_init_task_172.23.121.127_8091' started 2022-02-07 21:19:32,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: chown -R couchbase:couchbase /data 2022-02-07 21:19:32,575 | infra | DEBUG | pool-3-thread-7 | [task:wait_until:135] Wait for expected condition to get satisfied 2022-02-07 21:19:32,578 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:32,578 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2022-02-07 21:19:32,591 | test | DEBUG | MainThread | [rest_client:set_data_path:2273] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2022-02-07 21:19:32,601 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:32,601 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2022-02-07 21:19:32,634 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:32,634 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2022-02-07 21:19:32,687 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:32,687 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2022-02-07 21:19:32,785 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:32,785 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2022-02-07 21:19:32,953 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:32,954 | test | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2022-02-07 21:19:32,957 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:32,957 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2022-02-07 21:19:33,012 | test | DEBUG | pool-3-thread-5 | [task:call:6151] server: ip:172.23.121.124 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11584389120L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.121.124:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 9379, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.121.124', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.1.0-2255-enterprise', 'indexMemoryQuota': 512, 'uptime': u'168', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294119424L, 'memoryQuota': 6368, 'status': u'healthy'} 2022-02-07 21:19:33,013 | test | DEBUG | pool-3-thread-5 | [rest_client:set_service_mem_quota:945] Setting RAM quotas: {'memoryQuota': 8351} 2022-02-07 21:19:33,026 | test | DEBUG | pool-3-thread-5 | [rest_client:set_indexer_storage_mode:969] settings/indexes params: storageMode=plasma 2022-02-07 21:19:33,039 | test | DEBUG | pool-3-thread-5 | [rest_client:init_cluster:856] settings/web params on 172.23.121.124:8091:password=password&port=8091&username=Administrator 2022-02-07 21:19:33,289 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:33,289 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2022-02-07 21:19:33,944 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:33,946 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2022-02-07 21:19:35,128 | test | DEBUG | MainThread | [rest_client:set_data_path:2276] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2022-02-07 21:19:35,128 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.121.128_8091 2022-02-07 21:19:35,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: rm -rf /data/* 2022-02-07 21:19:35,131 | infra | INFO | pool-3-thread-8 | [task:start_task:91] Thread 'node_init_task_172.23.121.128_8091' started 2022-02-07 21:19:35,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: chown -R couchbase:couchbase /data 2022-02-07 21:19:35,151 | infra | DEBUG | pool-3-thread-8 | [task:wait_until:135] Wait for expected condition to get satisfied 2022-02-07 21:19:35,154 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:35,154 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2022-02-07 21:19:35,167 | test | DEBUG | MainThread | [rest_client:set_data_path:2273] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2022-02-07 21:19:35,177 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:35,177 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2022-02-07 21:19:35,210 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:35,210 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2022-02-07 21:19:35,239 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:35,239 | test | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2022-02-07 21:19:35,272 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:35,272 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2022-02-07 21:19:35,368 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:35,368 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2022-02-07 21:19:35,528 | test | DEBUG | pool-3-thread-6 | [task:call:6151] server: ip:172.23.121.126 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11535982592L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.121.126:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 9379, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.121.126', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.1.0-2255-enterprise', 'indexMemoryQuota': 512, 'uptime': u'159', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294119424L, 'memoryQuota': 6368, 'status': u'healthy'} 2022-02-07 21:19:35,529 | test | DEBUG | pool-3-thread-6 | [rest_client:set_service_mem_quota:945] Setting RAM quotas: {'memoryQuota': 8351} 2022-02-07 21:19:35,539 | test | DEBUG | pool-3-thread-6 | [rest_client:set_indexer_storage_mode:969] settings/indexes params: storageMode=plasma 2022-02-07 21:19:35,542 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:35,542 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2022-02-07 21:19:35,553 | test | DEBUG | pool-3-thread-6 | [rest_client:init_cluster:856] settings/web params on 172.23.121.126:8091:password=password&port=8091&username=Administrator 2022-02-07 21:19:35,881 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:35,881 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2022-02-07 21:19:36,536 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:36,536 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2022-02-07 21:19:37,427 | test | DEBUG | MainThread | [rest_client:set_data_path:2276] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2022-02-07 21:19:37,428 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.121.129_8091 2022-02-07 21:19:37,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: rm -rf /data/* 2022-02-07 21:19:37,430 | infra | INFO | pool-3-thread-9 | [task:start_task:91] Thread 'node_init_task_172.23.121.129_8091' started 2022-02-07 21:19:37,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: chown -R couchbase:couchbase /data 2022-02-07 21:19:37,454 | infra | DEBUG | pool-3-thread-9 | [task:wait_until:135] Wait for expected condition to get satisfied 2022-02-07 21:19:37,457 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:37,457 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2022-02-07 21:19:37,469 | test | DEBUG | MainThread | [rest_client:set_data_path:2273] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2022-02-07 21:19:37,483 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:37,483 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2022-02-07 21:19:37,516 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:37,516 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2022-02-07 21:19:37,571 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:37,571 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2022-02-07 21:19:37,673 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:37,674 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2022-02-07 21:19:37,809 | test | DEBUG | pool-3-thread-7 | [task:call:6151] server: ip:172.23.121.127 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11521613824L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.121.127:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 9379, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.121.127', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.1.0-2255-enterprise', 'indexMemoryQuota': 512, 'uptime': u'150', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294115328L, 'memoryQuota': 6368, 'status': u'healthy'} 2022-02-07 21:19:37,811 | test | DEBUG | pool-3-thread-7 | [rest_client:set_service_mem_quota:945] Setting RAM quotas: {'memoryQuota': 8351} 2022-02-07 21:19:37,822 | test | DEBUG | pool-3-thread-7 | [rest_client:set_indexer_storage_mode:969] settings/indexes params: storageMode=plasma 2022-02-07 21:19:37,831 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:37,832 | test | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2022-02-07 21:19:37,836 | test | DEBUG | pool-3-thread-7 | [rest_client:init_cluster:856] settings/web params on 172.23.121.127:8091:password=password&port=8091&username=Administrator 2022-02-07 21:19:37,848 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:37,849 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2022-02-07 21:19:38,184 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:38,184 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2022-02-07 21:19:38,841 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:38,842 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2022-02-07 21:19:39,730 | test | DEBUG | MainThread | [rest_client:set_data_path:2276] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2022-02-07 21:19:39,730 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.121.130_8091 2022-02-07 21:19:39,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: rm -rf /data/* 2022-02-07 21:19:39,733 | infra | INFO | pool-3-thread-10 | [task:start_task:91] Thread 'node_init_task_172.23.121.130_8091' started 2022-02-07 21:19:39,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: chown -R couchbase:couchbase /data 2022-02-07 21:19:39,759 | infra | DEBUG | pool-3-thread-10 | [task:wait_until:135] Wait for expected condition to get satisfied 2022-02-07 21:19:39,762 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:39,762 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2022-02-07 21:19:39,769 | test | DEBUG | MainThread | [rest_client:set_data_path:2273] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2022-02-07 21:19:39,785 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:39,785 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2022-02-07 21:19:39,819 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:39,819 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2022-02-07 21:19:39,872 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:39,872 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2022-02-07 21:19:39,967 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:39,969 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2022-02-07 21:19:40,137 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:40,138 | test | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2022-02-07 21:19:40,142 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:40,142 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2022-02-07 21:19:40,404 | test | DEBUG | pool-3-thread-8 | [task:call:6151] server: ip:172.23.121.128 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11525390336L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.121.128:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 9379, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.121.128', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.1.0-2255-enterprise', 'indexMemoryQuota': 512, 'uptime': u'140', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294119424L, 'memoryQuota': 6368, 'status': u'healthy'} 2022-02-07 21:19:40,404 | test | DEBUG | pool-3-thread-8 | [rest_client:set_service_mem_quota:945] Setting RAM quotas: {'memoryQuota': 8351} 2022-02-07 21:19:40,414 | test | DEBUG | pool-3-thread-8 | [rest_client:set_indexer_storage_mode:969] settings/indexes params: storageMode=plasma 2022-02-07 21:19:40,427 | test | DEBUG | pool-3-thread-8 | [rest_client:init_cluster:856] settings/web params on 172.23.121.128:8091:password=password&port=8091&username=Administrator 2022-02-07 21:19:40,474 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:40,476 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2022-02-07 21:19:41,131 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:41,131 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2022-02-07 21:19:42,223 | test | DEBUG | MainThread | [rest_client:set_data_path:2276] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2022-02-07 21:19:42,224 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.121.131_8091 2022-02-07 21:19:42,226 | infra | INFO | pool-3-thread-11 | [task:start_task:91] Thread 'node_init_task_172.23.121.131_8091' started 2022-02-07 21:19:42,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: rm -rf /data/* 2022-02-07 21:19:42,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: chown -R couchbase:couchbase /data 2022-02-07 21:19:42,253 | infra | DEBUG | pool-3-thread-11 | [task:wait_until:135] Wait for expected condition to get satisfied 2022-02-07 21:19:42,256 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:42,256 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2022-02-07 21:19:42,266 | test | DEBUG | MainThread | [rest_client:set_data_path:2273] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2022-02-07 21:19:42,282 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:42,282 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2022-02-07 21:19:42,321 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:42,321 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2022-02-07 21:19:42,384 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:42,385 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2022-02-07 21:19:42,434 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:42,434 | test | DEBUG | pool-3-thread-10 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2022-02-07 21:19:42,483 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:42,483 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2022-02-07 21:19:42,657 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:42,657 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2022-02-07 21:19:42,710 | test | DEBUG | pool-3-thread-9 | [task:call:6151] server: ip:172.23.121.129 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11530899456L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.121.129:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 9379, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.121.129', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.1.0-2255-enterprise', 'indexMemoryQuota': 512, 'uptime': u'131', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294119424L, 'memoryQuota': 6368, 'status': u'healthy'} 2022-02-07 21:19:42,711 | test | DEBUG | pool-3-thread-9 | [rest_client:set_service_mem_quota:945] Setting RAM quotas: {'memoryQuota': 8351} 2022-02-07 21:19:42,723 | test | DEBUG | pool-3-thread-9 | [rest_client:set_indexer_storage_mode:969] settings/indexes params: storageMode=plasma 2022-02-07 21:19:42,736 | test | DEBUG | pool-3-thread-9 | [rest_client:init_cluster:856] settings/web params on 172.23.121.129:8091:password=password&port=8091&username=Administrator 2022-02-07 21:19:42,996 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:42,996 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2022-02-07 21:19:43,650 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:43,650 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2022-02-07 21:19:44,546 | test | DEBUG | MainThread | [rest_client:set_data_path:2276] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2022-02-07 21:19:44,548 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.121.132_8091 2022-02-07 21:19:44,549 | infra | INFO | pool-3-thread-12 | [task:start_task:91] Thread 'node_init_task_172.23.121.132_8091' started 2022-02-07 21:19:44,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: rm -rf /data/* 2022-02-07 21:19:44,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: chown -R couchbase:couchbase /data 2022-02-07 21:19:44,569 | infra | DEBUG | pool-3-thread-12 | [task:wait_until:135] Wait for expected condition to get satisfied 2022-02-07 21:19:44,571 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:44,571 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2022-02-07 21:19:44,592 | test | DEBUG | MainThread | [rest_client:set_data_path:2273] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2022-02-07 21:19:44,595 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:44,595 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2022-02-07 21:19:44,631 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:44,631 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2022-02-07 21:19:44,687 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:44,687 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2022-02-07 21:19:44,790 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:44,792 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2022-02-07 21:19:44,946 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:44,946 | test | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2022-02-07 21:19:44,971 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:44,973 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2022-02-07 21:19:45,006 | test | DEBUG | pool-3-thread-10 | [task:call:6151] server: ip:172.23.121.130 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11562536960L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.121.130:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 9379, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.121.130', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.1.0-2255-enterprise', 'indexMemoryQuota': 512, 'uptime': u'122', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294119424L, 'memoryQuota': 6368, 'status': u'healthy'} 2022-02-07 21:19:45,009 | test | DEBUG | pool-3-thread-10 | [rest_client:set_service_mem_quota:945] Setting RAM quotas: {'memoryQuota': 8351} 2022-02-07 21:19:45,019 | test | DEBUG | pool-3-thread-10 | [rest_client:set_indexer_storage_mode:969] settings/indexes params: storageMode=plasma 2022-02-07 21:19:45,033 | test | DEBUG | pool-3-thread-10 | [rest_client:init_cluster:856] settings/web params on 172.23.121.130:8091:password=password&port=8091&username=Administrator 2022-02-07 21:19:45,305 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:45,305 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2022-02-07 21:19:45,961 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:45,961 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2022-02-07 21:19:46,848 | test | DEBUG | MainThread | [rest_client:set_data_path:2276] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2022-02-07 21:19:46,848 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.121.133_8091 2022-02-07 21:19:46,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: rm -rf /data/* 2022-02-07 21:19:46,849 | infra | INFO | pool-3-thread-13 | [task:start_task:91] Thread 'node_init_task_172.23.121.133_8091' started 2022-02-07 21:19:46,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: chown -R couchbase:couchbase /data 2022-02-07 21:19:46,869 | infra | DEBUG | pool-3-thread-13 | [task:wait_until:135] Wait for expected condition to get satisfied 2022-02-07 21:19:46,872 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:46,872 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2022-02-07 21:19:46,887 | test | DEBUG | MainThread | [rest_client:set_data_path:2273] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2022-02-07 21:19:46,894 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:46,894 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2022-02-07 21:19:46,930 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:46,931 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2022-02-07 21:19:46,986 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:46,986 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2022-02-07 21:19:47,078 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:47,078 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2022-02-07 21:19:47,255 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:47,255 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2022-02-07 21:19:47,256 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:47,256 | test | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2022-02-07 21:19:47,516 | test | DEBUG | pool-3-thread-11 | [task:call:6151] server: ip:172.23.121.131 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11541204992L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.121.131:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 9379, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.121.131', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.1.0-2255-enterprise', 'indexMemoryQuota': 512, 'uptime': u'113', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294119424L, 'memoryQuota': 6368, 'status': u'healthy'} 2022-02-07 21:19:47,517 | test | DEBUG | pool-3-thread-11 | [rest_client:set_service_mem_quota:945] Setting RAM quotas: {'memoryQuota': 8351} 2022-02-07 21:19:47,526 | test | DEBUG | pool-3-thread-11 | [rest_client:set_indexer_storage_mode:969] settings/indexes params: storageMode=plasma 2022-02-07 21:19:47,538 | test | DEBUG | pool-3-thread-11 | [rest_client:init_cluster:856] settings/web params on 172.23.121.131:8091:password=password&port=8091&username=Administrator 2022-02-07 21:19:47,586 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:47,588 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2022-02-07 21:19:48,244 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:48,244 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2022-02-07 21:19:49,134 | test | DEBUG | MainThread | [rest_client:set_data_path:2276] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2022-02-07 21:19:49,134 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.121.134_8091 2022-02-07 21:19:49,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: rm -rf /data/* 2022-02-07 21:19:49,135 | infra | INFO | pool-3-thread-14 | [task:start_task:91] Thread 'node_init_task_172.23.121.134_8091' started 2022-02-07 21:19:49,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: chown -R couchbase:couchbase /data 2022-02-07 21:19:49,160 | infra | DEBUG | pool-3-thread-14 | [task:wait_until:135] Wait for expected condition to get satisfied 2022-02-07 21:19:49,161 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:49,163 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2022-02-07 21:19:49,194 | test | DEBUG | MainThread | [rest_client:set_data_path:2273] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2022-02-07 21:19:49,230 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:49,230 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2022-02-07 21:19:49,265 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:49,266 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2022-02-07 21:19:49,319 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:49,319 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2022-02-07 21:19:49,415 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:49,417 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2022-02-07 21:19:49,539 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:49,539 | test | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2022-02-07 21:19:49,588 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:49,589 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2022-02-07 21:19:49,832 | test | DEBUG | pool-3-thread-12 | [task:call:6151] server: ip:172.23.121.132 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11567235072L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.121.132:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 9379, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.121.132', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.1.0-2255-enterprise', 'indexMemoryQuota': 512, 'uptime': u'103', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294119424L, 'memoryQuota': 6368, 'status': u'healthy'} 2022-02-07 21:19:49,832 | test | DEBUG | pool-3-thread-12 | [rest_client:set_service_mem_quota:945] Setting RAM quotas: {'memoryQuota': 8351} 2022-02-07 21:19:49,842 | test | DEBUG | pool-3-thread-12 | [rest_client:set_indexer_storage_mode:969] settings/indexes params: storageMode=plasma 2022-02-07 21:19:49,855 | test | DEBUG | pool-3-thread-12 | [rest_client:init_cluster:856] settings/web params on 172.23.121.132:8091:password=password&port=8091&username=Administrator 2022-02-07 21:19:49,926 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:49,927 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2022-02-07 21:19:50,581 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:50,582 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2022-02-07 21:19:51,440 | test | DEBUG | MainThread | [rest_client:set_data_path:2276] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2022-02-07 21:19:51,440 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.121.135_8091 2022-02-07 21:19:51,441 | infra | INFO | pool-3-thread-15 | [task:start_task:91] Thread 'node_init_task_172.23.121.135_8091' started 2022-02-07 21:19:51,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: rm -rf /data/* 2022-02-07 21:19:51,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: chown -R couchbase:couchbase /data 2022-02-07 21:19:51,463 | infra | DEBUG | pool-3-thread-15 | [task:wait_until:135] Wait for expected condition to get satisfied 2022-02-07 21:19:51,464 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:51,464 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2022-02-07 21:19:51,480 | test | DEBUG | MainThread | [rest_client:set_data_path:2273] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2022-02-07 21:19:51,486 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:51,486 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2022-02-07 21:19:51,516 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:51,516 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2022-02-07 21:19:51,571 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:51,572 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2022-02-07 21:19:51,664 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:51,664 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2022-02-07 21:19:51,836 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:51,838 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2022-02-07 21:19:51,875 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:51,875 | test | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2022-02-07 21:19:52,111 | test | DEBUG | pool-3-thread-13 | [task:call:6151] server: ip:172.23.121.133 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11552047104L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.121.133:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 9379, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.121.133', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.1.0-2255-enterprise', 'indexMemoryQuota': 512, 'uptime': u'94', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294119424L, 'memoryQuota': 6368, 'status': u'healthy'} 2022-02-07 21:19:52,111 | test | DEBUG | pool-3-thread-13 | [rest_client:set_service_mem_quota:945] Setting RAM quotas: {'memoryQuota': 8351} 2022-02-07 21:19:52,121 | test | DEBUG | pool-3-thread-13 | [rest_client:set_indexer_storage_mode:969] settings/indexes params: storageMode=plasma 2022-02-07 21:19:52,134 | test | DEBUG | pool-3-thread-13 | [rest_client:init_cluster:856] settings/web params on 172.23.121.133:8091:password=password&port=8091&username=Administrator 2022-02-07 21:19:52,171 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:52,171 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2022-02-07 21:19:52,828 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:52,828 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2022-02-07 21:19:53,766 | test | DEBUG | MainThread | [rest_client:set_data_path:2276] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2022-02-07 21:19:53,766 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.121.136_8091 2022-02-07 21:19:53,767 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.120.170_8091 2022-02-07 21:19:53,767 | infra | INFO | pool-3-thread-16 | [task:start_task:91] Thread 'node_init_task_172.23.121.136_8091' started 2022-02-07 21:19:53,767 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.121.115_8091 2022-02-07 21:19:53,769 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.121.116_8091 2022-02-07 21:19:53,769 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.121.123_8091 2022-02-07 21:19:53,769 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.121.124_8091 2022-02-07 21:19:53,769 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.121.126_8091 2022-02-07 21:19:53,769 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.121.127_8091 2022-02-07 21:19:53,769 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.121.128_8091 2022-02-07 21:19:53,769 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.121.129_8091 2022-02-07 21:19:53,769 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.121.130_8091 2022-02-07 21:19:53,770 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.121.131_8091 2022-02-07 21:19:53,770 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.121.132_8091 2022-02-07 21:19:53,770 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.121.133_8091 2022-02-07 21:19:53,770 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.121.134_8091 2022-02-07 21:19:53,788 | infra | DEBUG | pool-3-thread-16 | [task:wait_until:135] Wait for expected condition to get satisfied 2022-02-07 21:19:53,789 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:53,789 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2022-02-07 21:19:53,812 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:53,812 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2022-02-07 21:19:53,845 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:53,845 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2022-02-07 21:19:53,898 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:53,898 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2022-02-07 21:19:54,009 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:54,009 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2022-02-07 21:19:54,148 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:54,150 | test | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2022-02-07 21:19:54,181 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:54,181 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2022-02-07 21:19:54,447 | test | DEBUG | pool-3-thread-14 | [task:call:6151] server: ip:172.23.121.134 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11568246784L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.121.134:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 9379, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.121.134', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.1.0-2255-enterprise', 'indexMemoryQuota': 512, 'uptime': u'85', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294119424L, 'memoryQuota': 6368, 'status': u'healthy'} 2022-02-07 21:19:54,450 | test | DEBUG | pool-3-thread-14 | [rest_client:set_service_mem_quota:945] Setting RAM quotas: {'memoryQuota': 8351} 2022-02-07 21:19:54,460 | test | DEBUG | pool-3-thread-14 | [rest_client:set_indexer_storage_mode:969] settings/indexes params: storageMode=plasma 2022-02-07 21:19:54,473 | test | DEBUG | pool-3-thread-14 | [rest_client:init_cluster:856] settings/web params on 172.23.121.134:8091:password=password&port=8091&username=Administrator 2022-02-07 21:19:54,515 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:54,516 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2022-02-07 21:19:54,563 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.121.135_8091 2022-02-07 21:19:55,176 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:55,177 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2022-02-07 21:19:56,470 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L140 2022-02-07 21:19:56,470 | test | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2022-02-07 21:19:56,727 | test | DEBUG | pool-3-thread-15 | [task:call:6151] server: ip:172.23.121.135 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11526336512L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.121.135:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 9379, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.121.135', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.1.0-2255-enterprise', 'indexMemoryQuota': 512, 'uptime': u'76', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294119424L, 'memoryQuota': 6368, 'status': u'healthy'} 2022-02-07 21:19:56,729 | test | DEBUG | pool-3-thread-15 | [rest_client:set_service_mem_quota:945] Setting RAM quotas: {'memoryQuota': 8351} 2022-02-07 21:19:56,742 | test | DEBUG | pool-3-thread-15 | [rest_client:set_indexer_storage_mode:969] settings/indexes params: storageMode=plasma 2022-02-07 21:19:56,756 | test | DEBUG | pool-3-thread-15 | [rest_client:init_cluster:856] settings/web params on 172.23.121.135:8091:password=password&port=8091&username=Administrator 2022-02-07 21:19:56,842 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.121.136_8091 2022-02-07 21:19:59,042 | test | DEBUG | pool-3-thread-16 | [task:call:6151] server: ip:172.23.121.136 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11532541952L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.121.136:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 9379, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.121.136', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.1.0-2255-enterprise', 'indexMemoryQuota': 512, 'uptime': u'66', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294119424L, 'memoryQuota': 6368, 'status': u'healthy'} 2022-02-07 21:19:59,045 | test | DEBUG | pool-3-thread-16 | [rest_client:set_service_mem_quota:945] Setting RAM quotas: {'memoryQuota': 8351} 2022-02-07 21:19:59,055 | test | DEBUG | pool-3-thread-16 | [rest_client:set_indexer_storage_mode:969] settings/indexes params: storageMode=plasma 2022-02-07 21:19:59,069 | test | DEBUG | pool-3-thread-16 | [rest_client:init_cluster:856] settings/web params on 172.23.121.136:8091:password=password&port=8091&username=Administrator 2022-02-07 21:19:59,187 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:736] RAM quota was defined less than 100 MB: 2022-02-07 21:19:59,187 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:739] Node: 172.23.120.170: RAM: ['MemTotal: 12005972 kB\n', 'MemFree: 6763944 kB\n', 'MemAvailable: 11026392 kB\n', 'Buffers: 0 kB\n', 'Cached: 4391920 kB\n', 'SwapCached: 1568 kB\n', 'Active: 1507008 kB\n', 'Inactive: 3163100 kB\n', 'Active(anon): 273320 kB\n', 'Inactive(anon): 136416 kB\n', 'Active(file): 1233688 kB\n', 'Inactive(file): 3026684 kB\n', 'Unevictable: 9648 kB\n', 'Mlocked: 9648 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3491240 kB\n', 'Dirty: 64 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 286356 kB\n', 'Mapped: 114480 kB\n', 'Shmem: 127900 kB\n', 'Slab: 399752 kB\n', 'SReclaimable: 320648 kB\n', 'SUnreclaim: 79104 kB\n', 'KernelStack: 9088 kB\n', 'PageTables: 8088 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9672996 kB\n', 'Committed_AS: 1844976 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 77308 kB\n', 'VmallocChunk: 34359651324 kB\n', 'Percpu: 22528 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 0 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 124928 kB\n', 'DirectMap2M: 6053888 kB\n', 'DirectMap1G: 6291456 kB\n'] 2022-02-07 21:19:59,187 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:739] Node: 172.23.121.115: RAM: ['MemTotal: 12005976 kB\n', 'MemFree: 7014284 kB\n', 'MemAvailable: 11080772 kB\n', 'Buffers: 0 kB\n', 'Cached: 4237616 kB\n', 'SwapCached: 1640 kB\n', 'Active: 1420436 kB\n', 'Inactive: 3095940 kB\n', 'Active(anon): 271252 kB\n', 'Inactive(anon): 90364 kB\n', 'Active(file): 1149184 kB\n', 'Inactive(file): 3005576 kB\n', 'Unevictable: 9624 kB\n', 'Mlocked: 9624 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3555836 kB\n', 'Dirty: 32 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 286880 kB\n', 'Mapped: 114088 kB\n', 'Shmem: 79216 kB\n', 'Slab: 305176 kB\n', 'SReclaimable: 230244 kB\n', 'SUnreclaim: 74932 kB\n', 'KernelStack: 9056 kB\n', 'PageTables: 8256 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673000 kB\n', 'Committed_AS: 1727884 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 78332 kB\n', 'VmallocChunk: 34359651324 kB\n', 'Percpu: 22528 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 4096 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 104448 kB\n', 'DirectMap2M: 9220096 kB\n', 'DirectMap1G: 3145728 kB\n'] 2022-02-07 21:19:59,188 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:739] Node: 172.23.121.116: RAM: ['MemTotal: 12005976 kB\n', 'MemFree: 7009952 kB\n', 'MemAvailable: 11064956 kB\n', 'Buffers: 0 kB\n', 'Cached: 4258108 kB\n', 'SwapCached: 1284 kB\n', 'Active: 1419916 kB\n', 'Inactive: 3115904 kB\n', 'Active(anon): 270548 kB\n', 'Inactive(anon): 105832 kB\n', 'Active(file): 1149368 kB\n', 'Inactive(file): 3010072 kB\n', 'Unevictable: 8796 kB\n', 'Mlocked: 8796 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3560444 kB\n', 'Dirty: 20 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 285512 kB\n', 'Mapped: 115576 kB\n', 'Shmem: 95024 kB\n', 'Slab: 291172 kB\n', 'SReclaimable: 214080 kB\n', 'SUnreclaim: 77092 kB\n', 'KernelStack: 9056 kB\n', 'PageTables: 8472 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673000 kB\n', 'Committed_AS: 1723256 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 78332 kB\n', 'VmallocChunk: 34359651324 kB\n', 'Percpu: 22528 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 4096 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 137216 kB\n', 'DirectMap2M: 8138752 kB\n', 'DirectMap1G: 4194304 kB\n'] 2022-02-07 21:19:59,188 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:739] Node: 172.23.121.123: RAM: ['MemTotal: 12005976 kB\n', 'MemFree: 6997504 kB\n', 'MemAvailable: 11068280 kB\n', 'Buffers: 0 kB\n', 'Cached: 4259512 kB\n', 'SwapCached: 2036 kB\n', 'Active: 1413532 kB\n', 'Inactive: 3120280 kB\n', 'Active(anon): 270340 kB\n', 'Inactive(anon): 105624 kB\n', 'Active(file): 1143192 kB\n', 'Inactive(file): 3014656 kB\n', 'Unevictable: 8368 kB\n', 'Mlocked: 8368 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3565316 kB\n', 'Dirty: 52 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 282192 kB\n', 'Mapped: 115160 kB\n', 'Shmem: 98024 kB\n', 'Slab: 307056 kB\n', 'SReclaimable: 231444 kB\n', 'SUnreclaim: 75612 kB\n', 'KernelStack: 9056 kB\n', 'PageTables: 7724 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673000 kB\n', 'Committed_AS: 1720464 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 78332 kB\n', 'VmallocChunk: 34359651324 kB\n', 'Percpu: 22528 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 0 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 131072 kB\n', 'DirectMap2M: 8144896 kB\n', 'DirectMap1G: 4194304 kB\n'] 2022-02-07 21:19:59,188 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:739] Node: 172.23.121.124: RAM: ['MemTotal: 12005976 kB\n', 'MemFree: 7035656 kB\n', 'MemAvailable: 11054068 kB\n', 'Buffers: 0 kB\n', 'Cached: 4276808 kB\n', 'SwapCached: 2576 kB\n', 'Active: 1416516 kB\n', 'Inactive: 3135600 kB\n', 'Active(anon): 270868 kB\n', 'Inactive(anon): 121680 kB\n', 'Active(file): 1145648 kB\n', 'Inactive(file): 3013920 kB\n', 'Unevictable: 9396 kB\n', 'Mlocked: 9396 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3524496 kB\n', 'Dirty: 52 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 282996 kB\n', 'Mapped: 128964 kB\n', 'Shmem: 113600 kB\n', 'Slab: 248704 kB\n', 'SReclaimable: 177360 kB\n', 'SUnreclaim: 71344 kB\n', 'KernelStack: 9024 kB\n', 'PageTables: 7580 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673000 kB\n', 'Committed_AS: 1771972 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 77308 kB\n', 'VmallocChunk: 34359651324 kB\n', 'Percpu: 22528 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 0 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 110592 kB\n', 'DirectMap2M: 7116800 kB\n', 'DirectMap1G: 5242880 kB\n'] 2022-02-07 21:19:59,190 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:739] Node: 172.23.121.126: RAM: ['MemTotal: 12005976 kB\n', 'MemFree: 7009800 kB\n', 'MemAvailable: 11058472 kB\n', 'Buffers: 0 kB\n', 'Cached: 4254620 kB\n', 'SwapCached: 984 kB\n', 'Active: 1418220 kB\n', 'Inactive: 3125156 kB\n', 'Active(anon): 270284 kB\n', 'Inactive(anon): 114536 kB\n', 'Active(file): 1147936 kB\n', 'Inactive(file): 3010620 kB\n', 'Unevictable: 7784 kB\n', 'Mlocked: 7784 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3572988 kB\n', 'Dirty: 40 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 295764 kB\n', 'Mapped: 115456 kB\n', 'Shmem: 92424 kB\n', 'Slab: 284492 kB\n', 'SReclaimable: 208632 kB\n', 'SUnreclaim: 75860 kB\n', 'KernelStack: 9024 kB\n', 'PageTables: 8280 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673000 kB\n', 'Committed_AS: 1723636 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 77308 kB\n', 'VmallocChunk: 34359651324 kB\n', 'Percpu: 22528 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 4096 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 122880 kB\n', 'DirectMap2M: 7104512 kB\n', 'DirectMap1G: 5242880 kB\n'] 2022-02-07 21:19:59,190 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:739] Node: 172.23.121.127: RAM: ['MemTotal: 12005972 kB\n', 'MemFree: 7003892 kB\n', 'MemAvailable: 11081908 kB\n', 'Buffers: 0 kB\n', 'Cached: 4247148 kB\n', 'SwapCached: 1276 kB\n', 'Active: 1408380 kB\n', 'Inactive: 3113268 kB\n', 'Active(anon): 259360 kB\n', 'Inactive(anon): 102136 kB\n', 'Active(file): 1149020 kB\n', 'Inactive(file): 3011132 kB\n', 'Unevictable: 8804 kB\n', 'Mlocked: 8804 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3573072 kB\n', 'Dirty: 28 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 282700 kB\n', 'Mapped: 115940 kB\n', 'Shmem: 83356 kB\n', 'Slab: 312724 kB\n', 'SReclaimable: 236440 kB\n', 'SUnreclaim: 76284 kB\n', 'KernelStack: 8992 kB\n', 'PageTables: 8092 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9672996 kB\n', 'Committed_AS: 1699036 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 77308 kB\n', 'VmallocChunk: 34359651324 kB\n', 'Percpu: 22528 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 6144 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 114688 kB\n', 'DirectMap2M: 5015552 kB\n', 'DirectMap1G: 7340032 kB\n'] 2022-02-07 21:19:59,190 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:739] Node: 172.23.121.128: RAM: ['MemTotal: 12005976 kB\n', 'MemFree: 6994480 kB\n', 'MemAvailable: 11070036 kB\n', 'Buffers: 0 kB\n', 'Cached: 4260224 kB\n', 'SwapCached: 1604 kB\n', 'Active: 1416996 kB\n', 'Inactive: 3121276 kB\n', 'Active(anon): 263992 kB\n', 'Inactive(anon): 109496 kB\n', 'Active(file): 1153004 kB\n', 'Inactive(file): 3011780 kB\n', 'Unevictable: 8820 kB\n', 'Mlocked: 8820 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3530748 kB\n', 'Dirty: 44 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 285704 kB\n', 'Mapped: 119844 kB\n', 'Shmem: 91800 kB\n', 'Slab: 304356 kB\n', 'SReclaimable: 229288 kB\n', 'SUnreclaim: 75068 kB\n', 'KernelStack: 9008 kB\n', 'PageTables: 8140 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673000 kB\n', 'Committed_AS: 1759172 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 78332 kB\n', 'VmallocChunk: 34359651324 kB\n', 'Percpu: 22528 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 2048 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 120832 kB\n', 'DirectMap2M: 7106560 kB\n', 'DirectMap1G: 5242880 kB\n'] 2022-02-07 21:19:59,190 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:739] Node: 172.23.121.129: RAM: ['MemTotal: 12005976 kB\n', 'MemFree: 7018900 kB\n', 'MemAvailable: 11066892 kB\n', 'Buffers: 0 kB\n', 'Cached: 4242156 kB\n', 'SwapCached: 604 kB\n', 'Active: 1425664 kB\n', 'Inactive: 3100344 kB\n', 'Active(anon): 268868 kB\n', 'Inactive(anon): 109116 kB\n', 'Active(file): 1156796 kB\n', 'Inactive(file): 2991228 kB\n', 'Unevictable: 8660 kB\n', 'Mlocked: 8660 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3589180 kB\n', 'Dirty: 36 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 292400 kB\n', 'Mapped: 111748 kB\n', 'Shmem: 90488 kB\n', 'Slab: 293912 kB\n', 'SReclaimable: 218484 kB\n', 'SUnreclaim: 75428 kB\n', 'KernelStack: 9024 kB\n', 'PageTables: 8020 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673000 kB\n', 'Committed_AS: 1706704 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 77308 kB\n', 'VmallocChunk: 34359651324 kB\n', 'Percpu: 22528 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 4096 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 124928 kB\n', 'DirectMap2M: 8151040 kB\n', 'DirectMap1G: 4194304 kB\n'] 2022-02-07 21:19:59,190 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:739] Node: 172.23.121.130: RAM: ['MemTotal: 12005976 kB\n', 'MemFree: 6868368 kB\n', 'MemAvailable: 11053644 kB\n', 'Buffers: 0 kB\n', 'Cached: 4425120 kB\n', 'SwapCached: 1044 kB\n', 'Active: 1479368 kB\n', 'Inactive: 3222504 kB\n', 'Active(anon): 276688 kB\n', 'Inactive(anon): 119196 kB\n', 'Active(file): 1202680 kB\n', 'Inactive(file): 3103308 kB\n', 'Unevictable: 8792 kB\n', 'Mlocked: 8792 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3622428 kB\n', 'Dirty: 60 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 284984 kB\n', 'Mapped: 109864 kB\n', 'Shmem: 115492 kB\n', 'Slab: 268256 kB\n', 'SReclaimable: 197804 kB\n', 'SUnreclaim: 70452 kB\n', 'KernelStack: 9024 kB\n', 'PageTables: 8304 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673000 kB\n', 'Committed_AS: 1682796 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 77308 kB\n', 'VmallocChunk: 34359651324 kB\n', 'Percpu: 22528 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 4096 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 120832 kB\n', 'DirectMap2M: 8155136 kB\n', 'DirectMap1G: 4194304 kB\n'] 2022-02-07 21:19:59,191 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:739] Node: 172.23.121.131: RAM: ['MemTotal: 12005976 kB\n', 'MemFree: 6973100 kB\n', 'MemAvailable: 11041896 kB\n', 'Buffers: 0 kB\n', 'Cached: 4297420 kB\n', 'SwapCached: 496 kB\n', 'Active: 1485836 kB\n', 'Inactive: 3095728 kB\n', 'Active(anon): 267100 kB\n', 'Inactive(anon): 139216 kB\n', 'Active(file): 1218736 kB\n', 'Inactive(file): 2956512 kB\n', 'Unevictable: 8956 kB\n', 'Mlocked: 8956 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3629564 kB\n', 'Dirty: 52 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 292444 kB\n', 'Mapped: 108504 kB\n', 'Shmem: 118532 kB\n', 'Slab: 283520 kB\n', 'SReclaimable: 212064 kB\n', 'SUnreclaim: 71456 kB\n', 'KernelStack: 9008 kB\n', 'PageTables: 8072 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673000 kB\n', 'Committed_AS: 1681480 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 78332 kB\n', 'VmallocChunk: 34359651324 kB\n', 'Percpu: 22528 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 6144 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 114688 kB\n', 'DirectMap2M: 8161280 kB\n', 'DirectMap1G: 4194304 kB\n'] 2022-02-07 21:19:59,191 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:739] Node: 172.23.121.132: RAM: ['MemTotal: 12005976 kB\n', 'MemFree: 6864968 kB\n', 'MemAvailable: 11047428 kB\n', 'Buffers: 0 kB\n', 'Cached: 4430080 kB\n', 'SwapCached: 408 kB\n', 'Active: 1483684 kB\n', 'Inactive: 3217248 kB\n', 'Active(anon): 271768 kB\n', 'Inactive(anon): 123880 kB\n', 'Active(file): 1211916 kB\n', 'Inactive(file): 3093368 kB\n', 'Unevictable: 8960 kB\n', 'Mlocked: 8960 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3632380 kB\n', 'Dirty: 60 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 279132 kB\n', 'Mapped: 109436 kB\n', 'Shmem: 121144 kB\n', 'Slab: 268424 kB\n', 'SReclaimable: 195692 kB\n', 'SUnreclaim: 72732 kB\n', 'KernelStack: 9056 kB\n', 'PageTables: 8008 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673000 kB\n', 'Committed_AS: 1701416 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 78332 kB\n', 'VmallocChunk: 34359651324 kB\n', 'Percpu: 23552 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 2048 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 145408 kB\n', 'DirectMap2M: 7081984 kB\n', 'DirectMap1G: 5242880 kB\n'] 2022-02-07 21:19:59,191 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:739] Node: 172.23.121.133: RAM: ['MemTotal: 12005976 kB\n', 'MemFree: 7005320 kB\n', 'MemAvailable: 11052576 kB\n', 'Buffers: 0 kB\n', 'Cached: 4279900 kB\n', 'SwapCached: 356 kB\n', 'Active: 1460140 kB\n', 'Inactive: 3091508 kB\n', 'Active(anon): 260960 kB\n', 'Inactive(anon): 134596 kB\n', 'Active(file): 1199180 kB\n', 'Inactive(file): 2956912 kB\n', 'Unevictable: 8960 kB\n', 'Mlocked: 8960 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3628284 kB\n', 'Dirty: 28 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 280108 kB\n', 'Mapped: 108420 kB\n', 'Shmem: 120168 kB\n', 'Slab: 280444 kB\n', 'SReclaimable: 209680 kB\n', 'SUnreclaim: 70764 kB\n', 'KernelStack: 8960 kB\n', 'PageTables: 7672 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673000 kB\n', 'Committed_AS: 1693272 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 77308 kB\n', 'VmallocChunk: 34359651324 kB\n', 'Percpu: 22528 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 2048 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 110592 kB\n', 'DirectMap2M: 7116800 kB\n', 'DirectMap1G: 5242880 kB\n'] 2022-02-07 21:19:59,193 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:739] Node: 172.23.121.134: RAM: ['MemTotal: 12005976 kB\n', 'MemFree: 6987548 kB\n', 'MemAvailable: 11054192 kB\n', 'Buffers: 0 kB\n', 'Cached: 4312028 kB\n', 'SwapCached: 560 kB\n', 'Active: 1474452 kB\n', 'Inactive: 3115156 kB\n', 'Active(anon): 262072 kB\n', 'Inactive(anon): 132888 kB\n', 'Active(file): 1212380 kB\n', 'Inactive(file): 2982268 kB\n', 'Unevictable: 8404 kB\n', 'Mlocked: 8404 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3625724 kB\n', 'Dirty: 16 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 285596 kB\n', 'Mapped: 109284 kB\n', 'Shmem: 113740 kB\n', 'Slab: 261796 kB\n', 'SReclaimable: 190512 kB\n', 'SUnreclaim: 71284 kB\n', 'KernelStack: 9008 kB\n', 'PageTables: 7804 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673000 kB\n', 'Committed_AS: 1689820 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 77308 kB\n', 'VmallocChunk: 34359651324 kB\n', 'Percpu: 22528 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 6144 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 112640 kB\n', 'DirectMap2M: 7114752 kB\n', 'DirectMap1G: 5242880 kB\n'] 2022-02-07 21:19:59,193 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:739] Node: 172.23.121.135: RAM: ['MemTotal: 12005976 kB\n', 'MemFree: 6994256 kB\n', 'MemAvailable: 11060536 kB\n', 'Buffers: 0 kB\n', 'Cached: 4263304 kB\n', 'SwapCached: 1336 kB\n', 'Active: 1419408 kB\n', 'Inactive: 3121344 kB\n', 'Active(anon): 263236 kB\n', 'Inactive(anon): 117392 kB\n', 'Active(file): 1156172 kB\n', 'Inactive(file): 3003952 kB\n', 'Unevictable: 8804 kB\n', 'Mlocked: 8804 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3542268 kB\n', 'Dirty: 208 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 286080 kB\n', 'Mapped: 124304 kB\n', 'Shmem: 99540 kB\n', 'Slab: 301888 kB\n', 'SReclaimable: 224672 kB\n', 'SUnreclaim: 77216 kB\n', 'KernelStack: 9040 kB\n', 'PageTables: 8196 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673000 kB\n', 'Committed_AS: 1760056 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 77308 kB\n', 'VmallocChunk: 34359651324 kB\n', 'Percpu: 22528 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 2048 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 116736 kB\n', 'DirectMap2M: 7110656 kB\n', 'DirectMap1G: 5242880 kB\n'] 2022-02-07 21:19:59,193 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:739] Node: 172.23.121.136: RAM: ['MemTotal: 12005976 kB\n', 'MemFree: 7019584 kB\n', 'MemAvailable: 11072564 kB\n', 'Buffers: 0 kB\n', 'Cached: 4228912 kB\n', 'SwapCached: 1036 kB\n', 'Active: 1425828 kB\n', 'Inactive: 3099904 kB\n', 'Active(anon): 284316 kB\n', 'Inactive(anon): 84464 kB\n', 'Active(file): 1141512 kB\n', 'Inactive(file): 3015440 kB\n', 'Unevictable: 9812 kB\n', 'Mlocked: 9812 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3570428 kB\n', 'Dirty: 560 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 305480 kB\n', 'Mapped: 112892 kB\n', 'Shmem: 68320 kB\n', 'Slab: 291460 kB\n', 'SReclaimable: 214544 kB\n', 'SUnreclaim: 76916 kB\n', 'KernelStack: 9024 kB\n', 'PageTables: 7864 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673000 kB\n', 'Committed_AS: 1748124 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 77308 kB\n', 'VmallocChunk: 34359651324 kB\n', 'Percpu: 22528 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 2048 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 126976 kB\n', 'DirectMap2M: 6051840 kB\n', 'DirectMap1G: 6291456 kB\n'] 2022-02-07 21:19:59,194 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 21:19:59,198 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 21:19:59,198 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 21:19:59,200 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 21:19:59,200 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 21:19:59,200 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 21:19:59,200 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 21:19:59,201 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 21:19:59,201 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 21:19:59,203 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-07 21:19:59,203 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-07 21:19:59,203 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-07 21:19:59,203 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-07 21:19:59,203 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-07 21:19:59,204 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-07 21:19:59,204 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-07 21:19:59,209 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 21:19:59,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:59,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 21:19:59,545 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:59,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 21:19:59,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 21:19:59,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 21:19:59,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 21:19:59,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 21:19:59,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 21:19:59,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 21:19:59,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2022-02-07 21:19:59,664 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.120.170 port:8091 ssh_username:root: [] 2022-02-07 21:19:59,664 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 21:19:59,664 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2022-02-07 21:19:59,668 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 21:19:59,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:19:59,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 21:19:59,966 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:19:59,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 21:19:59,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 21:19:59,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 21:20:00,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 21:20:00,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 21:20:00,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 21:20:00,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 21:20:00,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2022-02-07 21:20:00,088 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.121.115 port:8091 ssh_username:root: [] 2022-02-07 21:20:00,088 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 21:20:00,088 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2022-02-07 21:20:00,094 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 21:20:00,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:20:00,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 21:20:00,441 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:20:00,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 21:20:00,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 21:20:00,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 21:20:00,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 21:20:00,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 21:20:00,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 21:20:00,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 21:20:00,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2022-02-07 21:20:00,556 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.121.116 port:8091 ssh_username:root: [] 2022-02-07 21:20:00,558 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 21:20:00,558 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2022-02-07 21:20:00,562 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 21:20:00,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:20:00,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 21:20:00,900 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:20:00,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 21:20:00,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 21:20:00,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 21:20:00,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 21:20:00,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 21:20:00,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 21:20:00,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 21:20:00,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2022-02-07 21:20:01,010 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.121.123 port:8091 ssh_username:root: [] 2022-02-07 21:20:01,010 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 21:20:01,010 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2022-02-07 21:20:01,015 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 21:20:01,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:20:01,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 21:20:01,367 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:20:01,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 21:20:01,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 21:20:01,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 21:20:01,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 21:20:01,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 21:20:01,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 21:20:01,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 21:20:01,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2022-02-07 21:20:01,490 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.121.124 port:8091 ssh_username:root: [] 2022-02-07 21:20:01,490 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 21:20:01,490 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2022-02-07 21:20:01,493 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 21:20:01,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:20:01,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 21:20:01,832 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:20:01,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 21:20:01,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 21:20:01,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 21:20:01,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 21:20:01,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 21:20:01,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 21:20:01,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 21:20:01,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2022-02-07 21:20:01,934 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.121.126 port:8091 ssh_username:root: [] 2022-02-07 21:20:01,934 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 21:20:01,934 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2022-02-07 21:20:01,938 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 21:20:02,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:20:02,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 21:20:02,270 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:20:02,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 21:20:02,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 21:20:02,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 21:20:02,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 21:20:02,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 21:20:02,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 21:20:02,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 21:20:02,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2022-02-07 21:20:02,382 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.121.127 port:8091 ssh_username:root: [] 2022-02-07 21:20:02,384 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 21:20:02,384 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2022-02-07 21:20:02,388 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 21:20:02,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:20:02,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 21:20:02,717 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:20:02,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 21:20:02,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 21:20:02,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 21:20:02,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 21:20:02,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 21:20:02,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 21:20:02,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 21:20:02,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2022-02-07 21:20:02,818 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.121.128 port:8091 ssh_username:root: [] 2022-02-07 21:20:02,819 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 21:20:02,819 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2022-02-07 21:20:02,823 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 21:20:02,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:20:03,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 21:20:03,163 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:20:03,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 21:20:03,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 21:20:03,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 21:20:03,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 21:20:03,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 21:20:03,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 21:20:03,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 21:20:03,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2022-02-07 21:20:03,267 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.121.129 port:8091 ssh_username:root: [] 2022-02-07 21:20:03,267 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 21:20:03,269 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2022-02-07 21:20:03,273 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-07 21:20:03,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:20:03,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-07 21:20:03,631 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:20:03,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-07 21:20:03,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-07 21:20:03,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-07 21:20:03,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-07 21:20:03,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-07 21:20:03,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-07 21:20:03,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-07 21:20:03,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2022-02-07 21:20:03,733 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.121.130 port:8091 ssh_username:root: [] 2022-02-07 21:20:03,733 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-07 21:20:03,734 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2022-02-07 21:20:03,739 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-07 21:20:03,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:20:04,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-07 21:20:04,085 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:20:04,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-07 21:20:04,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-07 21:20:04,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-07 21:20:04,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-07 21:20:04,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-07 21:20:04,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-07 21:20:04,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-07 21:20:04,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2022-02-07 21:20:04,193 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.121.131 port:8091 ssh_username:root: [] 2022-02-07 21:20:04,193 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-07 21:20:04,194 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2022-02-07 21:20:04,197 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-07 21:20:04,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:20:04,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-07 21:20:04,539 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:20:04,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-07 21:20:04,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-07 21:20:04,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-07 21:20:04,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-07 21:20:04,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-07 21:20:04,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-07 21:20:04,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-07 21:20:04,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2022-02-07 21:20:04,642 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.121.132 port:8091 ssh_username:root: [] 2022-02-07 21:20:04,644 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-07 21:20:04,644 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2022-02-07 21:20:04,648 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-07 21:20:04,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:20:04,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-07 21:20:05,030 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:20:05,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-07 21:20:05,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-07 21:20:05,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-07 21:20:05,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-07 21:20:05,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-07 21:20:05,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-07 21:20:05,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-07 21:20:05,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2022-02-07 21:20:05,130 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.121.133 port:8091 ssh_username:root: [] 2022-02-07 21:20:05,131 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-07 21:20:05,131 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2022-02-07 21:20:05,134 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-07 21:20:05,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:20:05,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-07 21:20:05,470 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:20:05,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-07 21:20:05,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-07 21:20:05,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-07 21:20:05,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-07 21:20:05,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-07 21:20:05,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-07 21:20:05,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-07 21:20:05,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2022-02-07 21:20:05,572 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.121.134 port:8091 ssh_username:root: [] 2022-02-07 21:20:05,572 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-07 21:20:05,572 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2022-02-07 21:20:05,575 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-07 21:20:05,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:20:05,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-07 21:20:05,950 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:20:05,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-07 21:20:05,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-07 21:20:05,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-07 21:20:05,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-07 21:20:06,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-07 21:20:06,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-07 21:20:06,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-07 21:20:06,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2022-02-07 21:20:06,059 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.121.135 port:8091 ssh_username:root: [] 2022-02-07 21:20:06,059 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-07 21:20:06,061 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2022-02-07 21:20:06,065 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-07 21:20:06,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:20:06,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-07 21:20:06,421 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:20:06,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-07 21:20:06,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-07 21:20:06,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-07 21:20:06,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-07 21:20:06,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-07 21:20:06,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-07 21:20:06,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-07 21:20:06,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2022-02-07 21:20:06,523 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.121.136 port:8091 ssh_username:root: [] 2022-02-07 21:20:06,523 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-07 21:20:06,523 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2022-02-07 21:20:06,525 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:361] Wait for all the services to come up after change_env_vars update 2022-02-07 21:20:06,532 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> change_env_variables():L363 2022-02-07 21:20:06,532 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-07 21:20:16,533 | test | INFO | MainThread | [basetestcase:initialize_cluster:458] Cluster C1 initialized 2022-02-07 21:20:16,533 | test | INFO | MainThread | [basetestcase:log_setup_status:684] ========= BaseTestCase setup finished for test #1 SystemTestMagma ========= ############################################################ # # Step 1: Create a 9 node cluster # ############################################################ 2022-02-07 21:20:16,592 | infra | DEBUG | MainThread | [task:__init__:249] Last known rebalance status_id: None 2022-02-07 21:20:16,594 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[172.23.121.115,172.23.121.116,172.23.121.123,172.23.121.124,172.23.121.126,172.23.121.127,172.23.121.128,172.23.121.129]_OUT=[]_1644297616.57 2022-02-07 21:20:16,595 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[172.23.121.115,172.23.121.116,172.23.121.123,172.23.121.124,172.23.121.126,172.23.121.127,172.23.121.128,172.23.121.129]_OUT=[]_1644297616.57 2022-02-07 21:20:16,595 | infra | INFO | pool-3-thread-17 | [task:start_task:91] Thread 'Rebalance_task_IN=[172.23.121.115,172.23.121.116,172.23.121.123,172.23.121.124,172.23.121.126,172.23.121.127,172.23.121.128,172.23.121.129]_OUT=[]_1644297616.57' started 2022-02-07 21:20:16,596 | test | DEBUG | pool-3-thread-17 | [rest_client:add_node:1199] Adding remote node 172.23.121.115:8091 to cluster 172.23.120.170:8091 2022-02-07 21:20:27,520 | test | DEBUG | pool-3-thread-17 | [rest_client:add_node:1199] Adding remote node 172.23.121.116:8091 to cluster 172.23.120.170:8091 2022-02-07 21:20:30,828 | test | DEBUG | pool-3-thread-17 | [rest_client:add_node:1199] Adding remote node 172.23.121.123:8091 to cluster 172.23.120.170:8091 2022-02-07 21:20:34,125 | test | DEBUG | pool-3-thread-17 | [rest_client:add_node:1199] Adding remote node 172.23.121.124:8091 to cluster 172.23.120.170:8091 2022-02-07 21:20:37,309 | test | DEBUG | pool-3-thread-17 | [rest_client:add_node:1199] Adding remote node 172.23.121.126:8091 to cluster 172.23.120.170:8091 2022-02-07 21:20:40,607 | test | DEBUG | pool-3-thread-17 | [rest_client:add_node:1199] Adding remote node 172.23.121.127:8091 to cluster 172.23.120.170:8091 2022-02-07 21:20:44,183 | test | DEBUG | pool-3-thread-17 | [rest_client:add_node:1199] Adding remote node 172.23.121.128:8091 to cluster 172.23.120.170:8091 2022-02-07 21:20:47,453 | test | DEBUG | pool-3-thread-17 | [rest_client:add_node:1199] Adding remote node 172.23.121.129:8091 to cluster 172.23.120.170:8091 2022-02-07 21:20:50,602 | test | DEBUG | pool-3-thread-17 | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.128,ns_1@172.23.121.115,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123', 'ejectedNodes': '', 'user': 'Administrator'} 2022-02-07 21:20:50,664 | test | DEBUG | pool-3-thread-17 | [rest_client:rebalance:1459] Rebalance operation started 2022-02-07 21:20:50,667 | test | INFO | pool-3-thread-17 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.120.170 | kv | 7.1.0-2255-enterprise | 1.06436263461 | Cluster node | | 172.23.121.115 | None | | | <--- IN --- | | 172.23.121.116 | None | | | <--- IN --- | | 172.23.121.123 | None | | | <--- IN --- | | 172.23.121.124 | None | | | <--- IN --- | | 172.23.121.126 | None | | | <--- IN --- | | 172.23.121.127 | None | | | <--- IN --- | | 172.23.121.128 | None | | | <--- IN --- | | 172.23.121.129 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2022-02-07 21:20:50,677 | infra | DEBUG | pool-3-thread-17 | [task:call:316] New rebalance status_id: b059a148dbf67828241d035a18f5973d 2022-02-07 21:20:50,687 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-07 21:20:50,687 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-07 21:20:50,707 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-07 21:20:50,710 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-07 21:20:50,710 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-07 21:21:00,723 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-07 21:21:00,723 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:21:05,733 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: none, progress: 100 2022-02-07 21:21:05,769 | test | INFO | pool-3-thread-17 | [task:check:473] Rebalance completed with progress: 100% in 15.1019999981 sec 2022-02-07 21:21:05,769 | infra | INFO | pool-3-thread-17 | [task:complete_task:106] Thread 'Rebalance_task_IN=[172.23.121.115,172.23.121.116,172.23.121.123,172.23.121.124,172.23.121.126,172.23.121.127,172.23.121.128,172.23.121.129]_OUT=[]_1644297616.57' completed 2022-02-07 21:21:05,832 | test | INFO | MainThread | [rest_client:set_metadata_purge_interval:2181] Successfully set Metadata Purge Interval to 0.04 ############################################################ # # Step 2: Create required buckets and collections. # ############################################################ 2022-02-07 21:21:05,832 | test | INFO | MainThread | [opd:create_required_buckets:49] Get the available memory quota 2022-02-07 21:21:05,861 | test | INFO | MainThread | [opd:create_required_buckets:57] Create CB buckets 2022-02-07 21:21:05,861 | test | DEBUG | MainThread | [bucket_ready_functions:async_create_bucket:1569] Creating bucket: GleamBookUsers0 2022-02-07 21:21:05,862 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: bucket_GleamBookUsers0_create_task 2022-02-07 21:21:05,864 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for bucket_GleamBookUsers0_create_task 2022-02-07 21:21:05,931 | infra | INFO | pool-3-thread-18 | [BucketOperations_Rest:create_bucket:344] Creating 'couchbase' bucket GleamBookUsers0 2022-02-07 21:21:05,931 | infra | DEBUG | pool-3-thread-18 | [BucketOperations_Rest:create_bucket:346] http://172.23.120.170:8091/pools/default/buckets with param: storageBackend=magma&viewFragmentationThreshold%255Bpercentage%255D=50&magmaFragmentationPercentage=50&replicaIndex=1&maxTTL=0&indexCompactionMode=circular&databaseFragmentationThreshold%255Bpercentage%255D=50&flushEnabled=1¶llelDBAndViewCompaction=false&durabilityMinLevel=none&compressionMode=passive&purgeInterval=3&bucketType=couchbase&autoCompactionDefined=false&conflictResolutionType=seqno&name=GleamBookUsers0&replicaNumber=1&ramQuotaMB=8251&evictionPolicy=fullEviction 2022-02-07 21:21:05,967 | infra | DEBUG | pool-3-thread-18 | [BucketOperations_Rest:create_bucket:354] 0.03 seconds to create bucket GleamBookUsers0 2022-02-07 21:21:05,967 | infra | INFO | pool-3-thread-18 | [task:complete_task:106] Thread 'bucket_GleamBookUsers0_create_task' completed 2022-02-07 21:21:05,970 | test | DEBUG | MainThread | [bucket_ready_functions:create_bucket:1581] Wait for memcached to accept cbstats or any other bucket request connections 2022-02-07 21:21:06,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> create_bucket():L1583 2022-02-07 21:21:06,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-07 21:21:08,068 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:651] All nodes in cluster: [ip:172.23.121.128 port:8091 ssh_username:root, ip:172.23.121.115 port:8091 ssh_username:root, ip:172.23.120.170 port:8091 ssh_username:root, ip:172.23.121.129 port:8091 ssh_username:root, ip:172.23.121.124 port:8091 ssh_username:root, ip:172.23.121.126 port:8091 ssh_username:root, ip:172.23.121.116 port:8091 ssh_username:root, ip:172.23.121.127 port:8091 ssh_username:root, ip:172.23.121.123 port:8091 ssh_username:root] 2022-02-07 21:21:08,069 | test | DEBUG | MainThread | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-07 21:21:08,073 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 21:21:08,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:21:08,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 21:21:08,428 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:21:08,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 21:21:08,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 21:21:08,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 21:21:08,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 21:21:08,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 21:21:08,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 21:21:08,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 21:21:08,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:21:08,671 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 21:21:08,676 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 21:21:08,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:21:08,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 21:21:09,032 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:21:09,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 21:21:09,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 21:21:09,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 21:21:09,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 21:21:09,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 21:21:09,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 21:21:09,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 21:21:09,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:21:09,265 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 21:21:09,267 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 21:21:09,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:21:09,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 21:21:09,627 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:21:09,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 21:21:09,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 21:21:09,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 21:21:09,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 21:21:09,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 21:21:09,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 21:21:09,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 21:21:09,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:21:09,862 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 21:21:09,865 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 21:21:09,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:21:10,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 21:21:10,224 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:21:10,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 21:21:10,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 21:21:10,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 21:21:10,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 21:21:10,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 21:21:10,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 21:21:10,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 21:21:10,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:21:10,473 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 21:21:10,477 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 21:21:10,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:21:10,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 21:21:10,857 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:21:10,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 21:21:10,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 21:21:10,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 21:21:10,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 21:21:10,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 21:21:10,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 21:21:10,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 21:21:10,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:21:11,108 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 21:21:11,112 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 21:21:11,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:21:11,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 21:21:11,448 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:21:11,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 21:21:11,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 21:21:11,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 21:21:11,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 21:21:11,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 21:21:11,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 21:21:11,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 21:21:11,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:21:11,686 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 21:21:11,690 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 21:21:11,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:21:11,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 21:21:12,012 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:21:12,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 21:21:12,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 21:21:12,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 21:21:12,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 21:21:12,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 21:21:12,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 21:21:12,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 21:21:12,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:21:12,246 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 21:21:12,252 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 21:21:12,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:21:12,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 21:21:12,569 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:21:12,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 21:21:12,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 21:21:12,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 21:21:12,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 21:21:12,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 21:21:12,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 21:21:12,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 21:21:12,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:21:12,806 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 21:21:12,812 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 21:21:12,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:21:13,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 21:21:13,164 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:21:13,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 21:21:13,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 21:21:13,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 21:21:13,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 21:21:13,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 21:21:13,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 21:21:13,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 21:21:13,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:21:13,400 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 21:21:13,400 | test | INFO | MainThread | [opd:create_required_buckets:79] Rebalance Starts 2022-02-07 21:21:13,430 | test | DEBUG | MainThread | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.128,ns_1@172.23.121.115,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123', 'ejectedNodes': '', 'user': 'Administrator'} 2022-02-07 21:21:13,484 | test | DEBUG | MainThread | [rest_client:rebalance:1459] Rebalance operation started 2022-02-07 21:21:13,499 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-07 21:21:13,507 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-07 21:21:13,507 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-07 21:21:16,526 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-07 21:21:16,526 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:21:21,546 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-07 21:21:21,546 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-07 21:21:24,555 | test | INFO | MainThread | [rest_client:monitorRebalance:1599] Rebalance done. Taken 11.0699999332 seconds to complete 2022-02-07 21:21:24,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1602 2022-02-07 21:21:24,562 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-07 21:21:29,562 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection0 2022-02-07 21:21:29,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:29,622 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:30,122 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection1 2022-02-07 21:21:30,188 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:30,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:30,690 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection2 2022-02-07 21:21:30,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:30,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:31,255 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection3 2022-02-07 21:21:31,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:31,316 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:31,816 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection4 2022-02-07 21:21:31,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:31,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:32,378 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection5 2022-02-07 21:21:32,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:32,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:32,937 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection6 2022-02-07 21:21:33,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:33,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:33,503 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection7 2022-02-07 21:21:33,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:33,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:34,068 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection8 2022-02-07 21:21:34,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:34,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:34,665 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection9 2022-02-07 21:21:34,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:34,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:35,226 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection10 2022-02-07 21:21:35,289 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:35,289 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:35,789 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection11 2022-02-07 21:21:35,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:35,846 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:36,346 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection12 2022-02-07 21:21:36,408 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:36,410 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:36,911 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection13 2022-02-07 21:21:36,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:36,977 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:37,479 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection14 2022-02-07 21:21:37,548 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:37,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:38,049 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection15 2022-02-07 21:21:38,107 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:38,107 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:38,607 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection16 2022-02-07 21:21:38,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:38,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:39,171 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection17 2022-02-07 21:21:39,385 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:39,387 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:39,887 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection18 2022-02-07 21:21:39,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:39,946 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:40,446 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection19 2022-02-07 21:21:40,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2022-02-07 21:21:40,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2022-02-07 21:21:41,009 | test | DEBUG | MainThread | [opd:create_required_collections:119] Collections list == ['_default', 'VolumeCollection9', 'VolumeCollection8', 'VolumeCollection19', 'VolumeCollection13', 'VolumeCollection14', 'VolumeCollection11', 'VolumeCollection12', 'VolumeCollection17', 'VolumeCollection18', 'VolumeCollection15', 'VolumeCollection16', 'VolumeCollection7', 'VolumeCollection6', 'VolumeCollection5', 'VolumeCollection4', 'VolumeCollection3', 'VolumeCollection2', 'VolumeCollection10', 'VolumeCollection1', 'VolumeCollection0'] 2022-02-07 21:21:41,023 | test | DEBUG | MainThread | [rest_client:update_autofailover_settings:2075] settings/autoFailover params: failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1&timeout=600&enabled=false&canAbortRebalance=false 2022-02-07 21:21:41,049 | test | DEBUG | MainThread | [rest_client:set_service_mem_quota:945] Setting RAM quotas: {'indexMemoryQuota': 9279} 2022-02-07 21:21:41,107 | infra | DEBUG | MainThread | [task:__init__:249] Last known rebalance status_id: 951331b60a3dc135f184acbe2c0b545a 2022-02-07 21:21:41,108 | infra | INFO | pool-3-thread-19 | [task:start_task:91] Thread 'Rebalance_task_IN=[172.23.121.130,172.23.121.131,172.23.121.132,172.23.121.133,172.23.121.134]_OUT=[]_1644297701.06' started 2022-02-07 21:21:41,108 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[172.23.121.130,172.23.121.131,172.23.121.132,172.23.121.133,172.23.121.134]_OUT=[]_1644297701.06 2022-02-07 21:21:41,111 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[172.23.121.130,172.23.121.131,172.23.121.132,172.23.121.133,172.23.121.134]_OUT=[]_1644297701.06 2022-02-07 21:21:41,114 | test | DEBUG | pool-3-thread-19 | [rest_client:add_node:1199] Adding remote node 172.23.121.130:8091 to cluster 172.23.120.170:8091 2022-02-07 21:21:44,690 | test | DEBUG | pool-3-thread-19 | [rest_client:add_node:1199] Adding remote node 172.23.121.131:8091 to cluster 172.23.120.170:8091 2022-02-07 21:21:48,615 | test | DEBUG | pool-3-thread-19 | [rest_client:add_node:1199] Adding remote node 172.23.121.132:8091 to cluster 172.23.120.170:8091 2022-02-07 21:21:51,904 | test | DEBUG | pool-3-thread-19 | [rest_client:add_node:1199] Adding remote node 172.23.121.133:8091 to cluster 172.23.120.170:8091 2022-02-07 21:21:55,411 | test | DEBUG | pool-3-thread-19 | [rest_client:add_node:1199] Adding remote node 172.23.121.134:8091 to cluster 172.23.120.170:8091 2022-02-07 21:21:59,944 | test | DEBUG | pool-3-thread-19 | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.128,ns_1@172.23.121.115,ns_1@172.23.121.132,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.134,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123,ns_1@172.23.121.131,ns_1@172.23.121.130,ns_1@172.23.121.133', 'ejectedNodes': '', 'user': 'Administrator'} 2022-02-07 21:22:00,005 | test | DEBUG | pool-3-thread-19 | [rest_client:rebalance:1459] Rebalance operation started 2022-02-07 21:22:00,006 | test | INFO | pool-3-thread-19 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.121.128 | kv | 7.1.0-2255-enterprise | 1.35644310475 | Cluster node | | 172.23.121.115 | kv | 7.1.0-2255-enterprise | 1.83509301156 | Cluster node | | 172.23.120.170 | kv | 7.1.0-2255-enterprise | 2.10975762903 | Cluster node | | 172.23.121.129 | kv | 7.1.0-2255-enterprise | 1.73541247485 | Cluster node | | 172.23.121.124 | kv | 7.1.0-2255-enterprise | 1.75945708181 | Cluster node | | 172.23.121.126 | kv | 7.1.0-2255-enterprise | 1.67253521127 | Cluster node | | 172.23.121.116 | kv | 7.1.0-2255-enterprise | 1.66750250752 | Cluster node | | 172.23.121.127 | kv | 7.1.0-2255-enterprise | 2.05539541296 | Cluster node | | 172.23.121.123 | kv | 7.1.0-2255-enterprise | 1.85953009172 | Cluster node | | 172.23.121.130 | ['index,n1ql'] | | | <--- IN --- | | 172.23.121.131 | ['index,n1ql'] | | | <--- IN --- | | 172.23.121.132 | ['index,n1ql'] | | | <--- IN --- | | 172.23.121.133 | ['index,n1ql'] | | | <--- IN --- | | 172.23.121.134 | ['index,n1ql'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2022-02-07 21:22:00,019 | infra | DEBUG | pool-3-thread-19 | [task:call:316] New rebalance status_id: 4b56e0bf9ef82ea24b66b2006488099a 2022-02-07 21:22:00,029 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-07 21:22:00,029 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-07 21:22:00,058 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-07 21:22:00,061 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-07 21:22:00,061 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-07 21:22:10,072 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-07 21:22:10,072 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:22:15,082 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: none, progress: 100 2022-02-07 21:22:15,108 | test | INFO | pool-3-thread-19 | [task:check:473] Rebalance completed with progress: 100% in 15.1039998531 sec 2022-02-07 21:22:15,109 | infra | INFO | pool-3-thread-19 | [task:complete_task:106] Thread 'Rebalance_task_IN=[172.23.121.130,172.23.121.131,172.23.121.132,172.23.121.133,172.23.121.134]_OUT=[]_1644297701.06' completed 2022-02-07 21:22:15,111 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:245] Creating SDK connection for 'None' [ip:172.23.121.135 port:8091 ssh_username:root, ip:172.23.121.136 port:8091 ssh_username:root] 2022-02-07 21:22:15,575 | infra | INFO | MainThread | [BucketOperations_Rest:update_memcached_settings:373] Updating memcached properties 2022-02-07 21:22:15,575 | infra | DEBUG | MainThread | [BucketOperations_Rest:update_memcached_settings:374] http://172.23.120.170:8091/pools/default/settings/memcached/global with param: num_reader_threads=disk_io_optimized&num_writer_threads=disk_io_optimized&num_storage_threads=40 2022-02-07 21:22:15,598 | infra | DEBUG | MainThread | [BucketOperations_Rest:update_memcached_settings:380] Memcached settings updated ############################################################ # # Step 1: Create 1000000 items sequentially # ############################################################ 2022-02-07 21:22:15,601 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 21:22:15,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:22:15,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 21:22:15,969 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:22:15,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 21:22:15,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 21:22:15,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 21:22:16,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 21:22:16,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 21:22:16,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 21:22:16,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 21:22:16,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-07 21:22:16,158 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 ############################################################ # # Step 2: Create 1000000 items sequentially # ############################################################ ############################################################ # # RAM FootPrint: 1058.64453125 # ############################################################ Read Start: 0 Read End: 0 Update Start: 0 Update End: 0 Expiry Start: 0 Expiry End: 0 Delete Start: 0 Delete End: 0 Create Start: 0 Create End: 1000000 Final Start: 0 Final End: 1000000 2022-02-07 21:22:16,499 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:16,499 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection9_10_1644297737.5 2022-02-07 21:22:17,522 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:17,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection8_9_1644297738.52 2022-02-07 21:22:18,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:18,540 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection19_8_1644297739.54 2022-02-07 21:22:19,558 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:19,558 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection13_7_1644297740.56 2022-02-07 21:22:20,604 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:20,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection14_6_1644297741.61 2022-02-07 21:22:21,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:21,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection11_5_1644297742.62 2022-02-07 21:22:22,640 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:22,640 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection12_4_1644297743.66 2022-02-07 21:22:23,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:23,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection17_3_1644297744.68 2022-02-07 21:22:24,688 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:24,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection18_2_1644297745.69 2022-02-07 21:22:25,706 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:25,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection15_1_1644297746.71 2022-02-07 21:22:26,720 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:26,720 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:22:27,734 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:27,734 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:22:28,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:28,752 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:22:29,778 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:29,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:22:30,792 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:30,792 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:22:31,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:31,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:22:32,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:32,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:22:33,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:33,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:22:34,871 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:34,871 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:22:35,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:22:35,882 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Loader_GleamBookUsers0__default_VolumeCollection8_9_1644297738.52 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection16_0_1644297747.72 Loader_GleamBookUsers0__default_VolumeCollection13_7_1644297740.56 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection7_-1_1644297748.74 Loader_GleamBookUsers0__default_VolumeCollection18_2_1644297745.69 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection6_-2_1644297749.76 Loader_GleamBookUsers0__default_VolumeCollection12_4_1644297743.66 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection5_-3_1644297750.78 Loader_GleamBookUsers0__default_VolumeCollection9_10_1644297737.5 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection4_-4_1644297751.82 Loader_GleamBookUsers0__default_VolumeCollection19_8_1644297739.54 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection3_-5_1644297752.83 Loader_GleamBookUsers0__default_VolumeCollection11_5_1644297742.62 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection2_-6_1644297753.85 Loader_GleamBookUsers0__default_VolumeCollection14_6_1644297741.61 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection10_-7_1644297754.86 Loader_GleamBookUsers0__default_VolumeCollection17_3_1644297744.68 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection1_-8_1644297755.87 Loader_GleamBookUsers0__default_VolumeCollection15_1_1644297746.71 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection0_-9_1644297756.88 Loader_GleamBookUsers0__default_VolumeCollection7_-1_1644297748.74 is completed! Loader_GleamBookUsers0__default_VolumeCollection6_-2_1644297749.76 is completed! Loader_GleamBookUsers0__default_VolumeCollection16_0_1644297747.72 is completed! Loader_GleamBookUsers0__default_VolumeCollection5_-3_1644297750.78 is completed! Loader_GleamBookUsers0__default_VolumeCollection4_-4_1644297751.82 is completed! Loader_GleamBookUsers0__default_VolumeCollection2_-6_1644297753.85 is completed! Loader_GleamBookUsers0__default_VolumeCollection3_-5_1644297752.83 is completed! Loader_GleamBookUsers0__default_VolumeCollection10_-7_1644297754.86 is completed! Loader_GleamBookUsers0__default_VolumeCollection1_-8_1644297755.87 is completed! Loader_GleamBookUsers0__default_VolumeCollection0_-9_1644297756.88 is completed! 2022-02-07 21:44:52,628 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:651] All nodes in cluster: [ip:172.23.121.128 port:8091 ssh_username:root, ip:172.23.121.115 port:8091 ssh_username:root, ip:172.23.120.170 port:8091 ssh_username:root, ip:172.23.121.129 port:8091 ssh_username:root, ip:172.23.121.124 port:8091 ssh_username:root, ip:172.23.121.126 port:8091 ssh_username:root, ip:172.23.121.116 port:8091 ssh_username:root, ip:172.23.121.127 port:8091 ssh_username:root, ip:172.23.121.123 port:8091 ssh_username:root] 2022-02-07 21:44:52,634 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 21:44:52,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:44:52,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 21:44:53,040 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:44:53,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 21:44:53,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 21:44:53,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 21:44:53,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 21:44:53,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 21:44:53,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 21:44:53,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 21:44:53,142 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 21:44:53,144 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299093.14 2022-02-07 21:44:53,145 | infra | INFO | pool-3-thread-20 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299093.14' started 2022-02-07 21:44:53,147 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:44:53,150 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 21:44:53,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:44:53,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 21:44:53,536 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:44:53,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 21:44:53,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 21:44:53,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 21:44:53,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 21:44:53,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 21:44:53,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 21:44:53,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 21:44:53,634 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 21:44:53,635 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299093.64 2022-02-07 21:44:53,637 | infra | INFO | pool-3-thread-21 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299093.64' started 2022-02-07 21:44:53,638 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:44:53,644 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 21:44:53,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:44:53,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 21:44:54,019 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:44:54,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 21:44:54,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 21:44:54,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 21:44:54,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 21:44:54,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 21:44:54,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 21:44:54,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 21:44:54,118 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 21:44:54,121 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299094.12 2022-02-07 21:44:54,124 | infra | INFO | pool-3-thread-22 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299094.12' started 2022-02-07 21:44:54,125 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 21:44:54,127 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:44:54,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:44:54,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 21:44:54,552 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:44:54,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 21:44:54,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 21:44:54,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 21:44:54,584 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 21:44:54,584 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.116': 0} for bucket 'GleamBookUsers0' 2022-02-07 21:44:54,584 | infra | INFO | pool-3-thread-22 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299094.12' completed 2022-02-07 21:44:54,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 21:44:54,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 21:44:54,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 21:44:54,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 21:44:54,651 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 21:44:54,653 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299094.65 2022-02-07 21:44:54,654 | infra | INFO | pool-3-thread-23 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299094.65' started 2022-02-07 21:44:54,657 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:44:54,658 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 21:44:54,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:44:54,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 21:44:55,023 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:44:55,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 21:44:55,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 21:44:55,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 21:44:55,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 21:44:55,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 21:44:55,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 21:44:55,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 21:44:55,141 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 21:44:55,142 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299095.14 2022-02-07 21:44:55,144 | infra | INFO | pool-3-thread-24 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299095.14' started 2022-02-07 21:44:55,144 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:44:55,148 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 21:44:55,157 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 1351095 == 0 2022-02-07 21:44:55,157 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1351095 == 0. Received: {'172.23.121.115': 1351095} for bucket 'GleamBookUsers0' 2022-02-07 21:44:55,157 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:44:55,160 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:44:55,161 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:44:55,204 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1877851 == 0 2022-02-07 21:44:55,204 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1877851 == 0. Received: {'172.23.120.170': 1877851} for bucket 'GleamBookUsers0' 2022-02-07 21:44:55,206 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:44:55,209 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:44:55,210 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:44:55,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:44:55,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 21:44:55,512 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:44:55,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 21:44:55,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 21:44:55,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 21:44:55,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 21:44:55,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 21:44:55,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 21:44:55,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 21:44:55,608 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 21:44:55,609 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299095.61 2022-02-07 21:44:55,611 | infra | INFO | pool-3-thread-25 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299095.61' started 2022-02-07 21:44:55,611 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:44:55,614 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 21:44:55,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:44:55,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 21:44:55,973 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:44:55,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 21:44:55,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 21:44:56,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 21:44:56,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 21:44:56,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 21:44:56,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 21:44:56,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 21:44:56,122 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 21:44:56,127 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299096.12 2022-02-07 21:44:56,130 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 21:44:56,135 | infra | INFO | pool-3-thread-26 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299096.12' started 2022-02-07 21:44:56,137 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:44:56,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:44:56,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 21:44:56,489 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:44:56,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 21:44:56,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 21:44:56,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 21:44:56,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 21:44:56,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 21:44:56,561 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 2024519 == 0 2022-02-07 21:44:56,562 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2024519 == 0. Received: {'172.23.121.123': 2024519} for bucket 'GleamBookUsers0' 2022-02-07 21:44:56,562 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:44:56,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 21:44:56,565 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:44:56,565 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:44:56,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 21:44:56,591 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 21:44:56,592 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299096.59 2022-02-07 21:44:56,592 | infra | INFO | pool-3-thread-27 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299096.59' started 2022-02-07 21:44:56,594 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:44:56,596 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 21:44:56,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:44:56,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 21:44:56,938 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:44:56,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 21:44:56,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 21:44:56,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 21:44:56,977 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1785577 == 0 2022-02-07 21:44:56,979 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1785577 == 0. Received: {'172.23.121.124': 1785577} for bucket 'GleamBookUsers0' 2022-02-07 21:44:56,979 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:44:56,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 21:44:56,982 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:44:56,982 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:44:56,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 21:44:57,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 21:44:57,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 21:44:57,042 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 21:44:57,043 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299097.04 2022-02-07 21:44:57,045 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299093.14 2022-02-07 21:44:57,046 | infra | INFO | pool-3-thread-28 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299097.04' started 2022-02-07 21:44:57,046 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:44:57,085 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 1149805 == 0 2022-02-07 21:44:57,086 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1149805 == 0. Received: {'172.23.121.126': 1149805} for bucket 'GleamBookUsers0' 2022-02-07 21:44:57,086 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:44:57,088 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:44:57,089 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:44:57,759 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1501841 == 0 2022-02-07 21:44:57,760 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1501841 == 0. Received: {'172.23.121.127': 1501841} for bucket 'GleamBookUsers0' 2022-02-07 21:44:57,760 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:44:57,763 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:44:57,765 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:44:58,247 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1641119 == 0 2022-02-07 21:44:58,249 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1641119 == 0. Received: {'172.23.121.128': 1641119} for bucket 'GleamBookUsers0' 2022-02-07 21:44:58,249 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:44:58,250 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:44:58,250 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:44:58,832 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1985281 == 0 2022-02-07 21:44:58,832 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1985281 == 0. Received: {'172.23.121.129': 1985281} for bucket 'GleamBookUsers0' 2022-02-07 21:44:58,832 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:44:58,835 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:44:58,835 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:00,161 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:00,210 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:01,565 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:01,594 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 1308340 == 0 2022-02-07 21:45:01,594 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1308340 == 0. Received: {'172.23.121.115': 1308340} for bucket 'GleamBookUsers0' 2022-02-07 21:45:01,594 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:01,596 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:01,598 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:01,917 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1819977 == 0 2022-02-07 21:45:01,917 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1819977 == 0. Received: {'172.23.120.170': 1819977} for bucket 'GleamBookUsers0' 2022-02-07 21:45:01,917 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:01,920 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:01,920 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:01,983 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:02,091 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:02,765 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:03,252 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:03,540 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1997412 == 0 2022-02-07 21:45:03,540 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1997412 == 0. Received: {'172.23.121.123': 1997412} for bucket 'GleamBookUsers0' 2022-02-07 21:45:03,542 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:03,543 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:03,543 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:03,721 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 1110126 == 0 2022-02-07 21:45:03,721 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1110126 == 0. Received: {'172.23.121.126': 1110126} for bucket 'GleamBookUsers0' 2022-02-07 21:45:03,723 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:03,724 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:03,724 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:03,835 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:03,861 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1762341 == 0 2022-02-07 21:45:03,862 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1762341 == 0. Received: {'172.23.121.124': 1762341} for bucket 'GleamBookUsers0' 2022-02-07 21:45:03,862 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:03,865 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:03,865 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:04,463 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1474595 == 0 2022-02-07 21:45:04,464 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1474595 == 0. Received: {'172.23.121.127': 1474595} for bucket 'GleamBookUsers0' 2022-02-07 21:45:04,464 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:04,467 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:04,467 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:04,882 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1575536 == 0 2022-02-07 21:45:04,884 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1575536 == 0. Received: {'172.23.121.128': 1575536} for bucket 'GleamBookUsers0' 2022-02-07 21:45:04,884 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:04,885 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:04,885 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:05,638 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1960455 == 0 2022-02-07 21:45:05,638 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1960455 == 0. Received: {'172.23.121.129': 1960455} for bucket 'GleamBookUsers0' 2022-02-07 21:45:05,638 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:05,640 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:05,641 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:06,598 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:06,920 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:07,944 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 1277340 == 0 2022-02-07 21:45:07,946 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1277340 == 0. Received: {'172.23.121.115': 1277340} for bucket 'GleamBookUsers0' 2022-02-07 21:45:07,946 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:07,947 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:07,948 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:08,545 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:08,555 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1790049 == 0 2022-02-07 21:45:08,555 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1790049 == 0. Received: {'172.23.120.170': 1790049} for bucket 'GleamBookUsers0' 2022-02-07 21:45:08,555 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:08,558 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:08,558 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:08,726 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:08,865 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:09,467 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:09,887 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:10,167 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 1091928 == 0 2022-02-07 21:45:10,167 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1091928 == 0. Received: {'172.23.121.126': 1091928} for bucket 'GleamBookUsers0' 2022-02-07 21:45:10,167 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:10,171 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:10,171 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:10,446 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1935912 == 0 2022-02-07 21:45:10,447 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1935912 == 0. Received: {'172.23.121.123': 1935912} for bucket 'GleamBookUsers0' 2022-02-07 21:45:10,447 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:10,448 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:10,450 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:10,584 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1752453 == 0 2022-02-07 21:45:10,584 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1752453 == 0. Received: {'172.23.121.124': 1752453} for bucket 'GleamBookUsers0' 2022-02-07 21:45:10,584 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:10,586 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:10,586 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:10,641 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:11,028 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1455835 == 0 2022-02-07 21:45:11,029 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1455835 == 0. Received: {'172.23.121.127': 1455835} for bucket 'GleamBookUsers0' 2022-02-07 21:45:11,029 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:11,030 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:11,030 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:11,464 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1555887 == 0 2022-02-07 21:45:11,464 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1555887 == 0. Received: {'172.23.121.128': 1555887} for bucket 'GleamBookUsers0' 2022-02-07 21:45:11,464 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:11,467 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:11,467 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:12,374 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1943909 == 0 2022-02-07 21:45:12,375 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1943909 == 0. Received: {'172.23.121.129': 1943909} for bucket 'GleamBookUsers0' 2022-02-07 21:45:12,375 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:12,378 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:12,378 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:12,948 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:13,558 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:14,237 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 1258044 == 0 2022-02-07 21:45:14,237 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1258044 == 0. Received: {'172.23.121.115': 1258044} for bucket 'GleamBookUsers0' 2022-02-07 21:45:14,239 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:14,240 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:14,240 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:15,171 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:15,177 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1718164 == 0 2022-02-07 21:45:15,177 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1718164 == 0. Received: {'172.23.120.170': 1718164} for bucket 'GleamBookUsers0' 2022-02-07 21:45:15,177 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:15,180 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:15,180 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:15,450 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:15,586 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:16,032 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:16,460 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 1033928 == 0 2022-02-07 21:45:16,460 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1033928 == 0. Received: {'172.23.121.126': 1033928} for bucket 'GleamBookUsers0' 2022-02-07 21:45:16,460 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:16,463 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:16,463 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:16,467 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:17,279 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1918373 == 0 2022-02-07 21:45:17,279 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1918373 == 0. Received: {'172.23.121.123': 1918373} for bucket 'GleamBookUsers0' 2022-02-07 21:45:17,279 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:17,282 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:17,282 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:17,378 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:17,427 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1713093 == 0 2022-02-07 21:45:17,427 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1713093 == 0. Received: {'172.23.121.124': 1713093} for bucket 'GleamBookUsers0' 2022-02-07 21:45:17,428 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:17,430 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:17,430 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:17,614 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1424363 == 0 2022-02-07 21:45:17,615 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1424363 == 0. Received: {'172.23.121.127': 1424363} for bucket 'GleamBookUsers0' 2022-02-07 21:45:17,615 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:17,617 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:17,618 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:18,035 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1523083 == 0 2022-02-07 21:45:18,035 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1523083 == 0. Received: {'172.23.121.128': 1523083} for bucket 'GleamBookUsers0' 2022-02-07 21:45:18,035 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:18,038 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:18,038 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:19,173 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1912443 == 0 2022-02-07 21:45:19,174 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1912443 == 0. Received: {'172.23.121.129': 1912443} for bucket 'GleamBookUsers0' 2022-02-07 21:45:19,174 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:19,176 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:19,177 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:19,242 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:20,180 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:20,503 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 1247548 == 0 2022-02-07 21:45:20,505 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1247548 == 0. Received: {'172.23.121.115': 1247548} for bucket 'GleamBookUsers0' 2022-02-07 21:45:20,505 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:20,506 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:20,507 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:21,464 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:21,792 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1701392 == 0 2022-02-07 21:45:21,792 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1701392 == 0. Received: {'172.23.120.170': 1701392} for bucket 'GleamBookUsers0' 2022-02-07 21:45:21,792 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:21,796 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:21,796 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:22,282 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:22,431 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:22,618 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:22,759 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 1016254 == 0 2022-02-07 21:45:22,760 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1016254 == 0. Received: {'172.23.121.126': 1016254} for bucket 'GleamBookUsers0' 2022-02-07 21:45:22,760 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:22,763 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:22,763 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:23,039 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:24,177 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:24,204 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1647932 == 0 2022-02-07 21:45:24,206 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1647932 == 0. Received: {'172.23.121.124': 1647932} for bucket 'GleamBookUsers0' 2022-02-07 21:45:24,206 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:24,209 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:24,209 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:24,266 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1918373 == 0 2022-02-07 21:45:24,267 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1918373 == 0. Received: {'172.23.121.123': 1918373} for bucket 'GleamBookUsers0' 2022-02-07 21:45:24,267 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:24,269 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:24,269 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:24,309 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1386782 == 0 2022-02-07 21:45:24,309 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1386782 == 0. Received: {'172.23.121.127': 1386782} for bucket 'GleamBookUsers0' 2022-02-07 21:45:24,311 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:24,312 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:24,312 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:24,634 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1490391 == 0 2022-02-07 21:45:24,635 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1490391 == 0. Received: {'172.23.121.128': 1490391} for bucket 'GleamBookUsers0' 2022-02-07 21:45:24,635 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:24,638 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:24,638 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:25,507 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:25,927 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1887999 == 0 2022-02-07 21:45:25,927 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1887999 == 0. Received: {'172.23.121.129': 1887999} for bucket 'GleamBookUsers0' 2022-02-07 21:45:25,927 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:25,931 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:25,931 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:26,796 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:26,822 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 1208688 == 0 2022-02-07 21:45:26,822 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1208688 == 0. Received: {'172.23.121.115': 1208688} for bucket 'GleamBookUsers0' 2022-02-07 21:45:26,822 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:26,825 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:26,825 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:27,763 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:28,403 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1682996 == 0 2022-02-07 21:45:28,404 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1682996 == 0. Received: {'172.23.120.170': 1682996} for bucket 'GleamBookUsers0' 2022-02-07 21:45:28,404 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:28,405 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:28,405 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:29,108 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 979660 == 0 2022-02-07 21:45:29,108 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 979660 == 0. Received: {'172.23.121.126': 979660} for bucket 'GleamBookUsers0' 2022-02-07 21:45:29,108 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:29,111 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:29,111 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:29,209 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:29,270 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:29,313 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:29,640 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:30,933 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:31,154 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1360788 == 0 2022-02-07 21:45:31,154 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1360788 == 0. Received: {'172.23.121.127': 1360788} for bucket 'GleamBookUsers0' 2022-02-07 21:45:31,154 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:31,157 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:31,157 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:31,197 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1629250 == 0 2022-02-07 21:45:31,197 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1629250 == 0. Received: {'172.23.121.124': 1629250} for bucket 'GleamBookUsers0' 2022-02-07 21:45:31,197 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:31,200 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:31,200 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:31,338 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1877045 == 0 2022-02-07 21:45:31,338 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1877045 == 0. Received: {'172.23.121.123': 1877045} for bucket 'GleamBookUsers0' 2022-02-07 21:45:31,338 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:31,341 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:31,341 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:31,436 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1444629 == 0 2022-02-07 21:45:31,437 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1444629 == 0. Received: {'172.23.121.128': 1444629} for bucket 'GleamBookUsers0' 2022-02-07 21:45:31,437 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:31,440 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:31,440 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:31,826 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:32,611 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1854317 == 0 2022-02-07 21:45:32,611 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1854317 == 0. Received: {'172.23.121.129': 1854317} for bucket 'GleamBookUsers0' 2022-02-07 21:45:32,611 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:32,612 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:32,615 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:33,069 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 1157796 == 0 2022-02-07 21:45:33,069 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1157796 == 0. Received: {'172.23.121.115': 1157796} for bucket 'GleamBookUsers0' 2022-02-07 21:45:33,069 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:33,072 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:33,072 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:33,407 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:34,112 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:34,960 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1616614 == 0 2022-02-07 21:45:34,960 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1616614 == 0. Received: {'172.23.120.170': 1616614} for bucket 'GleamBookUsers0' 2022-02-07 21:45:34,960 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:34,963 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:34,963 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:35,448 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 953518 == 0 2022-02-07 21:45:35,450 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 953518 == 0. Received: {'172.23.121.126': 953518} for bucket 'GleamBookUsers0' 2022-02-07 21:45:35,450 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:35,453 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:35,453 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:36,158 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:36,200 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:36,342 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:36,440 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:37,615 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:37,943 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1332636 == 0 2022-02-07 21:45:37,944 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1332636 == 0. Received: {'172.23.121.127': 1332636} for bucket 'GleamBookUsers0' 2022-02-07 21:45:37,944 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:37,946 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:37,947 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:37,963 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1566431 == 0 2022-02-07 21:45:37,963 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1566431 == 0. Received: {'172.23.121.124': 1566431} for bucket 'GleamBookUsers0' 2022-02-07 21:45:37,963 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:37,966 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:37,966 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:38,072 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:38,188 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1413993 == 0 2022-02-07 21:45:38,188 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1413993 == 0. Received: {'172.23.121.128': 1413993} for bucket 'GleamBookUsers0' 2022-02-07 21:45:38,188 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:38,190 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:38,191 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:38,224 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1858651 == 0 2022-02-07 21:45:38,224 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1858651 == 0. Received: {'172.23.121.123': 1858651} for bucket 'GleamBookUsers0' 2022-02-07 21:45:38,224 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:38,226 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:38,227 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:39,372 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 1131458 == 0 2022-02-07 21:45:39,372 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1131458 == 0. Received: {'172.23.121.115': 1131458} for bucket 'GleamBookUsers0' 2022-02-07 21:45:39,374 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:39,375 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:39,375 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:39,436 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1808019 == 0 2022-02-07 21:45:39,437 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1808019 == 0. Received: {'172.23.121.129': 1808019} for bucket 'GleamBookUsers0' 2022-02-07 21:45:39,437 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:39,438 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:39,438 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:39,963 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:40,453 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:41,552 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1579036 == 0 2022-02-07 21:45:41,553 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1579036 == 0. Received: {'172.23.120.170': 1579036} for bucket 'GleamBookUsers0' 2022-02-07 21:45:41,553 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:41,556 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:41,556 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:41,806 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 920600 == 0 2022-02-07 21:45:41,806 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 920600 == 0. Received: {'172.23.121.126': 920600} for bucket 'GleamBookUsers0' 2022-02-07 21:45:41,806 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:41,809 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:41,809 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:42,947 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:42,967 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:43,191 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:43,227 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:44,377 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:44,440 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:44,632 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1324384 == 0 2022-02-07 21:45:44,632 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1324384 == 0. Received: {'172.23.121.127': 1324384} for bucket 'GleamBookUsers0' 2022-02-07 21:45:44,632 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:44,635 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:44,637 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:44,877 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1549326 == 0 2022-02-07 21:45:44,878 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1549326 == 0. Received: {'172.23.121.124': 1549326} for bucket 'GleamBookUsers0' 2022-02-07 21:45:44,878 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:44,880 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:44,880 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:45,049 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1405823 == 0 2022-02-07 21:45:45,049 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1405823 == 0. Received: {'172.23.121.128': 1405823} for bucket 'GleamBookUsers0' 2022-02-07 21:45:45,049 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:45,052 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:45,052 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:45,190 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1825290 == 0 2022-02-07 21:45:45,191 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1825290 == 0. Received: {'172.23.121.123': 1825290} for bucket 'GleamBookUsers0' 2022-02-07 21:45:45,191 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:45,194 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:45,194 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:45,592 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 1110956 == 0 2022-02-07 21:45:45,592 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1110956 == 0. Received: {'172.23.121.115': 1110956} for bucket 'GleamBookUsers0' 2022-02-07 21:45:45,592 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:45,595 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:45,595 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:46,094 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1783571 == 0 2022-02-07 21:45:46,095 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1783571 == 0. Received: {'172.23.121.129': 1783571} for bucket 'GleamBookUsers0' 2022-02-07 21:45:46,095 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:46,096 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:46,096 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:46,558 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:46,809 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:48,029 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 902356 == 0 2022-02-07 21:45:48,030 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 902356 == 0. Received: {'172.23.121.126': 902356} for bucket 'GleamBookUsers0' 2022-02-07 21:45:48,032 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:48,036 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:48,036 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:48,155 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1571536 == 0 2022-02-07 21:45:48,155 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1571536 == 0. Received: {'172.23.120.170': 1571536} for bucket 'GleamBookUsers0' 2022-02-07 21:45:48,155 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:48,157 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:48,157 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:49,638 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:49,881 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:50,052 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:50,194 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:50,595 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:51,095 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1289328 == 0 2022-02-07 21:45:51,096 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1289328 == 0. Received: {'172.23.121.127': 1289328} for bucket 'GleamBookUsers0' 2022-02-07 21:45:51,096 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:51,098 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:51,099 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:51,099 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:51,668 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1524566 == 0 2022-02-07 21:45:51,668 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1524566 == 0. Received: {'172.23.121.124': 1524566} for bucket 'GleamBookUsers0' 2022-02-07 21:45:51,670 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:51,671 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:51,673 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:51,743 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1357609 == 0 2022-02-07 21:45:51,743 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1357609 == 0. Received: {'172.23.121.128': 1357609} for bucket 'GleamBookUsers0' 2022-02-07 21:45:51,743 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:51,746 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:51,746 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:52,003 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 1094466 == 0 2022-02-07 21:45:52,003 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1094466 == 0. Received: {'172.23.121.115': 1094466} for bucket 'GleamBookUsers0' 2022-02-07 21:45:52,003 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:52,006 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:52,006 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:52,042 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1792004 == 0 2022-02-07 21:45:52,043 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1792004 == 0. Received: {'172.23.121.123': 1792004} for bucket 'GleamBookUsers0' 2022-02-07 21:45:52,043 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:52,046 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:52,046 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:52,686 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1768923 == 0 2022-02-07 21:45:52,687 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1768923 == 0. Received: {'172.23.121.129': 1768923} for bucket 'GleamBookUsers0' 2022-02-07 21:45:52,687 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:52,688 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:52,688 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:53,036 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:53,158 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:54,176 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 858373 == 0 2022-02-07 21:45:54,177 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 858373 == 0. Received: {'172.23.121.126': 858373} for bucket 'GleamBookUsers0' 2022-02-07 21:45:54,177 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:54,178 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:54,180 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:54,690 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1503348 == 0 2022-02-07 21:45:54,690 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1503348 == 0. Received: {'172.23.120.170': 1503348} for bucket 'GleamBookUsers0' 2022-02-07 21:45:54,690 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:54,693 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:54,693 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:56,101 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:56,673 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:56,746 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:57,007 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:57,046 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:57,374 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1217492 == 0 2022-02-07 21:45:57,374 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1217492 == 0. Received: {'172.23.121.127': 1217492} for bucket 'GleamBookUsers0' 2022-02-07 21:45:57,375 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:57,377 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:57,377 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:57,690 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:58,461 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 1064184 == 0 2022-02-07 21:45:58,461 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1064184 == 0. Received: {'172.23.121.115': 1064184} for bucket 'GleamBookUsers0' 2022-02-07 21:45:58,461 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:58,464 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:58,464 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:58,479 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1344097 == 0 2022-02-07 21:45:58,480 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1344097 == 0. Received: {'172.23.121.128': 1344097} for bucket 'GleamBookUsers0' 2022-02-07 21:45:58,480 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:58,482 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:58,482 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:58,486 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1507198 == 0 2022-02-07 21:45:58,486 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1507198 == 0. Received: {'172.23.121.124': 1507198} for bucket 'GleamBookUsers0' 2022-02-07 21:45:58,486 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:58,490 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:58,490 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:58,809 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1753232 == 0 2022-02-07 21:45:58,809 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1753232 == 0. Received: {'172.23.121.123': 1753232} for bucket 'GleamBookUsers0' 2022-02-07 21:45:58,809 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:58,812 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:58,812 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:59,180 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:45:59,367 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1760929 == 0 2022-02-07 21:45:59,367 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1760929 == 0. Received: {'172.23.121.129': 1760929} for bucket 'GleamBookUsers0' 2022-02-07 21:45:59,368 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:45:59,371 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:45:59,371 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:45:59,694 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:00,335 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 825131 == 0 2022-02-07 21:46:00,336 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 825131 == 0. Received: {'172.23.121.126': 825131} for bucket 'GleamBookUsers0' 2022-02-07 21:46:00,336 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:00,338 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:00,339 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:01,151 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1457694 == 0 2022-02-07 21:46:01,153 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1457694 == 0. Received: {'172.23.120.170': 1457694} for bucket 'GleamBookUsers0' 2022-02-07 21:46:01,153 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:01,154 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:01,154 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:02,378 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:03,466 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:03,483 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:03,492 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:03,655 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1183528 == 0 2022-02-07 21:46:03,655 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1183528 == 0. Received: {'172.23.121.127': 1183528} for bucket 'GleamBookUsers0' 2022-02-07 21:46:03,657 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:03,658 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:03,658 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:03,815 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:04,371 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:04,729 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 1046342 == 0 2022-02-07 21:46:04,729 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1046342 == 0. Received: {'172.23.121.115': 1046342} for bucket 'GleamBookUsers0' 2022-02-07 21:46:04,729 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:04,733 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:04,733 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:05,111 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1299671 == 0 2022-02-07 21:46:05,111 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1299671 == 0. Received: {'172.23.121.128': 1299671} for bucket 'GleamBookUsers0' 2022-02-07 21:46:05,111 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:05,114 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:05,114 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:05,177 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1472576 == 0 2022-02-07 21:46:05,178 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1472576 == 0. Received: {'172.23.121.124': 1472576} for bucket 'GleamBookUsers0' 2022-02-07 21:46:05,178 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:05,181 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:05,181 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:05,339 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:05,484 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1736516 == 0 2022-02-07 21:46:05,486 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1736516 == 0. Received: {'172.23.121.123': 1736516} for bucket 'GleamBookUsers0' 2022-02-07 21:46:05,486 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:05,489 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:05,489 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:05,941 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1718093 == 0 2022-02-07 21:46:05,941 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1718093 == 0. Received: {'172.23.121.129': 1718093} for bucket 'GleamBookUsers0' 2022-02-07 21:46:05,941 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:05,944 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:05,944 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:06,155 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:06,404 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 783249 == 0 2022-02-07 21:46:06,404 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 783249 == 0. Received: {'172.23.121.126': 783249} for bucket 'GleamBookUsers0' 2022-02-07 21:46:06,404 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:06,407 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:06,407 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:07,588 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1441045 == 0 2022-02-07 21:46:07,588 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1441045 == 0. Received: {'172.23.120.170': 1441045} for bucket 'GleamBookUsers0' 2022-02-07 21:46:07,588 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:07,591 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:07,591 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:08,658 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:09,733 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:09,987 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1150972 == 0 2022-02-07 21:46:09,987 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1150972 == 0. Received: {'172.23.121.127': 1150972} for bucket 'GleamBookUsers0' 2022-02-07 21:46:09,987 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:09,990 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:09,990 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:10,115 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:10,183 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:10,489 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:10,921 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 1028121 == 0 2022-02-07 21:46:10,921 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1028121 == 0. Received: {'172.23.121.115': 1028121} for bucket 'GleamBookUsers0' 2022-02-07 21:46:10,923 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:10,924 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:10,924 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:10,944 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:11,407 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:11,657 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1271246 == 0 2022-02-07 21:46:11,657 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1271246 == 0. Received: {'172.23.121.128': 1271246} for bucket 'GleamBookUsers0' 2022-02-07 21:46:11,657 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:11,661 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:11,661 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:11,805 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1464030 == 0 2022-02-07 21:46:11,805 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1464030 == 0. Received: {'172.23.121.124': 1464030} for bucket 'GleamBookUsers0' 2022-02-07 21:46:11,805 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:11,809 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:11,809 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:12,099 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1720196 == 0 2022-02-07 21:46:12,099 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1720196 == 0. Received: {'172.23.121.123': 1720196} for bucket 'GleamBookUsers0' 2022-02-07 21:46:12,101 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:12,102 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:12,104 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:12,516 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 775921 == 0 2022-02-07 21:46:12,517 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 775921 == 0. Received: {'172.23.121.126': 775921} for bucket 'GleamBookUsers0' 2022-02-07 21:46:12,517 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:12,519 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:12,520 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:12,581 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1704638 == 0 2022-02-07 21:46:12,581 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1704638 == 0. Received: {'172.23.121.129': 1704638} for bucket 'GleamBookUsers0' 2022-02-07 21:46:12,582 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:12,584 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:12,584 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:12,592 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:14,017 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1376089 == 0 2022-02-07 21:46:14,019 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1376089 == 0. Received: {'172.23.120.170': 1376089} for bucket 'GleamBookUsers0' 2022-02-07 21:46:14,019 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:14,020 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:14,020 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:14,990 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:15,926 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:16,250 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1116839 == 0 2022-02-07 21:46:16,250 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1116839 == 0. Received: {'172.23.121.127': 1116839} for bucket 'GleamBookUsers0' 2022-02-07 21:46:16,250 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:16,253 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:16,253 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:16,661 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:16,809 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:17,053 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 990445 == 0 2022-02-07 21:46:17,053 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 990445 == 0. Received: {'172.23.121.115': 990445} for bucket 'GleamBookUsers0' 2022-02-07 21:46:17,055 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:17,055 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:17,056 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:17,105 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:17,520 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:17,585 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:18,023 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1239662 == 0 2022-02-07 21:46:18,023 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1239662 == 0. Received: {'172.23.121.128': 1239662} for bucket 'GleamBookUsers0' 2022-02-07 21:46:18,023 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:18,026 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:18,026 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:18,285 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1431242 == 0 2022-02-07 21:46:18,285 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1431242 == 0. Received: {'172.23.121.124': 1431242} for bucket 'GleamBookUsers0' 2022-02-07 21:46:18,285 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:18,288 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:18,288 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:18,678 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 742431 == 0 2022-02-07 21:46:18,680 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 742431 == 0. Received: {'172.23.121.126': 742431} for bucket 'GleamBookUsers0' 2022-02-07 21:46:18,680 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:18,681 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:18,684 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:18,904 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1703834 == 0 2022-02-07 21:46:18,904 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1703834 == 0. Received: {'172.23.121.123': 1703834} for bucket 'GleamBookUsers0' 2022-02-07 21:46:18,904 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:18,907 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:18,908 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:19,022 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:19,240 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1691140 == 0 2022-02-07 21:46:19,240 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1691140 == 0. Received: {'172.23.121.129': 1691140} for bucket 'GleamBookUsers0' 2022-02-07 21:46:19,240 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:19,246 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:19,247 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:20,413 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1341399 == 0 2022-02-07 21:46:20,413 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1341399 == 0. Received: {'172.23.120.170': 1341399} for bucket 'GleamBookUsers0' 2022-02-07 21:46:20,414 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:20,415 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:20,415 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:21,253 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:22,056 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:22,519 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1099791 == 0 2022-02-07 21:46:22,519 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1099791 == 0. Received: {'172.23.121.127': 1099791} for bucket 'GleamBookUsers0' 2022-02-07 21:46:22,519 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:22,522 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:22,522 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:23,026 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:23,275 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 956975 == 0 2022-02-07 21:46:23,275 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 956975 == 0. Received: {'172.23.121.115': 956975} for bucket 'GleamBookUsers0' 2022-02-07 21:46:23,276 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:23,278 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:23,279 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:23,289 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:23,684 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:23,908 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:24,249 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:24,434 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1201352 == 0 2022-02-07 21:46:24,436 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1201352 == 0. Received: {'172.23.121.128': 1201352} for bucket 'GleamBookUsers0' 2022-02-07 21:46:24,436 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:24,437 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:24,437 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:24,776 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 693602 == 0 2022-02-07 21:46:24,776 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 693602 == 0. Received: {'172.23.121.126': 693602} for bucket 'GleamBookUsers0' 2022-02-07 21:46:24,776 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:24,779 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:24,779 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:24,822 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1412902 == 0 2022-02-07 21:46:24,822 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1412902 == 0. Received: {'172.23.121.124': 1412902} for bucket 'GleamBookUsers0' 2022-02-07 21:46:24,822 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:24,825 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:24,825 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:25,417 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:25,497 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1649754 == 0 2022-02-07 21:46:25,497 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1649754 == 0. Received: {'172.23.121.123': 1649754} for bucket 'GleamBookUsers0' 2022-02-07 21:46:25,499 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:25,500 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:25,500 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:25,851 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1645752 == 0 2022-02-07 21:46:25,851 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1645752 == 0. Received: {'172.23.121.129': 1645752} for bucket 'GleamBookUsers0' 2022-02-07 21:46:25,851 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:25,854 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:25,855 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:26,809 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1305035 == 0 2022-02-07 21:46:26,811 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1305035 == 0. Received: {'172.23.120.170': 1305035} for bucket 'GleamBookUsers0' 2022-02-07 21:46:26,811 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:26,812 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:26,813 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:27,529 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:28,279 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:28,776 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1065825 == 0 2022-02-07 21:46:28,776 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1065825 == 0. Received: {'172.23.121.127': 1065825} for bucket 'GleamBookUsers0' 2022-02-07 21:46:28,776 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:28,779 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:28,779 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:29,404 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 910783 == 0 2022-02-07 21:46:29,404 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 910783 == 0. Received: {'172.23.121.115': 910783} for bucket 'GleamBookUsers0' 2022-02-07 21:46:29,404 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:29,407 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:29,407 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:29,440 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:29,779 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:29,825 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:30,502 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:30,786 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1177206 == 0 2022-02-07 21:46:30,788 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1177206 == 0. Received: {'172.23.121.128': 1177206} for bucket 'GleamBookUsers0' 2022-02-07 21:46:30,788 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:30,789 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:30,789 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:30,855 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:30,874 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 671476 == 0 2022-02-07 21:46:30,875 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 671476 == 0. Received: {'172.23.121.126': 671476} for bucket 'GleamBookUsers0' 2022-02-07 21:46:30,875 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:30,877 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:30,877 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:31,226 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1389096 == 0 2022-02-07 21:46:31,226 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1389096 == 0. Received: {'172.23.121.124': 1389096} for bucket 'GleamBookUsers0' 2022-02-07 21:46:31,226 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:31,229 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:31,229 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:31,813 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:31,992 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1628826 == 0 2022-02-07 21:46:31,993 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1628826 == 0. Received: {'172.23.121.123': 1628826} for bucket 'GleamBookUsers0' 2022-02-07 21:46:31,993 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:31,994 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:31,994 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:32,428 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1634436 == 0 2022-02-07 21:46:32,428 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1634436 == 0. Received: {'172.23.121.129': 1634436} for bucket 'GleamBookUsers0' 2022-02-07 21:46:32,430 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:32,430 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:32,431 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:33,148 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1234835 == 0 2022-02-07 21:46:33,148 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1234835 == 0. Received: {'172.23.120.170': 1234835} for bucket 'GleamBookUsers0' 2022-02-07 21:46:33,148 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:33,151 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:33,151 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:33,779 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:34,407 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:34,994 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1028732 == 0 2022-02-07 21:46:34,994 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1028732 == 0. Received: {'172.23.121.127': 1028732} for bucket 'GleamBookUsers0' 2022-02-07 21:46:34,996 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:34,997 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:34,997 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:35,492 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 885165 == 0 2022-02-07 21:46:35,493 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 885165 == 0. Received: {'172.23.121.115': 885165} for bucket 'GleamBookUsers0' 2022-02-07 21:46:35,493 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:35,494 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:35,494 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:35,790 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:35,878 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:36,230 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:36,920 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 631708 == 0 2022-02-07 21:46:36,920 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 631708 == 0. Received: {'172.23.121.126': 631708} for bucket 'GleamBookUsers0' 2022-02-07 21:46:36,920 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:36,923 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:36,924 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:36,996 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:37,088 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1165193 == 0 2022-02-07 21:46:37,088 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1165193 == 0. Received: {'172.23.121.128': 1165193} for bucket 'GleamBookUsers0' 2022-02-07 21:46:37,089 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:37,091 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:37,092 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:37,433 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:37,663 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1372072 == 0 2022-02-07 21:46:37,663 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1372072 == 0. Received: {'172.23.121.124': 1372072} for bucket 'GleamBookUsers0' 2022-02-07 21:46:37,663 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:37,664 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:37,664 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:38,153 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:38,523 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1605168 == 0 2022-02-07 21:46:38,523 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1605168 == 0. Received: {'172.23.121.123': 1605168} for bucket 'GleamBookUsers0' 2022-02-07 21:46:38,523 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:38,526 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:38,526 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:38,957 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1594534 == 0 2022-02-07 21:46:38,957 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1594534 == 0. Received: {'172.23.121.129': 1594534} for bucket 'GleamBookUsers0' 2022-02-07 21:46:38,957 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:38,960 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:38,960 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:39,459 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1211261 == 0 2022-02-07 21:46:39,460 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1211261 == 0. Received: {'172.23.120.170': 1211261} for bucket 'GleamBookUsers0' 2022-02-07 21:46:39,460 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:39,461 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:39,461 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:39,999 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:40,496 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:41,219 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1020412 == 0 2022-02-07 21:46:41,220 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1020412 == 0. Received: {'172.23.121.127': 1020412} for bucket 'GleamBookUsers0' 2022-02-07 21:46:41,220 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:41,221 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:41,221 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:41,619 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 875083 == 0 2022-02-07 21:46:41,621 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 875083 == 0. Received: {'172.23.121.115': 875083} for bucket 'GleamBookUsers0' 2022-02-07 21:46:41,621 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:41,622 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:41,622 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:41,924 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:42,092 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:42,665 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:42,891 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 617906 == 0 2022-02-07 21:46:42,891 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 617906 == 0. Received: {'172.23.121.126': 617906} for bucket 'GleamBookUsers0' 2022-02-07 21:46:42,891 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:42,894 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:42,894 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:43,371 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1138099 == 0 2022-02-07 21:46:43,371 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1138099 == 0. Received: {'172.23.121.128': 1138099} for bucket 'GleamBookUsers0' 2022-02-07 21:46:43,372 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:43,375 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:43,375 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:43,526 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:43,960 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:44,016 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1333868 == 0 2022-02-07 21:46:44,016 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1333868 == 0. Received: {'172.23.121.124': 1333868} for bucket 'GleamBookUsers0' 2022-02-07 21:46:44,016 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:44,019 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:44,019 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:44,461 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:45,036 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1583543 == 0 2022-02-07 21:46:45,036 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1583543 == 0. Received: {'172.23.121.123': 1583543} for bucket 'GleamBookUsers0' 2022-02-07 21:46:45,036 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:45,039 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:45,039 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:45,503 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1561388 == 0 2022-02-07 21:46:45,503 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1561388 == 0. Received: {'172.23.121.129': 1561388} for bucket 'GleamBookUsers0' 2022-02-07 21:46:45,503 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:45,506 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:45,506 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:45,763 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1161069 == 0 2022-02-07 21:46:45,763 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1161069 == 0. Received: {'172.23.120.170': 1161069} for bucket 'GleamBookUsers0' 2022-02-07 21:46:45,763 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:45,765 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:45,766 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:46,223 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:46,624 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:47,440 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1005986 == 0 2022-02-07 21:46:47,440 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1005986 == 0. Received: {'172.23.121.127': 1005986} for bucket 'GleamBookUsers0' 2022-02-07 21:46:47,440 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:47,444 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:47,444 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:47,736 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 846286 == 0 2022-02-07 21:46:47,737 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 846286 == 0. Received: {'172.23.121.115': 846286} for bucket 'GleamBookUsers0' 2022-02-07 21:46:47,737 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:47,740 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:47,740 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:47,894 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:48,375 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:48,888 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 596550 == 0 2022-02-07 21:46:48,890 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 596550 == 0. Received: {'172.23.121.126': 596550} for bucket 'GleamBookUsers0' 2022-02-07 21:46:48,890 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:48,891 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:48,891 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:49,019 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:49,642 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1126521 == 0 2022-02-07 21:46:49,642 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1126521 == 0. Received: {'172.23.121.128': 1126521} for bucket 'GleamBookUsers0' 2022-02-07 21:46:49,642 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:49,644 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:49,645 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:50,040 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:50,381 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1287262 == 0 2022-02-07 21:46:50,381 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1287262 == 0. Received: {'172.23.121.124': 1287262} for bucket 'GleamBookUsers0' 2022-02-07 21:46:50,381 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:50,382 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:50,382 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:50,506 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:50,766 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:51,528 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1576051 == 0 2022-02-07 21:46:51,528 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1576051 == 0. Received: {'172.23.121.123': 1576051} for bucket 'GleamBookUsers0' 2022-02-07 21:46:51,528 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:51,529 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:51,529 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:52,088 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1131459 == 0 2022-02-07 21:46:52,088 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1131459 == 0. Received: {'172.23.120.170': 1131459} for bucket 'GleamBookUsers0' 2022-02-07 21:46:52,088 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:52,092 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:52,092 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:52,101 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1545098 == 0 2022-02-07 21:46:52,102 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1545098 == 0. Received: {'172.23.121.129': 1545098} for bucket 'GleamBookUsers0' 2022-02-07 21:46:52,102 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:52,105 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:52,105 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:52,444 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:52,740 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:53,802 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 975064 == 0 2022-02-07 21:46:53,803 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 975064 == 0. Received: {'172.23.121.127': 975064} for bucket 'GleamBookUsers0' 2022-02-07 21:46:53,803 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:53,806 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:53,806 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:53,877 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 823178 == 0 2022-02-07 21:46:53,877 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 823178 == 0. Received: {'172.23.121.115': 823178} for bucket 'GleamBookUsers0' 2022-02-07 21:46:53,877 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:53,878 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:53,880 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:53,892 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:54,645 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:54,809 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 559398 == 0 2022-02-07 21:46:54,809 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 559398 == 0. Received: {'172.23.121.126': 559398} for bucket 'GleamBookUsers0' 2022-02-07 21:46:54,809 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:54,812 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:54,812 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:55,384 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:55,871 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1101773 == 0 2022-02-07 21:46:55,872 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1101773 == 0. Received: {'172.23.121.128': 1101773} for bucket 'GleamBookUsers0' 2022-02-07 21:46:55,872 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:55,875 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:55,875 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:56,530 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:56,775 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1271712 == 0 2022-02-07 21:46:56,776 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1271712 == 0. Received: {'172.23.121.124': 1271712} for bucket 'GleamBookUsers0' 2022-02-07 21:46:56,776 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:56,778 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:56,778 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:57,092 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:57,105 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:58,000 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1532536 == 0 2022-02-07 21:46:58,000 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1532536 == 0. Received: {'172.23.121.123': 1532536} for bucket 'GleamBookUsers0' 2022-02-07 21:46:58,000 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:58,003 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:58,003 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:58,312 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1108837 == 0 2022-02-07 21:46:58,313 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1108837 == 0. Received: {'172.23.120.170': 1108837} for bucket 'GleamBookUsers0' 2022-02-07 21:46:58,313 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:58,315 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:58,315 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:58,693 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1531918 == 0 2022-02-07 21:46:58,693 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1531918 == 0. Received: {'172.23.121.129': 1531918} for bucket 'GleamBookUsers0' 2022-02-07 21:46:58,693 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:58,696 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:58,697 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:46:58,806 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:58,880 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:59,812 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:46:59,969 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 805798 == 0 2022-02-07 21:46:59,970 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 805798 == 0. Received: {'172.23.121.115': 805798} for bucket 'GleamBookUsers0' 2022-02-07 21:46:59,970 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:46:59,971 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:46:59,971 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:00,040 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 946300 == 0 2022-02-07 21:47:00,042 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 946300 == 0. Received: {'172.23.121.127': 946300} for bucket 'GleamBookUsers0' 2022-02-07 21:47:00,042 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:00,043 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:00,043 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:00,739 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 524816 == 0 2022-02-07 21:47:00,740 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 524816 == 0. Received: {'172.23.121.126': 524816} for bucket 'GleamBookUsers0' 2022-02-07 21:47:00,740 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:00,742 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:00,742 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:00,875 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:01,779 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:02,078 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1078944 == 0 2022-02-07 21:47:02,079 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1078944 == 0. Received: {'172.23.121.128': 1078944} for bucket 'GleamBookUsers0' 2022-02-07 21:47:02,079 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:02,082 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:02,082 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:03,003 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:03,019 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1233564 == 0 2022-02-07 21:47:03,019 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1233564 == 0. Received: {'172.23.121.124': 1233564} for bucket 'GleamBookUsers0' 2022-02-07 21:47:03,019 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:03,022 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:03,022 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:03,316 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:03,697 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:04,529 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1057792 == 0 2022-02-07 21:47:04,529 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1057792 == 0. Received: {'172.23.120.170': 1057792} for bucket 'GleamBookUsers0' 2022-02-07 21:47:04,530 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:04,532 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:04,532 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:04,568 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1519581 == 0 2022-02-07 21:47:04,568 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1519581 == 0. Received: {'172.23.121.123': 1519581} for bucket 'GleamBookUsers0' 2022-02-07 21:47:04,568 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:04,569 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:04,571 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:04,973 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:05,045 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:05,151 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1483486 == 0 2022-02-07 21:47:05,153 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1483486 == 0. Received: {'172.23.121.129': 1483486} for bucket 'GleamBookUsers0' 2022-02-07 21:47:05,153 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:05,154 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:05,154 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:05,743 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:06,009 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 772956 == 0 2022-02-07 21:47:06,009 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 772956 == 0. Received: {'172.23.121.115': 772956} for bucket 'GleamBookUsers0' 2022-02-07 21:47:06,009 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:06,012 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:06,012 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:06,230 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 930016 == 0 2022-02-07 21:47:06,230 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 930016 == 0. Received: {'172.23.121.127': 930016} for bucket 'GleamBookUsers0' 2022-02-07 21:47:06,230 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:06,233 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:06,233 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:06,653 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 497436 == 0 2022-02-07 21:47:06,654 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 497436 == 0. Received: {'172.23.121.126': 497436} for bucket 'GleamBookUsers0' 2022-02-07 21:47:06,654 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:06,655 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:06,655 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:07,082 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:08,023 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:08,302 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1050352 == 0 2022-02-07 21:47:08,302 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1050352 == 0. Received: {'172.23.121.128': 1050352} for bucket 'GleamBookUsers0' 2022-02-07 21:47:08,302 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:08,305 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:08,305 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:09,328 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1198095 == 0 2022-02-07 21:47:09,328 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1198095 == 0. Received: {'172.23.121.124': 1198095} for bucket 'GleamBookUsers0' 2022-02-07 21:47:09,328 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:09,331 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:09,331 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:09,533 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:09,571 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:10,157 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:10,670 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1039257 == 0 2022-02-07 21:47:10,670 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1039257 == 0. Received: {'172.23.120.170': 1039257} for bucket 'GleamBookUsers0' 2022-02-07 21:47:10,670 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:10,673 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:10,673 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:11,007 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1498643 == 0 2022-02-07 21:47:11,007 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1498643 == 0. Received: {'172.23.121.123': 1498643} for bucket 'GleamBookUsers0' 2022-02-07 21:47:11,009 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:11,010 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:11,010 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:11,013 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:11,233 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:11,618 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1474010 == 0 2022-02-07 21:47:11,619 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1474010 == 0. Received: {'172.23.121.129': 1474010} for bucket 'GleamBookUsers0' 2022-02-07 21:47:11,619 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:11,621 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:11,621 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:11,657 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:12,029 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 750846 == 0 2022-02-07 21:47:12,029 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 750846 == 0. Received: {'172.23.121.115': 750846} for bucket 'GleamBookUsers0' 2022-02-07 21:47:12,029 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:12,032 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:12,032 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:12,352 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 894126 == 0 2022-02-07 21:47:12,352 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 894126 == 0. Received: {'172.23.121.127': 894126} for bucket 'GleamBookUsers0' 2022-02-07 21:47:12,352 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:12,355 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:12,355 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:12,543 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 477136 == 0 2022-02-07 21:47:12,543 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 477136 == 0. Received: {'172.23.121.126': 477136} for bucket 'GleamBookUsers0' 2022-02-07 21:47:12,543 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:12,546 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:12,546 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:13,305 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:14,332 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:14,510 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1011318 == 0 2022-02-07 21:47:14,510 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1011318 == 0. Received: {'172.23.121.128': 1011318} for bucket 'GleamBookUsers0' 2022-02-07 21:47:14,510 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:14,513 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:14,513 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:15,658 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1191803 == 0 2022-02-07 21:47:15,660 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1191803 == 0. Received: {'172.23.121.124': 1191803} for bucket 'GleamBookUsers0' 2022-02-07 21:47:15,660 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:15,661 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:15,661 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:15,673 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:16,012 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:16,622 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:16,780 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 1023207 == 0 2022-02-07 21:47:16,782 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1023207 == 0. Received: {'172.23.120.170': 1023207} for bucket 'GleamBookUsers0' 2022-02-07 21:47:16,782 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:16,783 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:16,783 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:17,032 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:17,355 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:17,466 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1457825 == 0 2022-02-07 21:47:17,467 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1457825 == 0. Received: {'172.23.121.123': 1457825} for bucket 'GleamBookUsers0' 2022-02-07 21:47:17,467 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:17,469 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:17,469 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:17,546 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:18,048 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 707734 == 0 2022-02-07 21:47:18,049 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 707734 == 0. Received: {'172.23.121.115': 707734} for bucket 'GleamBookUsers0' 2022-02-07 21:47:18,049 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:18,051 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:18,051 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:18,115 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1462162 == 0 2022-02-07 21:47:18,115 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1462162 == 0. Received: {'172.23.121.129': 1462162} for bucket 'GleamBookUsers0' 2022-02-07 21:47:18,115 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:18,118 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:18,118 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:18,506 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 453602 == 0 2022-02-07 21:47:18,507 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 453602 == 0. Received: {'172.23.121.126': 453602} for bucket 'GleamBookUsers0' 2022-02-07 21:47:18,507 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:18,509 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:18,509 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:18,542 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 874630 == 0 2022-02-07 21:47:18,542 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 874630 == 0. Received: {'172.23.121.127': 874630} for bucket 'GleamBookUsers0' 2022-02-07 21:47:18,543 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:18,546 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:18,546 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:19,513 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:20,663 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:20,717 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 987980 == 0 2022-02-07 21:47:20,717 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 987980 == 0. Received: {'172.23.121.128': 987980} for bucket 'GleamBookUsers0' 2022-02-07 21:47:20,717 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:20,720 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:20,720 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:21,785 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:21,901 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1173243 == 0 2022-02-07 21:47:21,901 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1173243 == 0. Received: {'172.23.121.124': 1173243} for bucket 'GleamBookUsers0' 2022-02-07 21:47:21,901 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:21,903 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:21,903 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:22,470 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:22,894 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 985879 == 0 2022-02-07 21:47:22,894 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 985879 == 0. Received: {'172.23.120.170': 985879} for bucket 'GleamBookUsers0' 2022-02-07 21:47:22,894 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:22,897 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:22,897 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:23,052 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:23,118 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:23,510 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:23,549 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:23,976 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1431909 == 0 2022-02-07 21:47:23,976 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1431909 == 0. Received: {'172.23.121.123': 1431909} for bucket 'GleamBookUsers0' 2022-02-07 21:47:23,976 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:23,979 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:23,980 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:24,082 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 694364 == 0 2022-02-07 21:47:24,082 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 694364 == 0. Received: {'172.23.121.115': 694364} for bucket 'GleamBookUsers0' 2022-02-07 21:47:24,082 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:24,085 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:24,085 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:24,428 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 440824 == 0 2022-02-07 21:47:24,430 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 440824 == 0. Received: {'172.23.121.126': 440824} for bucket 'GleamBookUsers0' 2022-02-07 21:47:24,430 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:24,431 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:24,433 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:24,769 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 828262 == 0 2022-02-07 21:47:24,769 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 828262 == 0. Received: {'172.23.121.127': 828262} for bucket 'GleamBookUsers0' 2022-02-07 21:47:24,769 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:24,805 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:24,805 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:24,825 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1446150 == 0 2022-02-07 21:47:24,825 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1446150 == 0. Received: {'172.23.121.129': 1446150} for bucket 'GleamBookUsers0' 2022-02-07 21:47:24,825 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:24,828 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:24,828 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:25,720 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:26,898 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 972722 == 0 2022-02-07 21:47:26,898 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 972722 == 0. Received: {'172.23.121.128': 972722} for bucket 'GleamBookUsers0' 2022-02-07 21:47:26,898 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:26,901 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:26,901 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:26,904 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:27,898 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:28,108 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1142342 == 0 2022-02-07 21:47:28,108 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1142342 == 0. Received: {'172.23.121.124': 1142342} for bucket 'GleamBookUsers0' 2022-02-07 21:47:28,108 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:28,111 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:28,111 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:28,961 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 951734 == 0 2022-02-07 21:47:28,961 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 951734 == 0. Received: {'172.23.120.170': 951734} for bucket 'GleamBookUsers0' 2022-02-07 21:47:28,961 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:28,963 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:28,963 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:28,980 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:29,085 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:29,433 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:29,805 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:29,828 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:30,046 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 661690 == 0 2022-02-07 21:47:30,048 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 661690 == 0. Received: {'172.23.121.115': 661690} for bucket 'GleamBookUsers0' 2022-02-07 21:47:30,048 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:30,049 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:30,049 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:30,273 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 408212 == 0 2022-02-07 21:47:30,275 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 408212 == 0. Received: {'172.23.121.126': 408212} for bucket 'GleamBookUsers0' 2022-02-07 21:47:30,275 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:30,276 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:30,276 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:30,510 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1415006 == 0 2022-02-07 21:47:30,510 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1415006 == 0. Received: {'172.23.121.123': 1415006} for bucket 'GleamBookUsers0' 2022-02-07 21:47:30,510 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:30,513 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:30,513 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:30,963 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 813029 == 0 2022-02-07 21:47:30,963 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 813029 == 0. Received: {'172.23.121.127': 813029} for bucket 'GleamBookUsers0' 2022-02-07 21:47:30,963 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:30,966 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:30,966 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:31,240 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1401850 == 0 2022-02-07 21:47:31,240 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1401850 == 0. Received: {'172.23.121.129': 1401850} for bucket 'GleamBookUsers0' 2022-02-07 21:47:31,242 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:31,243 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:31,243 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:31,901 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:33,042 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 929126 == 0 2022-02-07 21:47:33,042 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 929126 == 0. Received: {'172.23.121.128': 929126} for bucket 'GleamBookUsers0' 2022-02-07 21:47:33,042 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:33,043 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:33,045 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:33,111 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:33,964 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:34,359 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1117813 == 0 2022-02-07 21:47:34,361 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1117813 == 0. Received: {'172.23.121.124': 1117813} for bucket 'GleamBookUsers0' 2022-02-07 21:47:34,361 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:34,362 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:34,362 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:35,051 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:35,061 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 915202 == 0 2022-02-07 21:47:35,062 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 915202 == 0. Received: {'172.23.120.170': 915202} for bucket 'GleamBookUsers0' 2022-02-07 21:47:35,062 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:35,063 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:35,063 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:35,278 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:35,513 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:35,966 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:36,042 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 646132 == 0 2022-02-07 21:47:36,042 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 646132 == 0. Received: {'172.23.121.115': 646132} for bucket 'GleamBookUsers0' 2022-02-07 21:47:36,042 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:36,045 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:36,045 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:36,119 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 378270 == 0 2022-02-07 21:47:36,121 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 378270 == 0. Received: {'172.23.121.126': 378270} for bucket 'GleamBookUsers0' 2022-02-07 21:47:36,121 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:36,122 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:36,122 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:36,246 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:36,908 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1392684 == 0 2022-02-07 21:47:36,910 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1392684 == 0. Received: {'172.23.121.123': 1392684} for bucket 'GleamBookUsers0' 2022-02-07 21:47:36,910 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:36,911 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:36,911 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:37,043 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 782355 == 0 2022-02-07 21:47:37,043 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 782355 == 0. Received: {'172.23.121.127': 782355} for bucket 'GleamBookUsers0' 2022-02-07 21:47:37,043 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:37,046 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:37,046 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:37,696 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1383470 == 0 2022-02-07 21:47:37,697 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1383470 == 0. Received: {'172.23.121.129': 1383470} for bucket 'GleamBookUsers0' 2022-02-07 21:47:37,697 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:37,698 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:37,698 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:38,045 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:39,128 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 909206 == 0 2022-02-07 21:47:39,130 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 909206 == 0. Received: {'172.23.121.128': 909206} for bucket 'GleamBookUsers0' 2022-02-07 21:47:39,130 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:39,131 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:39,131 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:39,364 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:40,065 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:40,617 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1102216 == 0 2022-02-07 21:47:40,617 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1102216 == 0. Received: {'172.23.121.124': 1102216} for bucket 'GleamBookUsers0' 2022-02-07 21:47:40,618 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:40,619 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:40,619 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:41,046 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:41,081 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 868116 == 0 2022-02-07 21:47:41,081 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 868116 == 0. Received: {'172.23.120.170': 868116} for bucket 'GleamBookUsers0' 2022-02-07 21:47:41,081 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:41,082 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:41,084 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:41,122 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:41,882 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 356264 == 0 2022-02-07 21:47:41,884 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 356264 == 0. Received: {'172.23.121.126': 356264} for bucket 'GleamBookUsers0' 2022-02-07 21:47:41,884 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:41,885 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:41,885 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:41,913 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:42,003 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 618728 == 0 2022-02-07 21:47:42,003 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 618728 == 0. Received: {'172.23.121.115': 618728} for bucket 'GleamBookUsers0' 2022-02-07 21:47:42,003 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:42,006 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:42,006 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:42,046 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:42,700 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:43,187 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 754325 == 0 2022-02-07 21:47:43,187 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 754325 == 0. Received: {'172.23.121.127': 754325} for bucket 'GleamBookUsers0' 2022-02-07 21:47:43,187 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:43,190 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:43,190 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:43,341 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1359326 == 0 2022-02-07 21:47:43,342 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1359326 == 0. Received: {'172.23.121.123': 1359326} for bucket 'GleamBookUsers0' 2022-02-07 21:47:43,342 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:43,345 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:43,345 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:44,082 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1377072 == 0 2022-02-07 21:47:44,082 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1377072 == 0. Received: {'172.23.121.129': 1377072} for bucket 'GleamBookUsers0' 2022-02-07 21:47:44,082 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:44,085 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:44,085 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:44,132 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:45,200 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 881113 == 0 2022-02-07 21:47:45,200 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 881113 == 0. Received: {'172.23.121.128': 881113} for bucket 'GleamBookUsers0' 2022-02-07 21:47:45,200 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:45,203 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:45,203 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:45,621 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:46,085 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:46,816 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1081226 == 0 2022-02-07 21:47:46,816 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1081226 == 0. Received: {'172.23.121.124': 1081226} for bucket 'GleamBookUsers0' 2022-02-07 21:47:46,818 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:46,819 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:46,819 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:46,887 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:47,006 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:47,052 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 833080 == 0 2022-02-07 21:47:47,052 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 833080 == 0. Received: {'172.23.120.170': 833080} for bucket 'GleamBookUsers0' 2022-02-07 21:47:47,053 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:47,055 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:47,055 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:47,617 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 329502 == 0 2022-02-07 21:47:47,617 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 329502 == 0. Received: {'172.23.121.126': 329502} for bucket 'GleamBookUsers0' 2022-02-07 21:47:47,617 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:47,618 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:47,618 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:47,930 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 603892 == 0 2022-02-07 21:47:47,930 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 603892 == 0. Received: {'172.23.121.115': 603892} for bucket 'GleamBookUsers0' 2022-02-07 21:47:47,930 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:47,933 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:47,933 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:48,190 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:48,345 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:49,085 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:49,255 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 735442 == 0 2022-02-07 21:47:49,255 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 735442 == 0. Received: {'172.23.121.127': 735442} for bucket 'GleamBookUsers0' 2022-02-07 21:47:49,255 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:49,256 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:49,257 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:49,671 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1338004 == 0 2022-02-07 21:47:49,671 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1338004 == 0. Received: {'172.23.121.123': 1338004} for bucket 'GleamBookUsers0' 2022-02-07 21:47:49,671 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:49,674 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:49,674 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:50,203 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:50,423 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1338629 == 0 2022-02-07 21:47:50,423 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1338629 == 0. Received: {'172.23.121.129': 1338629} for bucket 'GleamBookUsers0' 2022-02-07 21:47:50,424 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:50,426 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:50,426 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:51,273 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 864909 == 0 2022-02-07 21:47:51,275 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 864909 == 0. Received: {'172.23.121.128': 864909} for bucket 'GleamBookUsers0' 2022-02-07 21:47:51,275 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:51,276 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:51,276 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:51,821 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:52,055 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:52,621 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:52,934 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:53,049 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1057712 == 0 2022-02-07 21:47:53,049 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1057712 == 0. Received: {'172.23.121.124': 1057712} for bucket 'GleamBookUsers0' 2022-02-07 21:47:53,049 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:53,052 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:53,052 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:53,081 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 811982 == 0 2022-02-07 21:47:53,082 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 811982 == 0. Received: {'172.23.120.170': 811982} for bucket 'GleamBookUsers0' 2022-02-07 21:47:53,082 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:53,084 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:53,084 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:53,328 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 308098 == 0 2022-02-07 21:47:53,329 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 308098 == 0. Received: {'172.23.121.126': 308098} for bucket 'GleamBookUsers0' 2022-02-07 21:47:53,329 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:53,332 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:53,332 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:53,842 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 578918 == 0 2022-02-07 21:47:53,844 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 578918 == 0. Received: {'172.23.121.115': 578918} for bucket 'GleamBookUsers0' 2022-02-07 21:47:53,844 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:53,845 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:53,845 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:54,257 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:54,674 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:55,289 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 704744 == 0 2022-02-07 21:47:55,289 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 704744 == 0. Received: {'172.23.121.127': 704744} for bucket 'GleamBookUsers0' 2022-02-07 21:47:55,289 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:55,290 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:55,290 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:55,426 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:55,967 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1319647 == 0 2022-02-07 21:47:55,967 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1319647 == 0. Received: {'172.23.121.123': 1319647} for bucket 'GleamBookUsers0' 2022-02-07 21:47:55,967 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:55,970 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:55,970 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:56,276 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:56,779 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1310753 == 0 2022-02-07 21:47:56,780 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1310753 == 0. Received: {'172.23.121.129': 1310753} for bucket 'GleamBookUsers0' 2022-02-07 21:47:56,780 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:56,782 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:56,782 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:57,365 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 837487 == 0 2022-02-07 21:47:57,365 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 837487 == 0. Received: {'172.23.121.128': 837487} for bucket 'GleamBookUsers0' 2022-02-07 21:47:57,365 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:57,368 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:57,368 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:58,052 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:58,084 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:58,332 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:58,846 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:47:59,108 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 283227 == 0 2022-02-07 21:47:59,108 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 283227 == 0. Received: {'172.23.121.126': 283227} for bucket 'GleamBookUsers0' 2022-02-07 21:47:59,108 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:59,111 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:59,111 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:59,119 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 753108 == 0 2022-02-07 21:47:59,119 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 753108 == 0. Received: {'172.23.120.170': 753108} for bucket 'GleamBookUsers0' 2022-02-07 21:47:59,119 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:59,121 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:59,122 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:59,309 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1043810 == 0 2022-02-07 21:47:59,309 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1043810 == 0. Received: {'172.23.121.124': 1043810} for bucket 'GleamBookUsers0' 2022-02-07 21:47:59,309 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:59,312 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:59,312 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:47:59,763 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 557391 == 0 2022-02-07 21:47:59,765 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 557391 == 0. Received: {'172.23.121.115': 557391} for bucket 'GleamBookUsers0' 2022-02-07 21:47:59,765 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:47:59,766 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:47:59,766 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:00,292 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:00,970 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:01,279 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 687507 == 0 2022-02-07 21:48:01,279 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 687507 == 0. Received: {'172.23.121.127': 687507} for bucket 'GleamBookUsers0' 2022-02-07 21:48:01,279 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:01,282 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:01,282 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:01,783 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:02,240 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1302431 == 0 2022-02-07 21:48:02,242 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1302431 == 0. Received: {'172.23.121.123': 1302431} for bucket 'GleamBookUsers0' 2022-02-07 21:48:02,242 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:02,243 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:02,243 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:02,368 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:03,144 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1298635 == 0 2022-02-07 21:48:03,145 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1298635 == 0. Received: {'172.23.121.129': 1298635} for bucket 'GleamBookUsers0' 2022-02-07 21:48:03,145 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:03,148 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:03,148 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:03,403 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 809847 == 0 2022-02-07 21:48:03,403 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 809847 == 0. Received: {'172.23.121.128': 809847} for bucket 'GleamBookUsers0' 2022-02-07 21:48:03,404 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:03,405 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:03,405 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:04,111 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:04,124 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:04,312 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:04,763 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 261073 == 0 2022-02-07 21:48:04,763 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 261073 == 0. Received: {'172.23.121.126': 261073} for bucket 'GleamBookUsers0' 2022-02-07 21:48:04,763 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:04,766 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:04,766 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:04,766 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:05,019 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 730156 == 0 2022-02-07 21:48:05,019 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 730156 == 0. Received: {'172.23.120.170': 730156} for bucket 'GleamBookUsers0' 2022-02-07 21:48:05,019 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:05,022 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:05,022 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:05,415 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1004332 == 0 2022-02-07 21:48:05,417 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1004332 == 0. Received: {'172.23.121.124': 1004332} for bucket 'GleamBookUsers0' 2022-02-07 21:48:05,417 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:05,418 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:05,418 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:05,648 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 540581 == 0 2022-02-07 21:48:05,648 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 540581 == 0. Received: {'172.23.121.115': 540581} for bucket 'GleamBookUsers0' 2022-02-07 21:48:05,648 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:05,651 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:05,651 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:06,282 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:07,244 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:07,246 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 670653 == 0 2022-02-07 21:48:07,246 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 670653 == 0. Received: {'172.23.121.127': 670653} for bucket 'GleamBookUsers0' 2022-02-07 21:48:07,246 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:07,249 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:07,249 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:08,148 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:08,405 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:08,519 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1268363 == 0 2022-02-07 21:48:08,519 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1268363 == 0. Received: {'172.23.121.123': 1268363} for bucket 'GleamBookUsers0' 2022-02-07 21:48:08,519 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:08,520 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:08,520 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:09,473 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 804231 == 0 2022-02-07 21:48:09,474 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 804231 == 0. Received: {'172.23.121.128': 804231} for bucket 'GleamBookUsers0' 2022-02-07 21:48:09,474 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:09,477 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:09,477 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:09,542 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1266954 == 0 2022-02-07 21:48:09,542 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1266954 == 0. Received: {'172.23.121.129': 1266954} for bucket 'GleamBookUsers0' 2022-02-07 21:48:09,542 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:09,545 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:09,546 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:09,766 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:10,023 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:10,411 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 240601 == 0 2022-02-07 21:48:10,411 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 240601 == 0. Received: {'172.23.121.126': 240601} for bucket 'GleamBookUsers0' 2022-02-07 21:48:10,411 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:10,414 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:10,414 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:10,418 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:10,651 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:10,884 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 706942 == 0 2022-02-07 21:48:10,884 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 706942 == 0. Received: {'172.23.120.170': 706942} for bucket 'GleamBookUsers0' 2022-02-07 21:48:10,884 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:10,887 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:10,888 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:11,588 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 512235 == 0 2022-02-07 21:48:11,589 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 512235 == 0. Received: {'172.23.121.115': 512235} for bucket 'GleamBookUsers0' 2022-02-07 21:48:11,589 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:11,592 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:11,592 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:11,660 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 985326 == 0 2022-02-07 21:48:11,660 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 985326 == 0. Received: {'172.23.121.124': 985326} for bucket 'GleamBookUsers0' 2022-02-07 21:48:11,660 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:11,661 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:11,661 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:12,249 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:13,193 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 651627 == 0 2022-02-07 21:48:13,193 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 651627 == 0. Received: {'172.23.121.127': 651627} for bucket 'GleamBookUsers0' 2022-02-07 21:48:13,193 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:13,194 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:13,194 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:13,522 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:14,479 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:14,546 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:14,786 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1253583 == 0 2022-02-07 21:48:14,788 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1253583 == 0. Received: {'172.23.121.123': 1253583} for bucket 'GleamBookUsers0' 2022-02-07 21:48:14,788 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:14,789 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:14,789 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:15,414 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:15,459 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 783708 == 0 2022-02-07 21:48:15,460 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 783708 == 0. Received: {'172.23.121.128': 783708} for bucket 'GleamBookUsers0' 2022-02-07 21:48:15,460 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:15,463 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:15,463 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:15,831 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1240914 == 0 2022-02-07 21:48:15,832 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1240914 == 0. Received: {'172.23.121.129': 1240914} for bucket 'GleamBookUsers0' 2022-02-07 21:48:15,832 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:15,834 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:15,835 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:15,888 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:16,053 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 212323 == 0 2022-02-07 21:48:16,053 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 212323 == 0. Received: {'172.23.121.126': 212323} for bucket 'GleamBookUsers0' 2022-02-07 21:48:16,053 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:16,055 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:16,056 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:16,594 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:16,663 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:16,743 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 680370 == 0 2022-02-07 21:48:16,743 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 680370 == 0. Received: {'172.23.120.170': 680370} for bucket 'GleamBookUsers0' 2022-02-07 21:48:16,743 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:16,746 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:16,746 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:17,464 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 493203 == 0 2022-02-07 21:48:17,466 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 493203 == 0. Received: {'172.23.121.115': 493203} for bucket 'GleamBookUsers0' 2022-02-07 21:48:17,466 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:17,467 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:17,469 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:17,875 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 948804 == 0 2022-02-07 21:48:17,875 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 948804 == 0. Received: {'172.23.121.124': 948804} for bucket 'GleamBookUsers0' 2022-02-07 21:48:17,875 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:17,877 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:17,878 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:18,196 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:19,112 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 632519 == 0 2022-02-07 21:48:19,112 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 632519 == 0. Received: {'172.23.121.127': 632519} for bucket 'GleamBookUsers0' 2022-02-07 21:48:19,112 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:19,115 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:19,115 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:19,789 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:20,463 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:20,835 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:21,022 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1214450 == 0 2022-02-07 21:48:21,023 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1214450 == 0. Received: {'172.23.121.123': 1214450} for bucket 'GleamBookUsers0' 2022-02-07 21:48:21,023 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:21,025 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:21,025 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:21,056 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:21,414 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 739778 == 0 2022-02-07 21:48:21,414 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 739778 == 0. Received: {'172.23.121.128': 739778} for bucket 'GleamBookUsers0' 2022-02-07 21:48:21,414 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:21,415 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:21,415 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:21,655 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 185917 == 0 2022-02-07 21:48:21,655 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 185917 == 0. Received: {'172.23.121.126': 185917} for bucket 'GleamBookUsers0' 2022-02-07 21:48:21,655 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:21,657 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:21,657 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:21,746 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:22,138 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1220672 == 0 2022-02-07 21:48:22,138 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1220672 == 0. Received: {'172.23.121.129': 1220672} for bucket 'GleamBookUsers0' 2022-02-07 21:48:22,138 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:22,141 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:22,141 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:22,565 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:22,654 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 667528 == 0 2022-02-07 21:48:22,654 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 667528 == 0. Received: {'172.23.120.170': 667528} for bucket 'GleamBookUsers0' 2022-02-07 21:48:22,654 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:22,657 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:22,657 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:22,878 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:23,382 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 464473 == 0 2022-02-07 21:48:23,382 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 464473 == 0. Received: {'172.23.121.115': 464473} for bucket 'GleamBookUsers0' 2022-02-07 21:48:23,382 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:23,384 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:23,384 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:23,980 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 926201 == 0 2022-02-07 21:48:23,980 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 926201 == 0. Received: {'172.23.121.124': 926201} for bucket 'GleamBookUsers0' 2022-02-07 21:48:23,980 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:23,983 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:23,983 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:24,115 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:24,996 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 600059 == 0 2022-02-07 21:48:24,997 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 600059 == 0. Received: {'172.23.121.127': 600059} for bucket 'GleamBookUsers0' 2022-02-07 21:48:24,997 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:25,000 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:25,000 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:26,025 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:26,417 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:26,658 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:27,141 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:27,243 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1187296 == 0 2022-02-07 21:48:27,243 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1187296 == 0. Received: {'172.23.121.123': 1187296} for bucket 'GleamBookUsers0' 2022-02-07 21:48:27,243 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:27,246 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:27,246 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:27,308 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 154228 == 0 2022-02-07 21:48:27,308 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 154228 == 0. Received: {'172.23.121.126': 154228} for bucket 'GleamBookUsers0' 2022-02-07 21:48:27,309 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:27,311 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:27,311 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:27,430 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 722232 == 0 2022-02-07 21:48:27,430 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 722232 == 0. Received: {'172.23.121.128': 722232} for bucket 'GleamBookUsers0' 2022-02-07 21:48:27,431 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:27,434 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:27,434 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:27,657 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:28,385 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:28,454 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1209654 == 0 2022-02-07 21:48:28,454 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1209654 == 0. Received: {'172.23.121.129': 1209654} for bucket 'GleamBookUsers0' 2022-02-07 21:48:28,454 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:28,457 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:28,457 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:28,516 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 633634 == 0 2022-02-07 21:48:28,517 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 633634 == 0. Received: {'172.23.120.170': 633634} for bucket 'GleamBookUsers0' 2022-02-07 21:48:28,517 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:28,519 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:28,519 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:28,984 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:29,181 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 446733 == 0 2022-02-07 21:48:29,181 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 446733 == 0. Received: {'172.23.121.115': 446733} for bucket 'GleamBookUsers0' 2022-02-07 21:48:29,183 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:29,184 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:29,184 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:30,000 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:30,028 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 910015 == 0 2022-02-07 21:48:30,029 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 910015 == 0. Received: {'172.23.121.124': 910015} for bucket 'GleamBookUsers0' 2022-02-07 21:48:30,029 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:30,030 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:30,030 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:30,865 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 589926 == 0 2022-02-07 21:48:30,865 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 589926 == 0. Received: {'172.23.121.127': 589926} for bucket 'GleamBookUsers0' 2022-02-07 21:48:30,865 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:30,867 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:30,867 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:32,246 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:32,311 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:32,434 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:32,940 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 139062 == 0 2022-02-07 21:48:32,940 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 139062 == 0. Received: {'172.23.121.126': 139062} for bucket 'GleamBookUsers0' 2022-02-07 21:48:32,940 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:32,943 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:32,943 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:33,459 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:33,460 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 699730 == 0 2022-02-07 21:48:33,460 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 699730 == 0. Received: {'172.23.121.128': 699730} for bucket 'GleamBookUsers0' 2022-02-07 21:48:33,460 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:33,463 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:33,463 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:33,467 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1178714 == 0 2022-02-07 21:48:33,467 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1178714 == 0. Received: {'172.23.121.123': 1178714} for bucket 'GleamBookUsers0' 2022-02-07 21:48:33,467 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:33,470 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:33,470 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:33,523 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:34,184 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:34,286 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 577052 == 0 2022-02-07 21:48:34,288 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 577052 == 0. Received: {'172.23.120.170': 577052} for bucket 'GleamBookUsers0' 2022-02-07 21:48:34,288 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:34,289 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:34,289 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:34,680 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1168510 == 0 2022-02-07 21:48:34,681 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1168510 == 0. Received: {'172.23.121.129': 1168510} for bucket 'GleamBookUsers0' 2022-02-07 21:48:34,681 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:34,684 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:34,684 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:34,980 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 435859 == 0 2022-02-07 21:48:34,980 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 435859 == 0. Received: {'172.23.121.115': 435859} for bucket 'GleamBookUsers0' 2022-02-07 21:48:34,980 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:34,983 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:34,983 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:35,030 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:35,868 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:36,025 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 889659 == 0 2022-02-07 21:48:36,025 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 889659 == 0. Received: {'172.23.121.124': 889659} for bucket 'GleamBookUsers0' 2022-02-07 21:48:36,026 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:36,028 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:36,028 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:36,726 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 571118 == 0 2022-02-07 21:48:36,726 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 571118 == 0. Received: {'172.23.121.127': 571118} for bucket 'GleamBookUsers0' 2022-02-07 21:48:36,726 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:36,727 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:36,729 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:37,944 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:38,463 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:38,473 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:38,476 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 108246 == 0 2022-02-07 21:48:38,476 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 108246 == 0. Received: {'172.23.121.126': 108246} for bucket 'GleamBookUsers0' 2022-02-07 21:48:38,476 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:38,479 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:38,479 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:39,290 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:39,407 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 687464 == 0 2022-02-07 21:48:39,408 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 687464 == 0. Received: {'172.23.121.128': 687464} for bucket 'GleamBookUsers0' 2022-02-07 21:48:39,408 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:39,411 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:39,411 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:39,621 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1146642 == 0 2022-02-07 21:48:39,621 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1146642 == 0. Received: {'172.23.121.123': 1146642} for bucket 'GleamBookUsers0' 2022-02-07 21:48:39,622 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:39,624 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:39,625 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:39,684 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:39,983 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:40,043 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 559245 == 0 2022-02-07 21:48:40,043 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 559245 == 0. Received: {'172.23.120.170': 559245} for bucket 'GleamBookUsers0' 2022-02-07 21:48:40,043 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:40,046 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:40,046 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:40,878 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 400892 == 0 2022-02-07 21:48:40,878 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 400892 == 0. Received: {'172.23.121.115': 400892} for bucket 'GleamBookUsers0' 2022-02-07 21:48:40,878 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:40,881 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:40,881 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:41,007 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1159480 == 0 2022-02-07 21:48:41,007 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1159480 == 0. Received: {'172.23.121.129': 1159480} for bucket 'GleamBookUsers0' 2022-02-07 21:48:41,007 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:41,010 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:41,010 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:41,028 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:41,729 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:42,043 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 884563 == 0 2022-02-07 21:48:42,043 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 884563 == 0. Received: {'172.23.121.124': 884563} for bucket 'GleamBookUsers0' 2022-02-07 21:48:42,043 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:42,046 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:42,046 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:42,549 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 519486 == 0 2022-02-07 21:48:42,549 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 519486 == 0. Received: {'172.23.121.127': 519486} for bucket 'GleamBookUsers0' 2022-02-07 21:48:42,549 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:42,552 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:42,552 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:43,480 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:44,003 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 91489 == 0 2022-02-07 21:48:44,003 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 91489 == 0. Received: {'172.23.121.126': 91489} for bucket 'GleamBookUsers0' 2022-02-07 21:48:44,003 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:44,006 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:44,006 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:44,411 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:44,625 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:45,046 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:45,316 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 652430 == 0 2022-02-07 21:48:45,316 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 652430 == 0. Received: {'172.23.121.128': 652430} for bucket 'GleamBookUsers0' 2022-02-07 21:48:45,316 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:45,319 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:45,319 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:45,881 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:45,901 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 535785 == 0 2022-02-07 21:48:45,901 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 535785 == 0. Received: {'172.23.120.170': 535785} for bucket 'GleamBookUsers0' 2022-02-07 21:48:45,901 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:45,904 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:45,904 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:45,928 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1139472 == 0 2022-02-07 21:48:45,928 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1139472 == 0. Received: {'172.23.121.123': 1139472} for bucket 'GleamBookUsers0' 2022-02-07 21:48:45,928 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:45,930 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:45,930 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:46,012 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:46,664 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 381784 == 0 2022-02-07 21:48:46,664 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 381784 == 0. Received: {'172.23.121.115': 381784} for bucket 'GleamBookUsers0' 2022-02-07 21:48:46,664 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:46,667 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:46,667 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:47,046 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:47,130 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1123194 == 0 2022-02-07 21:48:47,130 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1123194 == 0. Received: {'172.23.121.129': 1123194} for bucket 'GleamBookUsers0' 2022-02-07 21:48:47,130 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:47,131 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:47,131 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:47,553 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:48,036 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 872930 == 0 2022-02-07 21:48:48,036 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 872930 == 0. Received: {'172.23.121.124': 872930} for bucket 'GleamBookUsers0' 2022-02-07 21:48:48,036 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:48,039 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:48,039 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:48,332 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 504066 == 0 2022-02-07 21:48:48,332 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 504066 == 0. Received: {'172.23.121.127': 504066} for bucket 'GleamBookUsers0' 2022-02-07 21:48:48,332 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:48,335 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:48,335 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:49,006 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:49,494 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 71301 == 0 2022-02-07 21:48:49,496 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 71301 == 0. Received: {'172.23.121.126': 71301} for bucket 'GleamBookUsers0' 2022-02-07 21:48:49,496 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:49,499 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:49,499 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:50,319 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:50,904 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:50,931 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:51,213 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 630588 == 0 2022-02-07 21:48:51,214 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 630588 == 0. Received: {'172.23.121.128': 630588} for bucket 'GleamBookUsers0' 2022-02-07 21:48:51,214 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:51,217 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:51,217 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:51,671 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:51,694 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 510925 == 0 2022-02-07 21:48:51,694 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 510925 == 0. Received: {'172.23.120.170': 510925} for bucket 'GleamBookUsers0' 2022-02-07 21:48:51,696 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:51,697 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:51,698 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:52,075 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1129175 == 0 2022-02-07 21:48:52,075 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1129175 == 0. Received: {'172.23.121.123': 1129175} for bucket 'GleamBookUsers0' 2022-02-07 21:48:52,076 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:52,078 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:52,078 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:52,132 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:52,423 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 363947 == 0 2022-02-07 21:48:52,424 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 363947 == 0. Received: {'172.23.121.115': 363947} for bucket 'GleamBookUsers0' 2022-02-07 21:48:52,424 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:52,426 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:52,426 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:53,039 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:53,247 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1095210 == 0 2022-02-07 21:48:53,247 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1095210 == 0. Received: {'172.23.121.129': 1095210} for bucket 'GleamBookUsers0' 2022-02-07 21:48:53,249 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:53,250 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:53,250 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:53,335 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:54,029 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 857964 == 0 2022-02-07 21:48:54,030 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 857964 == 0. Received: {'172.23.121.124': 857964} for bucket 'GleamBookUsers0' 2022-02-07 21:48:54,030 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:54,032 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:54,032 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:54,118 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 481940 == 0 2022-02-07 21:48:54,118 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 481940 == 0. Received: {'172.23.121.127': 481940} for bucket 'GleamBookUsers0' 2022-02-07 21:48:54,118 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:54,121 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:54,121 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:54,500 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:54,956 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 46883 == 0 2022-02-07 21:48:54,956 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 46883 == 0. Received: {'172.23.121.126': 46883} for bucket 'GleamBookUsers0' 2022-02-07 21:48:54,957 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:54,957 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:54,959 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:56,217 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:56,700 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:57,045 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 600638 == 0 2022-02-07 21:48:57,046 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 600638 == 0. Received: {'172.23.121.128': 600638} for bucket 'GleamBookUsers0' 2022-02-07 21:48:57,046 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:57,048 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:57,048 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:57,078 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:57,398 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 470974 == 0 2022-02-07 21:48:57,398 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 470974 == 0. Received: {'172.23.120.170': 470974} for bucket 'GleamBookUsers0' 2022-02-07 21:48:57,398 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:57,401 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:57,401 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:57,426 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:58,219 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 349103 == 0 2022-02-07 21:48:58,219 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 349103 == 0. Received: {'172.23.121.115': 349103} for bucket 'GleamBookUsers0' 2022-02-07 21:48:58,219 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:58,221 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:58,221 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:58,246 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1096311 == 0 2022-02-07 21:48:58,247 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1096311 == 0. Received: {'172.23.121.123': 1096311} for bucket 'GleamBookUsers0' 2022-02-07 21:48:58,247 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:58,250 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:58,250 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:58,250 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:59,033 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:59,121 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:59,361 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1079630 == 0 2022-02-07 21:48:59,361 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1079630 == 0. Received: {'172.23.121.129': 1079630} for bucket 'GleamBookUsers0' 2022-02-07 21:48:59,361 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:59,364 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:59,364 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:59,908 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 463358 == 0 2022-02-07 21:48:59,908 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 463358 == 0. Received: {'172.23.121.127': 463358} for bucket 'GleamBookUsers0' 2022-02-07 21:48:59,908 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:59,911 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:59,911 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:48:59,960 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:48:59,993 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 818886 == 0 2022-02-07 21:48:59,994 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 818886 == 0. Received: {'172.23.121.124': 818886} for bucket 'GleamBookUsers0' 2022-02-07 21:48:59,994 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:48:59,996 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:48:59,996 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:00,384 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 33483 == 0 2022-02-07 21:49:00,384 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 33483 == 0. Received: {'172.23.121.126': 33483} for bucket 'GleamBookUsers0' 2022-02-07 21:49:00,384 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:00,385 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:00,387 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:02,048 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:02,401 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:02,869 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 583470 == 0 2022-02-07 21:49:02,871 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 583470 == 0. Received: {'172.23.121.128': 583470} for bucket 'GleamBookUsers0' 2022-02-07 21:49:02,871 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:02,872 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:02,872 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:03,056 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 464454 == 0 2022-02-07 21:49:03,058 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 464454 == 0. Received: {'172.23.120.170': 464454} for bucket 'GleamBookUsers0' 2022-02-07 21:49:03,058 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:03,059 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:03,059 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:03,221 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:03,250 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:03,970 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 315587 == 0 2022-02-07 21:49:03,970 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 315587 == 0. Received: {'172.23.121.115': 315587} for bucket 'GleamBookUsers0' 2022-02-07 21:49:03,970 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:03,971 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:03,973 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:04,358 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1092103 == 0 2022-02-07 21:49:04,358 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1092103 == 0. Received: {'172.23.121.123': 1092103} for bucket 'GleamBookUsers0' 2022-02-07 21:49:04,358 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:04,359 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:04,359 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:04,365 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:04,913 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:04,997 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:05,387 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:05,434 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1038462 == 0 2022-02-07 21:49:05,434 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1038462 == 0. Received: {'172.23.121.129': 1038462} for bucket 'GleamBookUsers0' 2022-02-07 21:49:05,434 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:05,437 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:05,437 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:05,621 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 453273 == 0 2022-02-07 21:49:05,622 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 453273 == 0. Received: {'172.23.121.127': 453273} for bucket 'GleamBookUsers0' 2022-02-07 21:49:05,622 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:05,625 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:05,625 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:05,877 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 23297 == 0 2022-02-07 21:49:05,877 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 23297 == 0. Received: {'172.23.121.126': 23297} for bucket 'GleamBookUsers0' 2022-02-07 21:49:05,877 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:05,878 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:05,880 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:05,948 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 787530 == 0 2022-02-07 21:49:05,950 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 787530 == 0. Received: {'172.23.121.124': 787530} for bucket 'GleamBookUsers0' 2022-02-07 21:49:05,950 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:05,951 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:05,951 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:07,874 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:08,059 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:08,730 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 574302 == 0 2022-02-07 21:49:08,732 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 574302 == 0. Received: {'172.23.121.128': 574302} for bucket 'GleamBookUsers0' 2022-02-07 21:49:08,732 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:08,733 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:08,733 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:08,809 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 441544 == 0 2022-02-07 21:49:08,809 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 441544 == 0. Received: {'172.23.120.170': 441544} for bucket 'GleamBookUsers0' 2022-02-07 21:49:08,809 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:08,811 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:08,811 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:08,973 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:09,361 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:09,753 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 309227 == 0 2022-02-07 21:49:09,753 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 309227 == 0. Received: {'172.23.121.115': 309227} for bucket 'GleamBookUsers0' 2022-02-07 21:49:09,753 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:09,756 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:09,756 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:10,437 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:10,441 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1065531 == 0 2022-02-07 21:49:10,443 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1065531 == 0. Received: {'172.23.121.123': 1065531} for bucket 'GleamBookUsers0' 2022-02-07 21:49:10,443 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:10,444 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:10,446 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:10,625 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:10,880 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:10,951 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:11,312 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 17501 == 0 2022-02-07 21:49:11,312 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 17501 == 0. Received: {'172.23.121.126': 17501} for bucket 'GleamBookUsers0' 2022-02-07 21:49:11,312 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:11,315 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:11,315 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:11,339 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 440575 == 0 2022-02-07 21:49:11,341 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 440575 == 0. Received: {'172.23.121.127': 440575} for bucket 'GleamBookUsers0' 2022-02-07 21:49:11,341 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:11,342 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:11,342 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:11,552 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1005786 == 0 2022-02-07 21:49:11,552 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1005786 == 0. Received: {'172.23.121.129': 1005786} for bucket 'GleamBookUsers0' 2022-02-07 21:49:11,552 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:11,555 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:11,555 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:11,881 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 778840 == 0 2022-02-07 21:49:11,881 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 778840 == 0. Received: {'172.23.121.124': 778840} for bucket 'GleamBookUsers0' 2022-02-07 21:49:11,881 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:11,884 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:11,884 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:13,734 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:13,812 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:14,471 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 409531 == 0 2022-02-07 21:49:14,471 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 409531 == 0. Received: {'172.23.120.170': 409531} for bucket 'GleamBookUsers0' 2022-02-07 21:49:14,473 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:14,474 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:14,474 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:14,558 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 558801 == 0 2022-02-07 21:49:14,558 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 558801 == 0. Received: {'172.23.121.128': 558801} for bucket 'GleamBookUsers0' 2022-02-07 21:49:14,559 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:14,561 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:14,561 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:14,756 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:15,446 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:15,456 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 288645 == 0 2022-02-07 21:49:15,457 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 288645 == 0. Received: {'172.23.121.115': 288645} for bucket 'GleamBookUsers0' 2022-02-07 21:49:15,457 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:15,459 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:15,459 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:16,315 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:16,342 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:16,493 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1039367 == 0 2022-02-07 21:49:16,494 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1039367 == 0. Received: {'172.23.121.123': 1039367} for bucket 'GleamBookUsers0' 2022-02-07 21:49:16,494 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:16,496 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:16,496 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:16,558 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:16,697 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 6374 == 0 2022-02-07 21:49:16,697 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 6374 == 0. Received: {'172.23.121.126': 6374} for bucket 'GleamBookUsers0' 2022-02-07 21:49:16,698 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:16,700 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:16,700 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:16,884 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:16,982 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 435239 == 0 2022-02-07 21:49:16,982 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 435239 == 0. Received: {'172.23.121.127': 435239} for bucket 'GleamBookUsers0' 2022-02-07 21:49:16,982 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:16,983 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:16,983 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:17,694 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 964788 == 0 2022-02-07 21:49:17,694 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 964788 == 0. Received: {'172.23.121.129': 964788} for bucket 'GleamBookUsers0' 2022-02-07 21:49:17,694 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:17,697 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:17,697 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:17,812 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 768958 == 0 2022-02-07 21:49:17,812 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 768958 == 0. Received: {'172.23.121.124': 768958} for bucket 'GleamBookUsers0' 2022-02-07 21:49:17,812 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:17,813 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:17,813 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:19,474 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:19,562 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:20,066 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 384987 == 0 2022-02-07 21:49:20,068 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 384987 == 0. Received: {'172.23.120.170': 384987} for bucket 'GleamBookUsers0' 2022-02-07 21:49:20,068 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:20,069 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:20,069 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:20,293 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 532878 == 0 2022-02-07 21:49:20,293 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 532878 == 0. Received: {'172.23.121.128': 532878} for bucket 'GleamBookUsers0' 2022-02-07 21:49:20,293 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:20,296 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:20,296 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:20,460 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:21,157 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 272665 == 0 2022-02-07 21:49:21,157 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 272665 == 0. Received: {'172.23.121.115': 272665} for bucket 'GleamBookUsers0' 2022-02-07 21:49:21,157 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:21,160 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:21,160 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:21,497 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:21,701 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:21,984 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:22,055 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 346 == 0 2022-02-07 21:49:22,055 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 346 == 0. Received: {'172.23.121.126': 346} for bucket 'GleamBookUsers0' 2022-02-07 21:49:22,055 | infra | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:22,058 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:22,058 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:22,589 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1037648 == 0 2022-02-07 21:49:22,589 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1037648 == 0. Received: {'172.23.121.123': 1037648} for bucket 'GleamBookUsers0' 2022-02-07 21:49:22,589 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:22,592 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:22,592 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:22,595 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 400654 == 0 2022-02-07 21:49:22,595 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 400654 == 0. Received: {'172.23.121.127': 400654} for bucket 'GleamBookUsers0' 2022-02-07 21:49:22,595 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:22,598 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:22,598 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:22,697 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:22,815 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:23,755 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 758562 == 0 2022-02-07 21:49:23,756 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 758562 == 0. Received: {'172.23.121.124': 758562} for bucket 'GleamBookUsers0' 2022-02-07 21:49:23,756 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:23,759 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:23,759 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:23,848 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 915158 == 0 2022-02-07 21:49:23,848 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 915158 == 0. Received: {'172.23.121.129': 915158} for bucket 'GleamBookUsers0' 2022-02-07 21:49:23,848 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:23,849 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:23,851 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:25,071 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:25,296 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:25,631 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 361909 == 0 2022-02-07 21:49:25,631 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 361909 == 0. Received: {'172.23.120.170': 361909} for bucket 'GleamBookUsers0' 2022-02-07 21:49:25,631 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:25,634 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:25,634 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:25,997 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 523125 == 0 2022-02-07 21:49:25,999 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 523125 == 0. Received: {'172.23.121.128': 523125} for bucket 'GleamBookUsers0' 2022-02-07 21:49:25,999 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:26,000 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:26,000 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:26,161 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:26,852 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 249275 == 0 2022-02-07 21:49:26,852 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 249275 == 0. Received: {'172.23.121.115': 249275} for bucket 'GleamBookUsers0' 2022-02-07 21:49:26,852 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:26,855 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:26,855 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:27,058 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:27,407 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 21:49:27,408 | test | DEBUG | pool-3-thread-25 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.126': 0} for bucket 'GleamBookUsers0' 2022-02-07 21:49:27,408 | infra | INFO | pool-3-thread-25 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299095.61' completed 2022-02-07 21:49:27,594 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:27,599 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:28,131 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 395416 == 0 2022-02-07 21:49:28,131 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 395416 == 0. Received: {'172.23.121.127': 395416} for bucket 'GleamBookUsers0' 2022-02-07 21:49:28,131 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:28,134 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:28,134 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:28,615 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 971464 == 0 2022-02-07 21:49:28,615 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 971464 == 0. Received: {'172.23.121.123': 971464} for bucket 'GleamBookUsers0' 2022-02-07 21:49:28,615 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:28,618 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:28,618 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:28,759 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:28,851 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:29,578 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 703226 == 0 2022-02-07 21:49:29,578 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 703226 == 0. Received: {'172.23.121.124': 703226} for bucket 'GleamBookUsers0' 2022-02-07 21:49:29,578 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:29,581 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:29,581 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:29,841 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 865568 == 0 2022-02-07 21:49:29,842 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 865568 == 0. Received: {'172.23.121.129': 865568} for bucket 'GleamBookUsers0' 2022-02-07 21:49:29,842 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:29,844 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:29,844 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:30,634 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:31,000 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:31,213 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 352266 == 0 2022-02-07 21:49:31,213 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 352266 == 0. Received: {'172.23.120.170': 352266} for bucket 'GleamBookUsers0' 2022-02-07 21:49:31,213 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:31,216 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:31,216 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:31,709 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 500097 == 0 2022-02-07 21:49:31,709 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 500097 == 0. Received: {'172.23.121.128': 500097} for bucket 'GleamBookUsers0' 2022-02-07 21:49:31,709 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:31,710 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:31,711 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:31,855 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:32,513 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 220471 == 0 2022-02-07 21:49:32,513 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 220471 == 0. Received: {'172.23.121.115': 220471} for bucket 'GleamBookUsers0' 2022-02-07 21:49:32,513 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:32,515 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:32,516 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:33,134 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:33,618 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:33,681 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 368394 == 0 2022-02-07 21:49:33,681 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 368394 == 0. Received: {'172.23.121.127': 368394} for bucket 'GleamBookUsers0' 2022-02-07 21:49:33,681 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:33,684 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:33,684 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:34,582 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:34,663 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 930450 == 0 2022-02-07 21:49:34,663 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 930450 == 0. Received: {'172.23.121.123': 930450} for bucket 'GleamBookUsers0' 2022-02-07 21:49:34,663 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:34,665 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:34,665 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:34,845 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:35,378 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 700944 == 0 2022-02-07 21:49:35,378 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 700944 == 0. Received: {'172.23.121.124': 700944} for bucket 'GleamBookUsers0' 2022-02-07 21:49:35,378 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:35,380 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:35,380 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:35,812 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 856216 == 0 2022-02-07 21:49:35,812 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 856216 == 0. Received: {'172.23.121.129': 856216} for bucket 'GleamBookUsers0' 2022-02-07 21:49:35,812 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:35,815 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:35,815 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:36,217 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:36,711 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:36,776 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 350818 == 0 2022-02-07 21:49:36,776 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 350818 == 0. Received: {'172.23.120.170': 350818} for bucket 'GleamBookUsers0' 2022-02-07 21:49:36,776 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:36,779 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:36,779 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:37,384 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 461549 == 0 2022-02-07 21:49:37,384 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 461549 == 0. Received: {'172.23.121.128': 461549} for bucket 'GleamBookUsers0' 2022-02-07 21:49:37,384 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:37,387 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:37,387 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:37,516 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:38,196 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 219021 == 0 2022-02-07 21:49:38,197 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 219021 == 0. Received: {'172.23.121.115': 219021} for bucket 'GleamBookUsers0' 2022-02-07 21:49:38,197 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:38,198 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:38,198 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:38,684 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:39,213 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 368162 == 0 2022-02-07 21:49:39,213 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 368162 == 0. Received: {'172.23.121.127': 368162} for bucket 'GleamBookUsers0' 2022-02-07 21:49:39,213 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:39,214 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:39,216 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:39,667 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:40,381 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:40,661 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 910158 == 0 2022-02-07 21:49:40,663 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 910158 == 0. Received: {'172.23.121.123': 910158} for bucket 'GleamBookUsers0' 2022-02-07 21:49:40,663 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:40,664 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:40,664 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:40,815 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:41,190 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 675890 == 0 2022-02-07 21:49:41,190 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 675890 == 0. Received: {'172.23.121.124': 675890} for bucket 'GleamBookUsers0' 2022-02-07 21:49:41,190 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:41,194 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:41,194 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:41,766 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 796718 == 0 2022-02-07 21:49:41,767 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 796718 == 0. Received: {'172.23.121.129': 796718} for bucket 'GleamBookUsers0' 2022-02-07 21:49:41,767 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:41,769 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:41,769 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:41,779 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:42,321 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 333290 == 0 2022-02-07 21:49:42,322 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 333290 == 0. Received: {'172.23.120.170': 333290} for bucket 'GleamBookUsers0' 2022-02-07 21:49:42,322 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:42,325 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:42,325 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:42,387 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:43,042 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 442435 == 0 2022-02-07 21:49:43,042 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 442435 == 0. Received: {'172.23.121.128': 442435} for bucket 'GleamBookUsers0' 2022-02-07 21:49:43,043 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:43,045 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:43,046 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:43,200 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:43,826 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 188979 == 0 2022-02-07 21:49:43,828 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 188979 == 0. Received: {'172.23.121.115': 188979} for bucket 'GleamBookUsers0' 2022-02-07 21:49:43,828 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:43,828 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:43,829 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:44,216 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:44,736 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 343440 == 0 2022-02-07 21:49:44,736 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 343440 == 0. Received: {'172.23.121.127': 343440} for bucket 'GleamBookUsers0' 2022-02-07 21:49:44,737 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:44,739 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:44,739 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:45,665 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:46,194 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:46,615 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 867720 == 0 2022-02-07 21:49:46,615 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 867720 == 0. Received: {'172.23.121.123': 867720} for bucket 'GleamBookUsers0' 2022-02-07 21:49:46,615 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:46,618 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:46,618 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:46,770 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:46,967 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 652488 == 0 2022-02-07 21:49:46,969 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 652488 == 0. Received: {'172.23.121.124': 652488} for bucket 'GleamBookUsers0' 2022-02-07 21:49:46,969 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:46,970 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:46,970 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:47,325 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:47,641 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 732607 == 0 2022-02-07 21:49:47,641 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 732607 == 0. Received: {'172.23.121.129': 732607} for bucket 'GleamBookUsers0' 2022-02-07 21:49:47,641 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:47,644 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:47,644 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:47,872 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 313652 == 0 2022-02-07 21:49:47,872 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 313652 == 0. Received: {'172.23.120.170': 313652} for bucket 'GleamBookUsers0' 2022-02-07 21:49:47,872 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:47,875 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:47,875 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:48,046 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:48,697 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 406263 == 0 2022-02-07 21:49:48,698 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 406263 == 0. Received: {'172.23.121.128': 406263} for bucket 'GleamBookUsers0' 2022-02-07 21:49:48,698 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:48,700 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:48,700 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:48,829 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:49,421 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 171019 == 0 2022-02-07 21:49:49,421 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 171019 == 0. Received: {'172.23.121.115': 171019} for bucket 'GleamBookUsers0' 2022-02-07 21:49:49,423 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:49,424 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:49,424 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:49,740 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:50,290 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 343440 == 0 2022-02-07 21:49:50,290 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 343440 == 0. Received: {'172.23.121.127': 343440} for bucket 'GleamBookUsers0' 2022-02-07 21:49:50,290 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:50,292 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:50,292 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:51,619 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:51,970 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:52,473 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 786476 == 0 2022-02-07 21:49:52,473 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 786476 == 0. Received: {'172.23.121.123': 786476} for bucket 'GleamBookUsers0' 2022-02-07 21:49:52,473 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:52,476 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:52,476 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:52,645 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:52,746 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 630507 == 0 2022-02-07 21:49:52,746 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 630507 == 0. Received: {'172.23.121.124': 630507} for bucket 'GleamBookUsers0' 2022-02-07 21:49:52,746 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:52,747 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:52,747 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:52,875 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:53,421 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 298880 == 0 2022-02-07 21:49:53,421 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 298880 == 0. Received: {'172.23.120.170': 298880} for bucket 'GleamBookUsers0' 2022-02-07 21:49:53,421 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:53,424 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:53,424 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:53,535 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 675939 == 0 2022-02-07 21:49:53,535 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 675939 == 0. Received: {'172.23.121.129': 675939} for bucket 'GleamBookUsers0' 2022-02-07 21:49:53,535 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:53,536 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:53,536 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:53,701 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:54,270 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 340501 == 0 2022-02-07 21:49:54,270 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 340501 == 0. Received: {'172.23.121.128': 340501} for bucket 'GleamBookUsers0' 2022-02-07 21:49:54,272 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:54,273 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:54,273 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:54,424 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:54,999 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 141101 == 0 2022-02-07 21:49:54,999 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 141101 == 0. Received: {'172.23.121.115': 141101} for bucket 'GleamBookUsers0' 2022-02-07 21:49:54,999 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:55,000 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:55,000 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:55,293 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:55,802 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 321830 == 0 2022-02-07 21:49:55,802 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 321830 == 0. Received: {'172.23.121.127': 321830} for bucket 'GleamBookUsers0' 2022-02-07 21:49:55,802 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:55,805 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:55,805 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:57,476 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:57,750 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:58,332 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 746836 == 0 2022-02-07 21:49:58,332 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 746836 == 0. Received: {'172.23.121.123': 746836} for bucket 'GleamBookUsers0' 2022-02-07 21:49:58,332 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:58,335 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:58,335 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:58,424 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:58,490 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 587533 == 0 2022-02-07 21:49:58,490 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 587533 == 0. Received: {'172.23.121.124': 587533} for bucket 'GleamBookUsers0' 2022-02-07 21:49:58,490 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:58,493 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:58,493 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:58,538 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:58,918 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 267494 == 0 2022-02-07 21:49:58,941 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 267494 == 0. Received: {'172.23.120.170': 267494} for bucket 'GleamBookUsers0' 2022-02-07 21:49:58,941 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:58,944 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:58,944 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:59,273 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:49:59,329 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 640928 == 0 2022-02-07 21:49:59,329 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 640928 == 0. Received: {'172.23.121.129': 640928} for bucket 'GleamBookUsers0' 2022-02-07 21:49:59,329 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:59,332 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:59,332 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:49:59,845 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 325685 == 0 2022-02-07 21:49:59,845 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 325685 == 0. Received: {'172.23.121.128': 325685} for bucket 'GleamBookUsers0' 2022-02-07 21:49:59,845 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:49:59,848 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:49:59,848 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:00,002 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:00,575 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 134839 == 0 2022-02-07 21:50:00,576 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 134839 == 0. Received: {'172.23.121.115': 134839} for bucket 'GleamBookUsers0' 2022-02-07 21:50:00,576 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:00,578 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:00,578 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:00,805 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:01,305 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 298512 == 0 2022-02-07 21:50:01,305 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 298512 == 0. Received: {'172.23.121.127': 298512} for bucket 'GleamBookUsers0' 2022-02-07 21:50:01,305 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:01,308 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:01,308 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:03,335 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:03,493 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:03,944 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:04,135 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 685872 == 0 2022-02-07 21:50:04,135 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 685872 == 0. Received: {'172.23.121.123': 685872} for bucket 'GleamBookUsers0' 2022-02-07 21:50:04,135 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:04,138 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:04,138 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:04,255 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 587533 == 0 2022-02-07 21:50:04,256 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 587533 == 0. Received: {'172.23.121.124': 587533} for bucket 'GleamBookUsers0' 2022-02-07 21:50:04,256 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:04,259 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:04,259 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:04,332 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:04,434 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 248632 == 0 2022-02-07 21:50:04,434 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 248632 == 0. Received: {'172.23.120.170': 248632} for bucket 'GleamBookUsers0' 2022-02-07 21:50:04,434 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:04,436 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:04,436 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:04,848 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:05,154 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 632546 == 0 2022-02-07 21:50:05,154 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 632546 == 0. Received: {'172.23.121.129': 632546} for bucket 'GleamBookUsers0' 2022-02-07 21:50:05,154 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:05,157 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:05,157 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:05,404 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 295692 == 0 2022-02-07 21:50:05,404 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 295692 == 0. Received: {'172.23.121.128': 295692} for bucket 'GleamBookUsers0' 2022-02-07 21:50:05,404 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:05,405 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:05,405 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:05,578 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:06,098 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 114677 == 0 2022-02-07 21:50:06,098 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 114677 == 0. Received: {'172.23.121.115': 114677} for bucket 'GleamBookUsers0' 2022-02-07 21:50:06,098 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:06,099 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:06,099 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:06,309 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:06,808 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 276392 == 0 2022-02-07 21:50:06,809 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 276392 == 0. Received: {'172.23.121.127': 276392} for bucket 'GleamBookUsers0' 2022-02-07 21:50:06,809 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:06,811 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:06,811 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:09,138 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:09,262 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:09,437 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:09,982 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 628462 == 0 2022-02-07 21:50:09,982 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 628462 == 0. Received: {'172.23.121.123': 628462} for bucket 'GleamBookUsers0' 2022-02-07 21:50:09,983 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:09,984 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:09,984 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:10,005 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 229120 == 0 2022-02-07 21:50:10,005 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 229120 == 0. Received: {'172.23.120.170': 229120} for bucket 'GleamBookUsers0' 2022-02-07 21:50:10,005 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:10,007 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:10,007 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:10,071 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 548181 == 0 2022-02-07 21:50:10,071 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 548181 == 0. Received: {'172.23.121.124': 548181} for bucket 'GleamBookUsers0' 2022-02-07 21:50:10,071 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:10,072 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:10,073 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:10,157 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:10,407 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:10,976 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 577822 == 0 2022-02-07 21:50:10,976 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 577822 == 0. Received: {'172.23.121.129': 577822} for bucket 'GleamBookUsers0' 2022-02-07 21:50:10,976 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:10,980 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:10,980 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:11,012 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 295692 == 0 2022-02-07 21:50:11,012 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 295692 == 0. Received: {'172.23.121.128': 295692} for bucket 'GleamBookUsers0' 2022-02-07 21:50:11,013 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:11,015 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:11,015 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:11,101 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:11,627 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 109723 == 0 2022-02-07 21:50:11,628 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 109723 == 0. Received: {'172.23.121.115': 109723} for bucket 'GleamBookUsers0' 2022-02-07 21:50:11,628 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:11,630 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:11,630 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:11,815 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:12,339 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 276392 == 0 2022-02-07 21:50:12,341 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 276392 == 0. Received: {'172.23.121.127': 276392} for bucket 'GleamBookUsers0' 2022-02-07 21:50:12,341 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:12,342 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:12,342 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:14,986 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:15,007 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:15,073 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:15,463 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 203616 == 0 2022-02-07 21:50:15,464 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 203616 == 0. Received: {'172.23.120.170': 203616} for bucket 'GleamBookUsers0' 2022-02-07 21:50:15,464 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:15,467 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:15,467 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:15,828 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 506033 == 0 2022-02-07 21:50:15,828 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 506033 == 0. Received: {'172.23.121.124': 506033} for bucket 'GleamBookUsers0' 2022-02-07 21:50:15,828 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:15,831 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:15,831 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:15,859 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 592746 == 0 2022-02-07 21:50:15,859 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 592746 == 0. Received: {'172.23.121.123': 592746} for bucket 'GleamBookUsers0' 2022-02-07 21:50:15,859 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:15,861 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:15,862 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:15,980 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:16,016 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:16,575 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 267758 == 0 2022-02-07 21:50:16,575 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 267758 == 0. Received: {'172.23.121.128': 267758} for bucket 'GleamBookUsers0' 2022-02-07 21:50:16,575 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:16,578 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:16,578 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:16,631 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:16,785 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 537126 == 0 2022-02-07 21:50:16,785 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 537126 == 0. Received: {'172.23.121.129': 537126} for bucket 'GleamBookUsers0' 2022-02-07 21:50:16,785 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:16,788 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:16,788 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:17,170 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 105263 == 0 2022-02-07 21:50:17,170 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 105263 == 0. Received: {'172.23.121.115': 105263} for bucket 'GleamBookUsers0' 2022-02-07 21:50:17,170 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:17,171 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:17,171 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:17,344 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:17,821 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 254580 == 0 2022-02-07 21:50:17,822 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 254580 == 0. Received: {'172.23.121.127': 254580} for bucket 'GleamBookUsers0' 2022-02-07 21:50:17,822 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:17,823 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:17,823 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:20,467 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:20,831 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:20,862 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:20,953 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 186866 == 0 2022-02-07 21:50:20,953 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 186866 == 0. Received: {'172.23.120.170': 186866} for bucket 'GleamBookUsers0' 2022-02-07 21:50:20,953 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:20,956 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:20,956 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:21,579 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:21,588 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 465659 == 0 2022-02-07 21:50:21,588 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 465659 == 0. Received: {'172.23.121.124': 465659} for bucket 'GleamBookUsers0' 2022-02-07 21:50:21,588 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:21,589 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 535730 == 0 2022-02-07 21:50:21,591 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 535730 == 0. Received: {'172.23.121.123': 535730} for bucket 'GleamBookUsers0' 2022-02-07 21:50:21,591 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:21,592 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:21,592 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:21,594 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:21,594 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:21,788 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:22,082 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 254186 == 0 2022-02-07 21:50:22,082 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 254186 == 0. Received: {'172.23.121.128': 254186} for bucket 'GleamBookUsers0' 2022-02-07 21:50:22,084 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:22,085 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:22,085 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:22,174 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:22,503 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 517566 == 0 2022-02-07 21:50:22,503 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 517566 == 0. Received: {'172.23.121.129': 517566} for bucket 'GleamBookUsers0' 2022-02-07 21:50:22,503 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:22,505 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:22,506 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:22,697 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 91627 == 0 2022-02-07 21:50:22,697 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 91627 == 0. Received: {'172.23.121.115': 91627} for bucket 'GleamBookUsers0' 2022-02-07 21:50:22,697 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:22,700 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:22,700 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:22,823 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:23,302 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 233620 == 0 2022-02-07 21:50:23,302 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 233620 == 0. Received: {'172.23.121.127': 233620} for bucket 'GleamBookUsers0' 2022-02-07 21:50:23,302 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:23,305 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:23,305 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:25,956 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:26,404 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 164704 == 0 2022-02-07 21:50:26,404 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 164704 == 0. Received: {'172.23.120.170': 164704} for bucket 'GleamBookUsers0' 2022-02-07 21:50:26,404 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:26,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:26,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:26,592 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:26,594 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:27,085 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:27,247 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 425707 == 0 2022-02-07 21:50:27,249 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 425707 == 0. Received: {'172.23.121.124': 425707} for bucket 'GleamBookUsers0' 2022-02-07 21:50:27,249 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:27,250 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:27,250 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:27,309 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 500716 == 0 2022-02-07 21:50:27,309 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 500716 == 0. Received: {'172.23.121.123': 500716} for bucket 'GleamBookUsers0' 2022-02-07 21:50:27,309 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:27,311 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:27,312 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:27,506 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:27,595 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 239675 == 0 2022-02-07 21:50:27,595 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 239675 == 0. Received: {'172.23.121.128': 239675} for bucket 'GleamBookUsers0' 2022-02-07 21:50:27,595 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:27,598 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:27,598 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:27,700 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:28,210 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 459444 == 0 2022-02-07 21:50:28,210 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 459444 == 0. Received: {'172.23.121.129': 459444} for bucket 'GleamBookUsers0' 2022-02-07 21:50:28,210 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:28,213 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:28,213 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:28,243 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 80478 == 0 2022-02-07 21:50:28,243 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 80478 == 0. Received: {'172.23.121.115': 80478} for bucket 'GleamBookUsers0' 2022-02-07 21:50:28,243 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:28,246 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:28,246 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:28,305 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:28,765 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 213060 == 0 2022-02-07 21:50:28,766 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 213060 == 0. Received: {'172.23.121.127': 213060} for bucket 'GleamBookUsers0' 2022-02-07 21:50:28,766 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:28,767 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:28,767 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:31,407 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:31,851 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 160988 == 0 2022-02-07 21:50:31,851 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 160988 == 0. Received: {'172.23.120.170': 160988} for bucket 'GleamBookUsers0' 2022-02-07 21:50:31,851 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:31,854 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:31,854 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:32,252 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:32,312 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:32,598 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:32,913 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 386667 == 0 2022-02-07 21:50:32,913 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 386667 == 0. Received: {'172.23.121.124': 386667} for bucket 'GleamBookUsers0' 2022-02-07 21:50:32,913 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:32,914 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:32,914 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:33,022 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 449804 == 0 2022-02-07 21:50:33,022 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 449804 == 0. Received: {'172.23.121.123': 449804} for bucket 'GleamBookUsers0' 2022-02-07 21:50:33,022 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:33,023 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:33,025 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:33,107 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 211704 == 0 2022-02-07 21:50:33,108 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 211704 == 0. Received: {'172.23.121.128': 211704} for bucket 'GleamBookUsers0' 2022-02-07 21:50:33,108 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:33,109 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:33,109 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:33,213 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:33,246 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:33,756 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 70034 == 0 2022-02-07 21:50:33,756 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 70034 == 0. Received: {'172.23.121.115': 70034} for bucket 'GleamBookUsers0' 2022-02-07 21:50:33,756 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:33,759 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:33,759 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:33,769 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:33,897 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 416786 == 0 2022-02-07 21:50:33,898 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 416786 == 0. Received: {'172.23.121.129': 416786} for bucket 'GleamBookUsers0' 2022-02-07 21:50:33,898 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:33,900 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:33,900 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:34,239 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 191080 == 0 2022-02-07 21:50:34,239 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 191080 == 0. Received: {'172.23.121.127': 191080} for bucket 'GleamBookUsers0' 2022-02-07 21:50:34,239 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:34,242 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:34,243 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:36,854 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:37,272 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 142414 == 0 2022-02-07 21:50:37,272 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 142414 == 0. Received: {'172.23.120.170': 142414} for bucket 'GleamBookUsers0' 2022-02-07 21:50:37,273 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:37,275 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:37,275 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:37,915 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:38,026 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:38,109 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:38,503 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 349502 == 0 2022-02-07 21:50:38,503 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 349502 == 0. Received: {'172.23.121.124': 349502} for bucket 'GleamBookUsers0' 2022-02-07 21:50:38,503 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:38,506 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:38,506 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:38,648 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 211704 == 0 2022-02-07 21:50:38,648 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 211704 == 0. Received: {'172.23.121.128': 211704} for bucket 'GleamBookUsers0' 2022-02-07 21:50:38,648 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:38,651 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:38,651 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:38,694 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 414864 == 0 2022-02-07 21:50:38,694 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 414864 == 0. Received: {'172.23.121.123': 414864} for bucket 'GleamBookUsers0' 2022-02-07 21:50:38,694 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:38,697 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:38,697 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:38,762 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:38,901 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:39,223 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 53872 == 0 2022-02-07 21:50:39,224 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 53872 == 0. Received: {'172.23.121.115': 53872} for bucket 'GleamBookUsers0' 2022-02-07 21:50:39,224 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:39,226 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:39,226 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:39,243 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:39,578 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 404018 == 0 2022-02-07 21:50:39,579 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 404018 == 0. Received: {'172.23.121.129': 404018} for bucket 'GleamBookUsers0' 2022-02-07 21:50:39,579 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:39,582 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:39,582 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:39,691 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 171858 == 0 2022-02-07 21:50:39,691 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 171858 == 0. Received: {'172.23.121.127': 171858} for bucket 'GleamBookUsers0' 2022-02-07 21:50:39,691 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:39,694 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:39,694 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:42,275 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:42,697 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 126984 == 0 2022-02-07 21:50:42,697 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 126984 == 0. Received: {'172.23.120.170': 126984} for bucket 'GleamBookUsers0' 2022-02-07 21:50:42,697 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:42,698 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:42,698 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:43,506 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:43,651 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:43,698 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:44,026 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 277119 == 0 2022-02-07 21:50:44,028 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 277119 == 0. Received: {'172.23.121.124': 277119} for bucket 'GleamBookUsers0' 2022-02-07 21:50:44,028 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:44,029 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:44,029 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:44,141 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 184371 == 0 2022-02-07 21:50:44,141 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 184371 == 0. Received: {'172.23.121.128': 184371} for bucket 'GleamBookUsers0' 2022-02-07 21:50:44,141 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:44,144 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:44,144 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:44,226 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:44,318 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 381150 == 0 2022-02-07 21:50:44,319 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 381150 == 0. Received: {'172.23.121.123': 381150} for bucket 'GleamBookUsers0' 2022-02-07 21:50:44,319 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:44,321 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:44,321 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:44,582 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:44,684 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 50962 == 0 2022-02-07 21:50:44,684 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 50962 == 0. Received: {'172.23.121.115': 50962} for bucket 'GleamBookUsers0' 2022-02-07 21:50:44,684 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:44,687 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:44,687 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:44,694 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:45,164 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 152192 == 0 2022-02-07 21:50:45,164 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 152192 == 0. Received: {'172.23.121.127': 152192} for bucket 'GleamBookUsers0' 2022-02-07 21:50:45,164 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:45,167 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:45,167 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:45,286 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 364182 == 0 2022-02-07 21:50:45,288 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 364182 == 0. Received: {'172.23.121.129': 364182} for bucket 'GleamBookUsers0' 2022-02-07 21:50:45,288 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:45,289 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:45,289 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:47,700 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:48,111 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 111794 == 0 2022-02-07 21:50:48,111 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 111794 == 0. Received: {'172.23.120.170': 111794} for bucket 'GleamBookUsers0' 2022-02-07 21:50:48,111 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:48,115 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:48,115 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:49,030 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:49,144 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:49,322 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:49,562 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 242766 == 0 2022-02-07 21:50:49,562 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 242766 == 0. Received: {'172.23.121.124': 242766} for bucket 'GleamBookUsers0' 2022-02-07 21:50:49,562 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:49,565 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:49,565 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:49,631 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 171253 == 0 2022-02-07 21:50:49,631 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 171253 == 0. Received: {'172.23.121.128': 171253} for bucket 'GleamBookUsers0' 2022-02-07 21:50:49,631 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:49,634 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:49,634 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:49,687 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:49,967 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 333267 == 0 2022-02-07 21:50:49,967 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 333267 == 0. Received: {'172.23.121.123': 333267} for bucket 'GleamBookUsers0' 2022-02-07 21:50:49,967 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:49,970 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:49,970 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:50,131 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 39498 == 0 2022-02-07 21:50:50,131 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 39498 == 0. Received: {'172.23.121.115': 39498} for bucket 'GleamBookUsers0' 2022-02-07 21:50:50,132 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:50,134 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:50,134 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:50,167 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:50,290 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:50,586 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 115714 == 0 2022-02-07 21:50:50,588 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 115714 == 0. Received: {'172.23.121.127': 115714} for bucket 'GleamBookUsers0' 2022-02-07 21:50:50,588 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:50,589 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:50,589 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:50,959 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 346498 == 0 2022-02-07 21:50:50,959 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 346498 == 0. Received: {'172.23.121.129': 346498} for bucket 'GleamBookUsers0' 2022-02-07 21:50:50,959 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:50,961 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:50,961 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:53,117 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:53,536 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 97798 == 0 2022-02-07 21:50:53,536 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 97798 == 0. Received: {'172.23.120.170': 97798} for bucket 'GleamBookUsers0' 2022-02-07 21:50:53,536 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:53,538 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:53,538 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:54,565 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:54,634 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:54,970 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:55,079 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 176896 == 0 2022-02-07 21:50:55,081 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 176896 == 0. Received: {'172.23.121.124': 176896} for bucket 'GleamBookUsers0' 2022-02-07 21:50:55,081 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:55,082 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:55,082 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:55,130 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 146033 == 0 2022-02-07 21:50:55,131 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 146033 == 0. Received: {'172.23.121.128': 146033} for bucket 'GleamBookUsers0' 2022-02-07 21:50:55,131 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:55,132 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:55,132 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:55,134 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:55,585 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 317429 == 0 2022-02-07 21:50:55,585 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 28705 == 0 2022-02-07 21:50:55,585 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 317429 == 0. Received: {'172.23.121.123': 317429} for bucket 'GleamBookUsers0' 2022-02-07 21:50:55,585 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 28705 == 0. Received: {'172.23.121.115': 28705} for bucket 'GleamBookUsers0' 2022-02-07 21:50:55,585 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:55,585 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:55,588 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:55,588 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:55,588 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:55,588 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:55,591 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:55,961 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:56,026 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 80650 == 0 2022-02-07 21:50:56,026 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 80650 == 0. Received: {'172.23.121.127': 80650} for bucket 'GleamBookUsers0' 2022-02-07 21:50:56,026 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:56,029 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:56,029 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:56,565 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 307806 == 0 2022-02-07 21:50:56,566 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 307806 == 0. Received: {'172.23.121.129': 307806} for bucket 'GleamBookUsers0' 2022-02-07 21:50:56,566 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:56,569 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:56,569 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:50:58,539 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:50:59,006 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 83906 == 0 2022-02-07 21:50:59,007 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 83906 == 0. Received: {'172.23.120.170': 83906} for bucket 'GleamBookUsers0' 2022-02-07 21:50:59,007 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:50:59,009 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:50:59,010 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:00,085 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:00,134 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:00,569 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 146464 == 0 2022-02-07 21:51:00,569 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 146464 == 0. Received: {'172.23.121.124': 146464} for bucket 'GleamBookUsers0' 2022-02-07 21:51:00,571 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:00,572 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:00,573 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:00,588 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:00,589 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:00,627 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 133676 == 0 2022-02-07 21:51:00,627 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 133676 == 0. Received: {'172.23.121.128': 133676} for bucket 'GleamBookUsers0' 2022-02-07 21:51:00,627 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:00,628 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:00,628 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:01,016 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 24881 == 0 2022-02-07 21:51:01,016 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 24881 == 0. Received: {'172.23.121.115': 24881} for bucket 'GleamBookUsers0' 2022-02-07 21:51:01,017 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:01,019 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:01,020 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:01,029 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:01,161 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 255124 == 0 2022-02-07 21:51:01,163 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 255124 == 0. Received: {'172.23.121.123': 255124} for bucket 'GleamBookUsers0' 2022-02-07 21:51:01,163 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:01,164 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:01,165 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:01,417 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 65450 == 0 2022-02-07 21:51:01,417 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 65450 == 0. Received: {'172.23.121.127': 65450} for bucket 'GleamBookUsers0' 2022-02-07 21:51:01,417 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:01,420 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:01,420 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:01,569 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:02,216 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 303594 == 0 2022-02-07 21:51:02,216 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 303594 == 0. Received: {'172.23.121.129': 303594} for bucket 'GleamBookUsers0' 2022-02-07 21:51:02,216 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:02,219 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:02,219 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:04,012 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:04,404 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 70350 == 0 2022-02-07 21:51:04,404 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 70350 == 0. Received: {'172.23.120.170': 70350} for bucket 'GleamBookUsers0' 2022-02-07 21:51:04,404 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:04,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:04,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:05,573 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:05,628 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:06,020 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:06,085 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 118364 == 0 2022-02-07 21:51:06,085 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 118364 == 0. Received: {'172.23.121.124': 118364} for bucket 'GleamBookUsers0' 2022-02-07 21:51:06,085 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:06,086 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:06,088 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:06,101 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 113900 == 0 2022-02-07 21:51:06,102 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 113900 == 0. Received: {'172.23.121.128': 113900} for bucket 'GleamBookUsers0' 2022-02-07 21:51:06,102 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:06,105 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:06,105 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:06,165 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:06,421 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:06,437 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 12161 == 0 2022-02-07 21:51:06,437 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 12161 == 0. Received: {'172.23.121.115': 12161} for bucket 'GleamBookUsers0' 2022-02-07 21:51:06,437 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:06,440 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:06,440 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:06,727 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 227124 == 0 2022-02-07 21:51:06,729 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 227124 == 0. Received: {'172.23.121.123': 227124} for bucket 'GleamBookUsers0' 2022-02-07 21:51:06,729 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:06,730 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:06,730 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:06,848 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 51380 == 0 2022-02-07 21:51:06,848 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 51380 == 0. Received: {'172.23.121.127': 51380} for bucket 'GleamBookUsers0' 2022-02-07 21:51:06,848 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:06,849 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:06,851 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:07,219 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:07,848 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 258618 == 0 2022-02-07 21:51:07,848 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 258618 == 0. Received: {'172.23.121.129': 258618} for bucket 'GleamBookUsers0' 2022-02-07 21:51:07,848 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:07,851 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:07,851 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:09,407 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:09,805 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 57710 == 0 2022-02-07 21:51:09,805 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 57710 == 0. Received: {'172.23.120.170': 57710} for bucket 'GleamBookUsers0' 2022-02-07 21:51:09,806 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:09,809 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:09,809 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:11,088 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:11,105 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:11,440 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:11,546 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 85300 == 0 2022-02-07 21:51:11,546 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 85300 == 0. Received: {'172.23.121.128': 85300} for bucket 'GleamBookUsers0' 2022-02-07 21:51:11,546 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:11,549 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:11,549 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:11,582 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 91944 == 0 2022-02-07 21:51:11,582 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 91944 == 0. Received: {'172.23.121.124': 91944} for bucket 'GleamBookUsers0' 2022-02-07 21:51:11,582 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:11,584 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:11,584 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:11,732 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:11,822 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 6017 == 0 2022-02-07 21:51:11,822 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 6017 == 0. Received: {'172.23.121.115': 6017} for bucket 'GleamBookUsers0' 2022-02-07 21:51:11,822 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:11,825 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:11,825 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:11,851 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:12,253 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 8584 == 0 2022-02-07 21:51:12,253 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 8584 == 0. Received: {'172.23.121.127': 8584} for bucket 'GleamBookUsers0' 2022-02-07 21:51:12,253 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:12,256 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:12,256 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:12,276 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 198482 == 0 2022-02-07 21:51:12,276 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 198482 == 0. Received: {'172.23.121.123': 198482} for bucket 'GleamBookUsers0' 2022-02-07 21:51:12,276 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:12,279 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:12,279 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:12,851 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:13,365 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 216129 == 0 2022-02-07 21:51:13,365 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 216129 == 0. Received: {'172.23.121.129': 216129} for bucket 'GleamBookUsers0' 2022-02-07 21:51:13,365 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:13,368 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:13,368 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:14,812 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:15,204 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 46528 == 0 2022-02-07 21:51:15,204 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 46528 == 0. Received: {'172.23.120.170': 46528} for bucket 'GleamBookUsers0' 2022-02-07 21:51:15,204 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:15,207 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:15,207 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:16,549 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:16,585 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:16,825 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:16,956 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 42680 == 0 2022-02-07 21:51:16,956 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 42680 == 0. Received: {'172.23.121.128': 42680} for bucket 'GleamBookUsers0' 2022-02-07 21:51:16,956 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:16,957 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:16,957 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:17,035 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 80034 == 0 2022-02-07 21:51:17,036 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 80034 == 0. Received: {'172.23.121.124': 80034} for bucket 'GleamBookUsers0' 2022-02-07 21:51:17,036 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:17,038 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:17,038 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:17,201 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 2527 == 0 2022-02-07 21:51:17,201 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2527 == 0. Received: {'172.23.121.115': 2527} for bucket 'GleamBookUsers0' 2022-02-07 21:51:17,201 | infra | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:17,204 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:17,204 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:17,257 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:17,279 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:17,624 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 3306 == 0 2022-02-07 21:51:17,624 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 3306 == 0. Received: {'172.23.121.127': 3306} for bucket 'GleamBookUsers0' 2022-02-07 21:51:17,624 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:17,625 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:17,625 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:17,806 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 160530 == 0 2022-02-07 21:51:17,806 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 160530 == 0. Received: {'172.23.121.123': 160530} for bucket 'GleamBookUsers0' 2022-02-07 21:51:17,808 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:17,809 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:17,811 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:18,368 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:18,881 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 204451 == 0 2022-02-07 21:51:18,881 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 204451 == 0. Received: {'172.23.121.129': 204451} for bucket 'GleamBookUsers0' 2022-02-07 21:51:18,881 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:18,884 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:18,884 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:20,209 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:20,601 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 46528 == 0 2022-02-07 21:51:20,601 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 46528 == 0. Received: {'172.23.120.170': 46528} for bucket 'GleamBookUsers0' 2022-02-07 21:51:20,601 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:20,604 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:20,604 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:21,959 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:22,038 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:22,204 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:22,364 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 21900 == 0 2022-02-07 21:51:22,364 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 21900 == 0. Received: {'172.23.121.128': 21900} for bucket 'GleamBookUsers0' 2022-02-07 21:51:22,365 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:22,367 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:22,367 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:22,467 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 45426 == 0 2022-02-07 21:51:22,469 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 45426 == 0. Received: {'172.23.121.124': 45426} for bucket 'GleamBookUsers0' 2022-02-07 21:51:22,469 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:22,470 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:22,470 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:22,558 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 21:51:22,558 | test | DEBUG | pool-3-thread-21 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.115': 0} for bucket 'GleamBookUsers0' 2022-02-07 21:51:22,559 | infra | INFO | pool-3-thread-21 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299093.64' completed 2022-02-07 21:51:22,627 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:22,812 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:22,993 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 21:51:22,994 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.127': 0} for bucket 'GleamBookUsers0' 2022-02-07 21:51:22,994 | infra | INFO | pool-3-thread-26 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299096.12' completed 2022-02-07 21:51:23,255 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 124076 == 0 2022-02-07 21:51:23,256 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 124076 == 0. Received: {'172.23.121.123': 124076} for bucket 'GleamBookUsers0' 2022-02-07 21:51:23,256 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:23,257 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:23,257 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:23,884 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:24,351 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 152643 == 0 2022-02-07 21:51:24,351 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 152643 == 0. Received: {'172.23.121.129': 152643} for bucket 'GleamBookUsers0' 2022-02-07 21:51:24,351 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:24,354 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:24,354 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:25,605 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:25,990 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 27250 == 0 2022-02-07 21:51:25,990 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 27250 == 0. Received: {'172.23.120.170': 27250} for bucket 'GleamBookUsers0' 2022-02-07 21:51:25,992 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:25,993 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:25,993 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:27,367 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:27,470 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:27,717 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 464 == 0 2022-02-07 21:51:27,717 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 464 == 0. Received: {'172.23.121.128': 464} for bucket 'GleamBookUsers0' 2022-02-07 21:51:27,717 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:27,720 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:27,720 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:27,841 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 13612 == 0 2022-02-07 21:51:27,841 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 13612 == 0. Received: {'172.23.121.124': 13612} for bucket 'GleamBookUsers0' 2022-02-07 21:51:27,841 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:27,842 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:27,842 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:28,259 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:28,667 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 68712 == 0 2022-02-07 21:51:28,667 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 68712 == 0. Received: {'172.23.121.123': 68712} for bucket 'GleamBookUsers0' 2022-02-07 21:51:28,667 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:28,668 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:28,668 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:29,355 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:29,828 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 128007 == 0 2022-02-07 21:51:29,828 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 128007 == 0. Received: {'172.23.121.129': 128007} for bucket 'GleamBookUsers0' 2022-02-07 21:51:29,828 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:29,831 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:29,831 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:30,996 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:31,398 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 18984 == 0 2022-02-07 21:51:31,398 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 18984 == 0. Received: {'172.23.120.170': 18984} for bucket 'GleamBookUsers0' 2022-02-07 21:51:31,398 | infra | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:31,401 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:31,401 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:32,720 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:32,844 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:33,082 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 21:51:33,082 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.128': 0} for bucket 'GleamBookUsers0' 2022-02-07 21:51:33,082 | infra | INFO | pool-3-thread-27 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299096.59' completed 2022-02-07 21:51:33,209 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1016 == 0 2022-02-07 21:51:33,210 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1016 == 0. Received: {'172.23.121.124': 1016} for bucket 'GleamBookUsers0' 2022-02-07 21:51:33,210 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:33,211 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:33,211 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:33,670 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:34,059 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 31620 == 0 2022-02-07 21:51:34,059 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 31620 == 0. Received: {'172.23.121.123': 31620} for bucket 'GleamBookUsers0' 2022-02-07 21:51:34,059 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:34,062 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:34,062 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:34,831 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:35,270 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 73413 == 0 2022-02-07 21:51:35,272 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 73413 == 0. Received: {'172.23.121.129': 73413} for bucket 'GleamBookUsers0' 2022-02-07 21:51:35,272 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:35,273 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:35,275 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:36,401 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:36,755 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 21:51:36,755 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.120.170': 0} for bucket 'GleamBookUsers0' 2022-02-07 21:51:36,755 | infra | INFO | pool-3-thread-20 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299093.14' completed 2022-02-07 21:51:36,756 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 21:51:36,756 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299093.64 2022-02-07 21:51:36,757 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 21:51:36,760 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299094.12 2022-02-07 21:51:36,760 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 21:51:36,762 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299094.65 2022-02-07 21:51:38,211 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:38,563 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 21:51:38,563 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.124': 0} for bucket 'GleamBookUsers0' 2022-02-07 21:51:38,563 | infra | INFO | pool-3-thread-24 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299095.14' completed 2022-02-07 21:51:39,062 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:39,451 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 6816 == 0 2022-02-07 21:51:39,451 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 6816 == 0. Received: {'172.23.121.123': 6816} for bucket 'GleamBookUsers0' 2022-02-07 21:51:39,453 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:39,454 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:39,454 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:40,275 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:40,635 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 498 == 0 2022-02-07 21:51:40,637 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 498 == 0. Received: {'172.23.121.129': 498} for bucket 'GleamBookUsers0' 2022-02-07 21:51:40,637 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 21:51:40,638 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 21:51:40,638 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 21:51:44,456 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:44,815 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 21:51:44,815 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.123': 0} for bucket 'GleamBookUsers0' 2022-02-07 21:51:44,815 | infra | INFO | pool-3-thread-23 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299094.65' completed 2022-02-07 21:51:44,815 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 21:51:44,816 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299095.14 2022-02-07 21:51:44,816 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 21:51:44,818 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299095.61 2022-02-07 21:51:44,819 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 21:51:44,819 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299096.12 2022-02-07 21:51:44,819 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 21:51:44,821 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299096.59 2022-02-07 21:51:44,822 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 21:51:44,822 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299097.04 2022-02-07 21:51:45,638 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 21:51:45,980 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 21:51:45,980 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.129': 0} for bucket 'GleamBookUsers0' 2022-02-07 21:51:45,980 | infra | INFO | pool-3-thread-28 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644299097.04' completed 2022-02-07 21:51:45,982 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 21:51:46,059 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:651] All nodes in cluster: [ip:172.23.121.128 port:8091 ssh_username:root, ip:172.23.121.115 port:8091 ssh_username:root, ip:172.23.120.170 port:8091 ssh_username:root, ip:172.23.121.129 port:8091 ssh_username:root, ip:172.23.121.124 port:8091 ssh_username:root, ip:172.23.121.126 port:8091 ssh_username:root, ip:172.23.121.116 port:8091 ssh_username:root, ip:172.23.121.127 port:8091 ssh_username:root, ip:172.23.121.123 port:8091 ssh_username:root] 2022-02-07 21:51:46,062 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 21:51:46,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:46,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 21:51:46,428 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:46,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 21:51:46,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 21:51:46,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 21:51:46,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 21:51:46,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 21:51:46,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 21:51:46,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 21:51:46,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 21:51:46,719 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 21:51:46,723 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 21:51:46,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:47,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 21:51:47,082 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:47,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 21:51:47,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 21:51:47,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 21:51:47,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 21:51:47,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 21:51:47,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 21:51:47,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 21:51:47,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 21:51:47,365 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 21:51:47,371 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 21:51:47,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:47,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 21:51:47,709 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:47,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 21:51:47,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 21:51:47,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 21:51:47,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 21:51:47,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 21:51:47,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 21:51:47,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 21:51:47,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 21:51:47,990 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 21:51:47,993 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 21:51:48,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:48,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 21:51:48,345 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:48,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 21:51:48,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 21:51:48,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 21:51:48,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 21:51:48,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 21:51:48,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 21:51:48,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 21:51:48,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 21:51:48,628 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 21:51:48,632 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 21:51:48,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:48,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 21:51:48,984 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:48,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 21:51:48,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 21:51:49,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 21:51:49,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 21:51:49,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 21:51:49,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 21:51:49,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 21:51:49,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 21:51:49,282 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 21:51:49,289 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 21:51:49,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:49,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 21:51:49,654 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:49,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 21:51:49,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 21:51:49,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 21:51:49,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 21:51:49,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 21:51:49,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 21:51:49,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 21:51:49,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 21:51:49,930 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 21:51:49,934 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 21:51:50,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:50,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 21:51:50,259 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:50,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 21:51:50,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 21:51:50,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 21:51:50,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 21:51:50,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 21:51:50,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 21:51:50,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 21:51:50,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 21:51:50,543 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 21:51:50,548 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 21:51:50,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:50,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 21:51:50,868 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:50,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 21:51:50,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 21:51:50,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 21:51:50,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 21:51:50,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 21:51:50,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 21:51:50,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 21:51:50,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 21:51:51,150 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 21:51:51,154 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 21:51:51,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:51,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 21:51:51,506 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:51,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 21:51:51,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 21:51:51,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 21:51:51,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 21:51:51,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 21:51:51,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 21:51:51,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 21:51:51,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 21:51:51,782 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 21:51:51,783 | test | DEBUG | MainThread | [bucket_ready_functions:verify_stats_for_bucket:2318] Verifying stats for bucket GleamBookUsers0 2022-02-07 21:51:51,848 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:651] All nodes in cluster: [ip:172.23.121.128 port:8091 ssh_username:root, ip:172.23.121.115 port:8091 ssh_username:root, ip:172.23.120.170 port:8091 ssh_username:root, ip:172.23.121.129 port:8091 ssh_username:root, ip:172.23.121.124 port:8091 ssh_username:root, ip:172.23.121.126 port:8091 ssh_username:root, ip:172.23.121.116 port:8091 ssh_username:root, ip:172.23.121.127 port:8091 ssh_username:root, ip:172.23.121.123 port:8091 ssh_username:root] 2022-02-07 21:51:51,851 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 21:51:51,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:52,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 21:51:52,187 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:52,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 21:51:52,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 21:51:52,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 21:51:52,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 21:51:52,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 21:51:52,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 21:51:52,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 21:51:52,282 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 21:51:52,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:52,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 21:51:52,607 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:52,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 21:51:52,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 21:51:52,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 21:51:52,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 21:51:52,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 21:51:52,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 21:51:52,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 21:51:52,701 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 21:51:52,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:53,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 21:51:53,065 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:53,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 21:51:53,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 21:51:53,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 21:51:53,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 21:51:53,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 21:51:53,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 21:51:53,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 21:51:53,157 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 21:51:53,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:53,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 21:51:53,490 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:53,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 21:51:53,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 21:51:53,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 21:51:53,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 21:51:53,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 21:51:53,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 21:51:53,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 21:51:53,598 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 21:51:53,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:53,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 21:51:53,959 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:53,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 21:51:53,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 21:51:53,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 21:51:54,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 21:51:54,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 21:51:54,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 21:51:54,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 21:51:54,081 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 21:51:54,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:54,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 21:51:54,400 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:54,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 21:51:54,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 21:51:54,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 21:51:54,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 21:51:54,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 21:51:54,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 21:51:54,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 21:51:54,502 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 21:51:54,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:54,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 21:51:54,841 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:54,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 21:51:54,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 21:51:54,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 21:51:54,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 21:51:54,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 21:51:54,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 21:51:54,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 21:51:54,933 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 21:51:55,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:55,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 21:51:55,279 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:55,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 21:51:55,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 21:51:55,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 21:51:55,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 21:51:55,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 21:51:55,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 21:51:55,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 21:51:55,367 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 21:51:55,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:55,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 21:51:55,694 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:55,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 21:51:55,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 21:51:55,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 21:51:55,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 21:51:55,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 21:51:55,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 21:51:55,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 21:51:55,806 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 21:51:55,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:56,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 21:51:56,154 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:56,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 21:51:56,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 21:51:56,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 21:51:56,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 21:51:56,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 21:51:56,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 21:51:56,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 21:51:56,262 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 21:51:56,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:56,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 21:51:56,602 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:56,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 21:51:56,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 21:51:56,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 21:51:56,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 21:51:56,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 21:51:56,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 21:51:56,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 21:51:56,696 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 21:51:56,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:56,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 21:51:57,040 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:57,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 21:51:57,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 21:51:57,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 21:51:57,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 21:51:57,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 21:51:57,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 21:51:57,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 21:51:57,130 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 21:51:57,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:57,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 21:51:57,476 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:57,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 21:51:57,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 21:51:57,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 21:51:57,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 21:51:57,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 21:51:57,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 21:51:57,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 21:51:57,572 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 21:51:57,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:57,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 21:51:57,887 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:57,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 21:51:57,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 21:51:57,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 21:51:57,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 21:51:57,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 21:51:57,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 21:51:57,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 21:51:57,976 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 21:51:58,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:58,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 21:51:58,282 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:58,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 21:51:58,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 21:51:58,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 21:51:58,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 21:51:58,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 21:51:58,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 21:51:58,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 21:51:58,368 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 21:51:58,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:58,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 21:51:58,703 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:58,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 21:51:58,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 21:51:58,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 21:51:58,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 21:51:58,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 21:51:58,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 21:51:58,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 21:51:58,798 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 21:51:58,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:59,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 21:51:59,117 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:59,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 21:51:59,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 21:51:59,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 21:51:59,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 21:51:59,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 21:51:59,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 21:51:59,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 21:51:59,207 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 21:51:59,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:51:59,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 21:51:59,529 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:51:59,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 21:51:59,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 21:51:59,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 21:51:59,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 21:51:59,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 21:51:59,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 21:51:59,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 21:51:59,619 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for vb_replica_curr_items on bucket GleamBookUsers0 2022-02-07 21:51:59,619 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_vb_replica_curr_items_1644299519.62 2022-02-07 21:51:59,621 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for curr_items_tot on bucket GleamBookUsers0 2022-02-07 21:51:59,621 | infra | INFO | pool-3-thread-29 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_vb_replica_curr_items_1644299519.62' started 2022-02-07 21:51:59,624 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:51:59,625 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_curr_items_tot_1644299519.62 2022-02-07 21:51:59,625 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_vb_replica_curr_items_1644299519.62 2022-02-07 21:51:59,627 | infra | INFO | pool-3-thread-30 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_curr_items_tot_1644299519.62' started 2022-02-07 21:51:59,628 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:51:59,763 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:51:59,769 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:51:59,901 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:51:59,904 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:52:00,039 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:52:00,039 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:52:00,180 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:52:00,180 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:52:00,322 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:52:00,323 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:52:00,457 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:52:00,470 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:52:00,601 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:52:00,604 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:52:00,740 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:52:00,742 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 21:52:00,878 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 20000000 == 20000000 2022-02-07 21:52:00,878 | test | DEBUG | pool-3-thread-29 | [task:_get_all_stats_and_compare:2707] Ready: vb_replica_curr_items 20000000 == 20000000. Received: {'172.23.121.123': '2222460', '172.23.121.124': '2235940', '172.23.121.115': '2221300', '172.23.121.126': '2220600', '172.23.121.116': '2226240', '172.23.121.127': '2245920', '172.23.121.128': '2197500', '172.23.121.129': '2203020', '172.23.120.170': '2227020'} for bucket 'GleamBookUsers0' 2022-02-07 21:52:00,878 | infra | INFO | pool-3-thread-29 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_vb_replica_curr_items_1644299519.62' completed 2022-02-07 21:52:00,878 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 40000000 == 40000000 2022-02-07 21:52:00,880 | test | DEBUG | pool-3-thread-30 | [task:_get_all_stats_and_compare:2707] Ready: curr_items_tot 40000000 == 40000000. Received: {'172.23.121.123': '4462920', '172.23.121.124': '4452900', '172.23.121.115': '4445440', '172.23.121.126': '4441460', '172.23.121.116': '4461840', '172.23.121.127': '4470780', '172.23.121.128': '4400860', '172.23.121.129': '4405480', '172.23.120.170': '4458320'} for bucket 'GleamBookUsers0' 2022-02-07 21:52:00,880 | infra | INFO | pool-3-thread-30 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_curr_items_tot_1644299519.62' completed 2022-02-07 21:52:00,878 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_curr_items_tot_1644299519.62 2022-02-07 21:52:00,880 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 21:52:00,881 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 21:52:00,881 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 21:52:00,882 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 21:52:00,882 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 21:52:00,884 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 21:52:00,885 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 21:52:00,887 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 21:52:00,887 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 21:52:00,887 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 21:52:00,888 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 21:52:00,890 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 21:52:00,890 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 21:52:00,891 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 21:52:00,891 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 21:52:00,891 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 21:52:00,891 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 21:52:00,891 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 21:52:00,971 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3970] Stats: {u'opsPerSec': 0, u'diskFetches': 0, u'dataUsed': 40884961662L, u'itemCount': 20000000, u'quotaPercentUsed': 59.43466863787725, u'storageTotals': {u'hdd': {u'usedByData': 50725785151L, u'used': 147170962137L, u'quotaTotal': 1168799956992L, u'total': 1168799956992L, u'free': 892306910457L}, u'ram': {u'usedByData': 46279521032L, u'used': 102267531264L, u'quotaTotal': 78809923584L, u'total': 110647066624L, u'quotaUsed': 77866205184L, u'quotaUsedPerNode': 8651800576L, u'quotaTotalPerNode': 8756658176L}}, u'diskUsed': 50725785151L, u'vbActiveNumNonResident': 0, u'memUsed': 46279521032L} 2022-02-07 21:52:01,242 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+-----+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 20000000 | 72.52 GiB | 43.10 GiB | 47.24 GiB | 100 | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+-----+ 2022-02-07 21:52:01,242 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+---------------+---------------+---------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+---------------+---------------+---------------+---------------+ | 0 | 20000000 | 0-0 | 0-1000000 | 0-0 | 0-0 | +---------------+---------------+---------------+---------------+---------------+---------------+ 2022-02-07 21:52:01,506 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 100 2022-02-07 21:52:01,509 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 21:52:01,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:01,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 21:52:01,857 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:01,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 21:52:01,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 21:52:01,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 21:52:01,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 21:52:01,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 21:52:01,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 21:52:01,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 21:52:01,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 21:52:02,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 21:52:02,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 21:52:02,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 21:52:02,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 21:52:02,273 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 21:52:02,276 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 21:52:02,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:02,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 21:52:02,650 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:02,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 21:52:02,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 21:52:02,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 21:52:02,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 21:52:02,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 21:52:02,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 21:52:02,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 21:52:02,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 21:52:02,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 21:52:02,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 21:52:02,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 21:52:02,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 21:52:03,095 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 21:52:03,096 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 21:52:03,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:03,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 21:52:03,443 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:03,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 21:52:03,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 21:52:03,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 21:52:03,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 21:52:03,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 21:52:03,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 21:52:03,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 21:52:03,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 21:52:03,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 21:52:03,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 21:52:03,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 21:52:03,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 21:52:03,914 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 21:52:03,917 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 21:52:04,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:04,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 21:52:04,276 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:04,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 21:52:04,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 21:52:04,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 21:52:04,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 21:52:04,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 21:52:04,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 21:52:04,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 21:52:04,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 21:52:04,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 21:52:04,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 21:52:04,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 21:52:04,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 21:52:04,710 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 21:52:04,713 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 21:52:04,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:05,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 21:52:05,069 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:05,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 21:52:05,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 21:52:05,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 21:52:05,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 21:52:05,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 21:52:05,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 21:52:05,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 21:52:05,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 21:52:05,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 21:52:05,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 21:52:05,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 21:52:05,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 21:52:05,506 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 21:52:05,509 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 21:52:05,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:05,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 21:52:05,838 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:05,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 21:52:05,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 21:52:05,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 21:52:05,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 21:52:05,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 21:52:05,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 21:52:05,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 21:52:05,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 21:52:06,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 21:52:06,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 21:52:06,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 21:52:06,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 21:52:06,292 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 21:52:06,295 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 21:52:06,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:06,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 21:52:06,648 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:06,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 21:52:06,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 21:52:06,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 21:52:06,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 21:52:06,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 21:52:06,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 21:52:06,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 21:52:06,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 21:52:06,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 21:52:06,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 21:52:06,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 21:52:06,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 21:52:07,084 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 21:52:07,086 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 21:52:07,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:07,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 21:52:07,401 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:07,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 21:52:07,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 21:52:07,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 21:52:07,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 21:52:07,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 21:52:07,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 21:52:07,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 21:52:07,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 21:52:07,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 21:52:07,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 21:52:07,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 21:52:07,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 21:52:07,832 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 21:52:07,835 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 21:52:07,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:08,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 21:52:08,160 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:08,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 21:52:08,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 21:52:08,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 21:52:08,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 21:52:08,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 21:52:08,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 21:52:08,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 21:52:08,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 21:52:08,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 21:52:08,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 21:52:08,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 21:52:08,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 21:52:08,575 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 21:52:08,579 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-07 21:52:08,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:08,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-07 21:52:08,921 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:08,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-07 21:52:08,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-07 21:52:08,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-07 21:52:08,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-07 21:52:08,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-07 21:52:08,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-07 21:52:08,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-07 21:52:09,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 21:52:09,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 21:52:09,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 21:52:09,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 21:52:09,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 21:52:09,096 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-07 21:52:09,099 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-07 21:52:09,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:09,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-07 21:52:09,463 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:09,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-07 21:52:09,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-07 21:52:09,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-07 21:52:09,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-07 21:52:09,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-07 21:52:09,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-07 21:52:09,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-07 21:52:09,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 21:52:09,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 21:52:09,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 21:52:09,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 21:52:09,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 21:52:09,671 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-07 21:52:09,674 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-07 21:52:09,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:09,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-07 21:52:10,003 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:10,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-07 21:52:10,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-07 21:52:10,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-07 21:52:10,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-07 21:52:10,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-07 21:52:10,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-07 21:52:10,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-07 21:52:10,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 21:52:10,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 21:52:10,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 21:52:10,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 21:52:10,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 21:52:10,184 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-07 21:52:10,186 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-07 21:52:10,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:10,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-07 21:52:10,532 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:10,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-07 21:52:10,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-07 21:52:10,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-07 21:52:10,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-07 21:52:10,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-07 21:52:10,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-07 21:52:10,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-07 21:52:10,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 21:52:10,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 21:52:10,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 21:52:10,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 21:52:10,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 21:52:10,746 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-07 21:52:10,749 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-07 21:52:10,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:11,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-07 21:52:11,124 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:11,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-07 21:52:11,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-07 21:52:11,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-07 21:52:11,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-07 21:52:11,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-07 21:52:11,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-07 21:52:11,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-07 21:52:11,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 21:52:11,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 21:52:11,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 21:52:11,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 21:52:11,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 21:52:11,305 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-07 21:52:11,305 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for kvstore is 43429MB 2022-02-07 21:52:11,305 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for wal is 4321MB 2022-02-07 21:52:11,305 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:642] Total Disk usage for keyTree is 4618MB 2022-02-07 21:52:11,305 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:643] Total Disk usage for seqTree is 38639MB 2022-02-07 21:52:11,308 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 21:52:11,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:11,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 21:52:11,651 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:11,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 21:52:11,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 21:52:11,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 21:52:11,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 21:52:11,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 21:52:11,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 21:52:11,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 21:52:11,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 21:52:11,801 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.115 - core(s): 8 2022-02-07 21:52:11,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 21:52:11,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 21:52:12,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 21:52:12,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 21:52:12,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 21:52:12,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 21:52:12,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 21:52:12,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 21:52:12,880 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 21:52:12,882 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 21:52:12,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:13,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 21:52:13,214 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:13,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 21:52:13,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 21:52:13,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 21:52:13,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 21:52:13,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 21:52:13,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 21:52:13,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 21:52:13,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 21:52:13,362 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.116 - core(s): 8 2022-02-07 21:52:13,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 21:52:13,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 21:52:13,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 21:52:13,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 21:52:13,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 21:52:14,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 21:52:14,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 21:52:14,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 21:52:14,438 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 21:52:14,441 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 21:52:14,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:14,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 21:52:14,785 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:14,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 21:52:14,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 21:52:14,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 21:52:14,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 21:52:14,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 21:52:14,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 21:52:14,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 21:52:14,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 21:52:14,904 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.123 - core(s): 8 2022-02-07 21:52:14,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 21:52:15,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 21:52:15,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 21:52:15,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 21:52:15,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 21:52:15,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 21:52:15,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 21:52:15,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 21:52:15,964 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 21:52:15,969 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 21:52:16,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:16,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 21:52:16,315 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:16,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 21:52:16,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 21:52:16,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 21:52:16,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 21:52:16,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 21:52:16,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 21:52:16,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 21:52:16,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 21:52:16,456 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.124 - core(s): 8 2022-02-07 21:52:16,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 21:52:16,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 21:52:16,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 21:52:16,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 21:52:16,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 21:52:17,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 21:52:17,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 21:52:17,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 21:52:17,517 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 21:52:17,519 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 21:52:17,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:17,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 21:52:17,858 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:17,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 21:52:17,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 21:52:17,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 21:52:17,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 21:52:17,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 21:52:17,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 21:52:17,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 21:52:17,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 21:52:17,976 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.126 - core(s): 8 2022-02-07 21:52:17,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 21:52:18,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 21:52:18,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 21:52:18,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 21:52:18,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 21:52:18,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 21:52:18,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 21:52:18,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 21:52:19,042 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 21:52:19,046 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 21:52:19,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:19,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 21:52:19,418 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:19,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 21:52:19,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 21:52:19,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 21:52:19,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 21:52:19,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 21:52:19,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 21:52:19,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 21:52:19,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 21:52:19,540 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.127 - core(s): 8 2022-02-07 21:52:19,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 21:52:19,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 21:52:19,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 21:52:19,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 21:52:20,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 21:52:20,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 21:52:20,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 21:52:20,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 21:52:20,581 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 21:52:20,585 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 21:52:20,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:20,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 21:52:20,905 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:20,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 21:52:20,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 21:52:20,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 21:52:20,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 21:52:20,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 21:52:20,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 21:52:20,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 21:52:20,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 21:52:21,019 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.128 - core(s): 8 2022-02-07 21:52:21,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 21:52:21,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 21:52:21,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 21:52:21,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 21:52:21,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 21:52:21,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 21:52:21,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 21:52:21,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 21:52:22,112 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 21:52:22,115 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 21:52:22,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:22,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 21:52:22,434 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:22,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 21:52:22,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 21:52:22,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 21:52:22,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 21:52:22,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 21:52:22,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 21:52:22,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 21:52:22,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 21:52:22,569 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.129 - core(s): 8 2022-02-07 21:52:22,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 21:52:22,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 21:52:22,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 21:52:22,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 21:52:23,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 21:52:23,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 21:52:23,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 21:52:23,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 21:52:23,619 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 21:52:23,622 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 21:52:23,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:23,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 21:52:23,984 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:23,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 21:52:23,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 21:52:24,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 21:52:24,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 21:52:24,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 21:52:24,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 21:52:24,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 21:52:24,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 21:52:24,096 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.120.170 - core(s): 8 2022-02-07 21:52:24,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 21:52:24,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 21:52:24,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 21:52:24,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 21:52:24,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 21:52:24,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 21:52:24,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 21:52:25,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 21:52:25,134 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 21:52:25,134 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:723] {'172.23.121.115': {'rw_0:magma': {u'NReadBytesGetPerSec': 1616464.5, u'ReplayWALOffset': u'73:6176768', u'NTablesCreated': 1859, u'RecentReadIOAmpSet': 0.016343826428055763, u'ReadIOAmp': 45.8199348449707, u'BlockCacheMisses': 19822, u'NFlushes': 806, u'RecentWriteAmp': 37.95006561279297, u'NumSyncFlushes': 0, u'TotalMemUsed': 31171403, u'NBloomFilterHits': 8545744, u'Fragmentation': 0.018835170194506645, u'NumNormalFlushes': 271, u'RecentBlockCacheHitRatio': 60.59602355957031, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 693, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 1728296, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 4876053, u'CheckpointOverhead': 83644416, u'NItemsGet': 0, u'NWriteIOs': 2490, u'NReadBytes': 301973799, u'LevelStats': [{u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 5865472, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 5865472, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 0}, {u'TargetSize': 244718630, u'LogicalSize': 9784973, u'SizeRatio': 0.03998458556261123, u'NWriteBytes': 5865472, u'BloomFilterMemUsed': 0, u'NumTables': 1, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 237568}, {u'TargetSize': 973804046, u'LogicalSize': 489437261, u'SizeRatio': 0.5026034375297718, u'NWriteBytes': 788393984, u'BloomFilterMemUsed': 0, u'NumTables': 194, u'WastedSpace': 2535238, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 489431040}], u'NItemsGarbageCollected': 244893, u'NInternalFragmentationCompacts': 2, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 3979582, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 23712, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 119178, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 69654, u'CompactWriter': {u'NDataBlockWrites': 67731, u'NWriteBytes': 205864960, u'NIndexBlockWrites': 631, u'NBytesIncoming': 319293800, u'NSyncs': 0, u'NInserts': 339058, u'NWriteIOs': 488}, u'NonResidentBloomFilterSize': 0, u'IteratorMemUsed': 0, u'NWriteBytes': 794464353, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 244893, u'LogicalDiskSize': 489437261, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.7479158082586563, u'BufferMemUsed': 344064, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 479652288, u'NTableFiles': 251, u'NFileCountCompacts': 0, u'TableObjectMemUsed': 59280, u'TreeSnapshotMemoryUsed': 145920, u'TotalDiskUsage': 592490496, u'BytesIncoming': 1062237680, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 83, u'NItemsPut': 1136900, u'NSyncs': 535, u'CompactReader': {u'NReadIOs': 119178, u'NReadBytes': 301973799, u'NDataBlockReads': 118166, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1012}, u'ActiveDiskUsage': 489668608, u'NTablesDeleted': 134, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 228565, u'NWriteBytes': 588599393, u'NIndexBlockWrites': 2579, u'NBytesIncoming': 1076525632, u'NSyncs': 535, u'NInserts': 1390521, u'NWriteIOs': 2002}}, u'NReadBytes': 43903248166L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 15.866666793823242, u'NInternalFragmentationCompacts': 2, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 4533570, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 5811079, u'NTxns': 56518, u'NWriteBytes': 1240735744, u'NSegments': 75, u'NSyncs': 56518, u'NLoggedBytes': 1240735744, u'NActiveSegments': 3, u'NWrites': 58004, u'PreallocateSupported': True, u'NInserts': 1526047, u'DiskUsed': 49631232}, u'NGets': 275020, u'NReadBytesSet': 38958087, u'NReadBytesCompact': 320343733, u'IndexResidentRatio': 0.6144738793373108, u'CompactQueueSize': 0, u'NSyncsPerSec': 3.200000047683716, u'BlockCacheHitRatio': 91.79460906982422, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 43582904433L, u'BloomFilterMemUsedRatio': 0.008102858439087868, u'NonResidentBloomFilterSize': 212920, u'NWriteBytes': 2146194536, u'BytesOutgoing': 0, u'RecentReadIOAmp': 0.0, u'WriteAmp': 1.9859309196472168, u'NBlocksCached': 5113, u'LogicalDataSize': 509720470, u'ActiveBloomFilterMemUsed': 876304, u'NFileCountCompacts': 56, u'NTableFiles': 631, u'FlushQueueSize': 8, u'TableObjectMemUsed': 150480, u'BlockCacheQuota': 50137406, u'TotalDiskUsage': 652419072, u'NPurgerRuns': 0, u'ReadIOAmpSet': 0.0, u'NSyncs': 1605, u'NReadIOsPerSec': 1210.36669921875, u'NReadIOsSetPerSec': 0.8999999761581421, u'NTablesDeleted': 1086, u'BloomFilterFPR': 0.012160636484622955, u'NBloomFilterHitsPerSec': 254.43333435058594, u'WriteCacheMemUsed': 1973624, u'RecentBloomFilterFPR': 0.016365978866815567, u'BytesPerRead': 158330.109375, u'BytesIncomingPerSec': 50886.19921875, u'NReadIOsGetPerSec': 452.3666687011719, u'NumKVStores': 29, u'NGetStats': 11918, u'TotalIndexBlocksSize': 5565084, u'NGetStatsPerSec': 5.900000095367432, u'NWriteBytesCompact': 233123840, u'NReadIOsSet': 10526, u'ActiveDataSize': 509720470, u'NWriteBytesCompactPerSec': 1349632.0, u'NBloomFilterMissesPerSec': 4.233333110809326, u'NCommitBatches': 56518, u'NGetStatsComputed': 11918, u'NWriteIOs': 62788, u'NReadIOsGet': 12611924, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 51367038, u'LSMTreeObjectMemUsed': 500192, u'NSetsPerSec': 55.06666564941406, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.7666666507720947, u'ManagedFileObjectMemUsed': 60192, u'ReadAmpGet': 0.0, u'BlockCacheMemUsed': 21207832, u'NWriterCompacts': 0, u'NReadIOs': 12736545, u'WALBufferMemUsed': 5242880, u'ReadAmp': 0.0, u'keyStats': {u'NItemsTransformed': 597904, u'NTablesCreated': 583, u'ReadIOAmp': 8.932463595671143, u'MemtablesMemUsed': 188200, u'NBloomFilterHits': 8545744, u'TotalIndexBlocksSize': 679553, u'CheckpointOverhead': 12214272, u'NItemsGet': 1411920, u'NWriteIOs': 1176, u'NReadBytes': 43600803069L, u'LevelStats': [{u'TargetSize': 121634816, u'LogicalSize': 19233890, u'SizeRatio': 0.15812816291019835, u'NWriteBytes': 78700544, u'BloomFilterMemUsed': 368232, u'NumTables': 79, u'WastedSpace': 8970, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 19128320}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 26996662, u'SizeRatio': 0.2219484756732809, u'NWriteBytes': 27029504, u'BloomFilterMemUsed': 505856, u'NumTables': 28, u'WastedSpace': 37863, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 27029504}], u'NItemsGarbageCollected': 255273, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 548559, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 18240, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 12616415, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 255273, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 57746, u'CompactWriter': {u'NDataBlockWrites': 9266, u'NWriteBytes': 27029504, u'NIndexBlockWrites': 149, u'NBytesIncoming': 35017706, u'NSyncs': 0, u'NInserts': 597904, u'NWriteIOs': 66}, u'NonResidentBloomFilterSize': 211648, u'IteratorMemUsed': 0, u'NWriteBytes': 106011314, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 12611924, u'NReadBytes': 43582904433L, u'NDataBlockReads': 138688215, u'NBytesOutgoing': 0, u'NIndexBlockReads': 189060926}, u'NBloomFilterMisses': 105201, u'NItemsMerged': 0, u'LogicalDiskSize': 29637750, u'TotalBloomFilterMemUsed': 874088, u'WriteAmp': 1.865033060340399, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 874088, u'LogicalDataSize': 29637750, u'NTableFiles': 190, u'NFileCountCompacts': 28, u'TableObjectMemUsed': 45600, u'TreeSnapshotMemoryUsed': 138768, u'TotalDiskUsage': 58372096, u'BytesIncoming': 56841520, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 28, u'NItemsPut': 1136900, u'NSyncs': 535, u'CompactReader': {u'NReadIOs': 4491, u'NReadBytes': 17898636, u'NDataBlockReads': 13559, u'NBytesOutgoing': 0, u'NIndexBlockReads': 502}, u'ActiveDiskUsage': 46157824, u'NTablesDeleted': 476, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.012160636785923388, u'Writer': {u'NDataBlockWrites': 17987, u'NWriteBytes': 78981810, u'NIndexBlockWrites': 621, u'NBytesIncoming': 66976152, u'NSyncs': 535, u'NInserts': 1135248, u'NWriteIOs': 1110}}, u'NReadBytesCompactPerSec': 1921323.875, u'TableMetaMemUsed': 182035, u'NDeletes': 0, u'BlockCacheHitsPerSec': 24.399999618530273, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 105201, u'LogicalDiskSize': 519505443, u'NSets': 1136900, u'TotalBloomFilterMemUsed': 876304, u'BufferMemUsed': 344064, u'RecentReadAmp': 0.0, u'NGetsPerSec': 0.0, u'NInserts': 0, u'NCommitBatchesPerSec': 0.06666667014360428, u'FragmentationWithOverhead': 0.21872231364250183, u'NCompactsPerSec': 0.46666666865348816, u'WALMemUsed': 5308496, u'MemoryQuota': 108147512, u'NReadBytesPerSec': 3537788.25, u'TreeSnapshotMemoryUsed': 384480, u'BytesIncoming': 1080699470, u'NCompacts': 139, u'NumPersistentFlushes': 535, u'NWriteIOsPerSec': 8.100000381469727, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 0.0, u'ActiveDiskUsage': 536702976, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 5.900000095367432, u'NReadBytesSetPerSec': 3304.89990234375, u'BlockCacheHits': 221751, u'NWriteBytesPerSec': 1931134.625, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 583, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 57128, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 9478, u'CheckpointOverhead': 679936, u'NItemsGet': 636, u'NWriteIOs': 1118, u'NReadBytes': 471298, u'LevelStats': [{u'TargetSize': 121634816, u'LogicalSize': 736626, u'SizeRatio': 0.0060560456637678475, u'NWriteBytes': 4546560, u'BloomFilterMemUsed': 1320, u'NumTables': 79, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 647168}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 280348, u'SizeRatio': 0.002304833510826374, u'NWriteBytes': 229376, u'BloomFilterMemUsed': 896, u'NumTables': 28, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 229376}], u'NItemsGarbageCollected': 218107, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 5429, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 18240, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 952, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 54635, u'CompactWriter': {u'NDataBlockWrites': 28, u'NWriteBytes': 229376, u'NIndexBlockWrites': 28, u'NBytesIncoming': 61970, u'NSyncs': 0, u'NInserts': 644, u'NWriteIOs': 28}, u'NonResidentBloomFilterSize': 1272, u'IteratorMemUsed': 0, u'NWriteBytes': 4983125, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 430432, u'TotalBloomFilterMemUsed': 2216, u'WriteAmp': 0.15521534459714328, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 2216, u'LogicalDataSize': 430432, u'NTableFiles': 190, u'NFileCountCompacts': 28, u'TableObjectMemUsed': 45600, u'TreeSnapshotMemoryUsed': 99792, u'TotalDiskUsage': 1556480, u'BytesIncoming': 32104590, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 28, u'NItemsPut': 219635, u'NSyncs': 535, u'CompactReader': {u'NReadIOs': 952, u'NReadBytes': 471298, u'NDataBlockReads': 476, u'NBytesOutgoing': 0, u'NIndexBlockReads': 476}, u'ActiveDiskUsage': 876544, u'NTablesDeleted': 476, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 555, u'NWriteBytes': 4753749, u'NIndexBlockWrites': 555, u'NBytesIncoming': 460492, u'NSyncs': 535, u'NInserts': 6587, u'NWriteIOs': 1090}}}}} 2022-02-07 21:52:25,153 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:728] magma stats fragmentation result {'172.23.121.123': [0.0, 0.0, 0.0, 0.12591227889060974, 0.0, 0.06901592761278152, 0.13909728825092316, 0.0], '172.23.121.124': [0.0, 0.0, 0.0, 0.00876854732632637, 0.0, 0.04390653595328331, 0.0, 0.0], '172.23.121.115': [0.018835170194506645, 0.07703123241662979, 0.01792817749083042, 0.11341724544763565, 0.017452213913202286, 0.012231131084263325, 0.017940308898687363, 0.1079280748963356], '172.23.121.126': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], '172.23.121.116': [0.3603515923023224, 0.3774920701980591, 0.3680947422981262, 0.37252020835876465, 0.3451406955718994, 0.35776394605636597, 0.3845444619655609, 0.3567633628845215], '172.23.121.127': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], '172.23.121.128': [0.0, 0.0, 0.07665316015481949, 0.0, 0.02150210179388523, 0.0, 0.0, 0.0], '172.23.121.129': [0.0, 0.0, 0.0529823824763298, 0.0, 0.0, 0.0, 0.08972679823637009, 0.12085703015327454], '172.23.120.170': [0.0, 0.0, 0.0, 0.0, 0.08083724230527878, 0.11909717321395874, 0.0, 0.10532058775424957]} 2022-02-07 21:52:25,424 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.115 is 7.0 2022-02-07 21:52:25,686 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.116 is 7.0 2022-02-07 21:52:25,957 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.123 is 7.0 2022-02-07 21:52:26,196 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.124 is 7.0 2022-02-07 21:52:26,459 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.126 is 7.0 2022-02-07 21:52:26,944 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.127 is 7.0 2022-02-07 21:52:27,206 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.128 is 7.0 2022-02-07 21:52:27,483 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.129 is 7.0 2022-02-07 21:52:27,720 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.120.170 is 7.0 2022-02-07 21:52:27,720 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:677] KV stats fragmentation values {'172.23.121.123': 7.0, '172.23.121.124': 7.0, '172.23.121.115': 7.0, '172.23.121.126': 7.0, '172.23.121.116': 7.0, '172.23.121.127': 7.0, '172.23.121.128': 7.0, '172.23.121.129': 7.0, '172.23.120.170': 7.0} 2022-02-07 21:52:27,723 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 21:52:27,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:27,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 21:52:28,058 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:28,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 21:52:28,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 21:52:28,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 21:52:28,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 21:52:28,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 21:52:28,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 21:52:28,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 21:52:28,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 21:52:28,160 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-07 21:52:28,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 21:52:28,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:28,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 21:52:28,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:28,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:28,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 21:52:28,454 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-07 21:52:28,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 21:52:28,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:28,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 21:52:28,493 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-07 21:52:28,493 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 21:52:28,500 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 21:52:28,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:28,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 21:52:28,875 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:28,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 21:52:28,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 21:52:28,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 21:52:28,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 21:52:28,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 21:52:28,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 21:52:28,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 21:52:28,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 21:52:28,980 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-07 21:52:29,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 21:52:29,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:29,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 21:52:29,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:29,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:29,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 21:52:29,197 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-07 21:52:29,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 21:52:29,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:29,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 21:52:29,239 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-07 21:52:29,239 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 21:52:29,243 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 21:52:29,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:29,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 21:52:29,582 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:29,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 21:52:29,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 21:52:29,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 21:52:29,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 21:52:29,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 21:52:29,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 21:52:29,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 21:52:29,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 21:52:29,683 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-07 21:52:29,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 21:52:29,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:29,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 21:52:29,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:29,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:29,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 21:52:29,832 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-07 21:52:29,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 21:52:29,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:29,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 21:52:29,871 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-07 21:52:29,871 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 21:52:29,874 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 21:52:29,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:30,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 21:52:30,230 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:30,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 21:52:30,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 21:52:30,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 21:52:30,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 21:52:30,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 21:52:30,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 21:52:30,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 21:52:30,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 21:52:30,334 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-07 21:52:30,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 21:52:30,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:30,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 21:52:30,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:30,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:30,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 21:52:30,565 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-07 21:52:30,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 21:52:30,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:30,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 21:52:30,612 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-07 21:52:30,614 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 21:52:30,617 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 21:52:30,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:30,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 21:52:30,993 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:30,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 21:52:31,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 21:52:31,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 21:52:31,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 21:52:31,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 21:52:31,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 21:52:31,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 21:52:31,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 21:52:31,108 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-07 21:52:31,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 21:52:31,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:31,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 21:52:31,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:31,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:31,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 21:52:31,351 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-07 21:52:31,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 21:52:31,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:31,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 21:52:31,397 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-07 21:52:31,397 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 21:52:31,400 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 21:52:31,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:31,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 21:52:31,740 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:31,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 21:52:31,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 21:52:31,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 21:52:31,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 21:52:31,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 21:52:31,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 21:52:31,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 21:52:31,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 21:52:31,845 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-07 21:52:31,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 21:52:31,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:31,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 21:52:32,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:32,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:32,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 21:52:32,082 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-07 21:52:32,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 21:52:32,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:32,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 21:52:32,124 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-07 21:52:32,124 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 21:52:32,127 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 21:52:32,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:32,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 21:52:32,443 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:32,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 21:52:32,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 21:52:32,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 21:52:32,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 21:52:32,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 21:52:32,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 21:52:32,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 21:52:32,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 21:52:32,558 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-07 21:52:32,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 21:52:32,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:32,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 21:52:32,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:32,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:32,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 21:52:32,759 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-07 21:52:32,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 21:52:32,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:32,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 21:52:32,802 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-07 21:52:32,803 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 21:52:32,805 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 21:52:32,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:33,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 21:52:33,128 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:33,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 21:52:33,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 21:52:33,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 21:52:33,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 21:52:33,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 21:52:33,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 21:52:33,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 21:52:33,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 21:52:33,232 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-07 21:52:33,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 21:52:33,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:33,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 21:52:33,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:33,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:33,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 21:52:33,392 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-07 21:52:33,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 21:52:33,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:33,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 21:52:33,444 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-07 21:52:33,446 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 21:52:33,450 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 21:52:33,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:33,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 21:52:33,785 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:33,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 21:52:33,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 21:52:33,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 21:52:33,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 21:52:33,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 21:52:33,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 21:52:33,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 21:52:33,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 21:52:33,887 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-07 21:52:33,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 21:52:33,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:33,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 21:52:34,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:34,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:34,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 21:52:34,066 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-07 21:52:34,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 21:52:34,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:34,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 21:52:34,104 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-07 21:52:34,104 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 21:52:34,107 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-07 21:52:34,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:34,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-07 21:52:34,447 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:34,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-07 21:52:34,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-07 21:52:34,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-07 21:52:34,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-07 21:52:34,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-07 21:52:34,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-07 21:52:34,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-07 21:52:34,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 21:52:34,551 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-07 21:52:34,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 21:52:34,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:34,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 21:52:34,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:34,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:34,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 21:52:34,684 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-07 21:52:34,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 21:52:34,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:34,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 21:52:34,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:34,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-07 21:52:34,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:34,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:34,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:34,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:34,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:34,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:34,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:34,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:34,845 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-07 21:52:34,848 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-07 21:52:34,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:35,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-07 21:52:35,190 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:35,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-07 21:52:35,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-07 21:52:35,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-07 21:52:35,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-07 21:52:35,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-07 21:52:35,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-07 21:52:35,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-07 21:52:35,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 21:52:35,290 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-07 21:52:35,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 21:52:35,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:35,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 21:52:35,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:35,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:35,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 21:52:35,407 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-07 21:52:35,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 21:52:35,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:35,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 21:52:35,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:35,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-07 21:52:35,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:35,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:35,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:35,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:35,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:35,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:35,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:35,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:35,586 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-07 21:52:35,589 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-07 21:52:35,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:35,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-07 21:52:35,914 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:35,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-07 21:52:35,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-07 21:52:35,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-07 21:52:35,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-07 21:52:35,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-07 21:52:35,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-07 21:52:35,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-07 21:52:36,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 21:52:36,013 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-07 21:52:36,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 21:52:36,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:36,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 21:52:36,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:36,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:36,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 21:52:36,131 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-07 21:52:36,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 21:52:36,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:36,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 21:52:36,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:36,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-07 21:52:36,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:36,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:36,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:36,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:36,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:36,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:36,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:36,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:36,286 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-07 21:52:36,288 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-07 21:52:36,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:36,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-07 21:52:36,628 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:36,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-07 21:52:36,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-07 21:52:36,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-07 21:52:36,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-07 21:52:36,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-07 21:52:36,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-07 21:52:36,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-07 21:52:36,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 21:52:36,727 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-07 21:52:36,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 21:52:36,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:36,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 21:52:36,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:36,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:36,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 21:52:36,845 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-07 21:52:36,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 21:52:36,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:36,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 21:52:36,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:36,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-07 21:52:36,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:36,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:36,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:36,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:36,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:36,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:36,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:36,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:37,003 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-07 21:52:37,006 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-07 21:52:37,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 21:52:37,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-07 21:52:37,332 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 21:52:37,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-07 21:52:37,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-07 21:52:37,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-07 21:52:37,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-07 21:52:37,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-07 21:52:37,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-07 21:52:37,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-07 21:52:37,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 21:52:37,433 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-07 21:52:37,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 21:52:37,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:37,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 21:52:37,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:37,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 21:52:37,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 21:52:37,555 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-07 21:52:37,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 21:52:37,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 21:52:37,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 21:52:37,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:37,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-07 21:52:37,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:37,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:37,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:37,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:37,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:37,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:37,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:37,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 21:52:37,721 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 ############################################################ # # Step 2.1: Update 1000000 RandonKey keys to create 50 percent fragmentation # ############################################################ ############################################################ # # RAM FootPrint: 3450.96484375 # ############################################################ Read Start: 0 Read End: 0 Update Start: 0 Update End: 1000000 Expiry Start: 0 Expiry End: 0 Delete Start: 0 Delete End: 0 Create Start: 0 Create End: 0 Final Start: 0 Final End: 1000000 2022-02-07 21:52:38,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:38,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection9_10_1644299559.16 2022-02-07 21:52:39,173 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:39,173 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection8_9_1644299560.17 2022-02-07 21:52:40,187 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:40,187 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection19_8_1644299561.19 2022-02-07 21:52:41,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:41,201 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection13_7_1644299562.2 2022-02-07 21:52:42,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:42,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection14_6_1644299563.22 2022-02-07 21:52:43,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:43,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection11_5_1644299564.23 2022-02-07 21:52:44,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:44,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection12_4_1644299565.25 2022-02-07 21:52:45,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:45,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection17_3_1644299566.26 2022-02-07 21:52:46,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:46,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection18_2_1644299567.32 2022-02-07 21:52:47,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:47,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection15_1_1644299568.34 2022-02-07 21:52:48,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:48,346 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:52:49,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:49,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:52:50,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:50,369 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:52:51,381 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:51,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:52:52,392 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:52,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:52:53,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:53,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:52:54,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:54,418 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:52:55,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:55,431 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:52:56,441 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:56,443 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 21:52:57,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 21:52:57,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Loader_GleamBookUsers0__default_VolumeCollection8_9_1644299560.17 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection16_0_1644299569.35 Loader_GleamBookUsers0__default_VolumeCollection9_10_1644299559.16 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection7_-1_1644299570.36 Loader_GleamBookUsers0__default_VolumeCollection13_7_1644299562.2 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection6_-2_1644299571.37 Loader_GleamBookUsers0__default_VolumeCollection19_8_1644299561.19 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection5_-3_1644299572.38 Loader_GleamBookUsers0__default_VolumeCollection14_6_1644299563.22 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection4_-4_1644299573.39 Loader_GleamBookUsers0__default_VolumeCollection17_3_1644299566.26 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection3_-5_1644299574.41 Loader_GleamBookUsers0__default_VolumeCollection11_5_1644299564.23 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection2_-6_1644299575.42 Loader_GleamBookUsers0__default_VolumeCollection12_4_1644299565.25 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection10_-7_1644299576.43 Loader_GleamBookUsers0__default_VolumeCollection18_2_1644299567.32 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection1_-8_1644299577.44 Loader_GleamBookUsers0__default_VolumeCollection15_1_1644299568.34 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection0_-9_1644299578.45 Loader_GleamBookUsers0__default_VolumeCollection16_0_1644299569.35 is completed! Loader_GleamBookUsers0__default_VolumeCollection10_-7_1644299576.43 is completed! Loader_GleamBookUsers0__default_VolumeCollection6_-2_1644299571.37 is completed! Loader_GleamBookUsers0__default_VolumeCollection2_-6_1644299575.42 is completed! Loader_GleamBookUsers0__default_VolumeCollection5_-3_1644299572.38 is completed! Loader_GleamBookUsers0__default_VolumeCollection3_-5_1644299574.41 is completed! Loader_GleamBookUsers0__default_VolumeCollection0_-9_1644299578.45 is completed! Loader_GleamBookUsers0__default_VolumeCollection7_-1_1644299570.36 is completed! Loader_GleamBookUsers0__default_VolumeCollection4_-4_1644299573.39 is completed! Loader_GleamBookUsers0__default_VolumeCollection1_-8_1644299577.44 is completed! 2022-02-07 22:30:20,634 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:651] All nodes in cluster: [ip:172.23.121.128 port:8091 ssh_username:root, ip:172.23.121.115 port:8091 ssh_username:root, ip:172.23.120.170 port:8091 ssh_username:root, ip:172.23.121.129 port:8091 ssh_username:root, ip:172.23.121.124 port:8091 ssh_username:root, ip:172.23.121.126 port:8091 ssh_username:root, ip:172.23.121.116 port:8091 ssh_username:root, ip:172.23.121.127 port:8091 ssh_username:root, ip:172.23.121.123 port:8091 ssh_username:root] 2022-02-07 22:30:20,638 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 22:30:20,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:30:21,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 22:30:21,167 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:30:21,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 22:30:21,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 22:30:21,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 22:30:21,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 22:30:21,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 22:30:21,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 22:30:21,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 22:30:21,311 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 22:30:21,311 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301821.31 2022-02-07 22:30:21,312 | infra | INFO | pool-3-thread-1 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301821.31' started 2022-02-07 22:30:21,313 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:21,315 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 22:30:21,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:30:21,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 22:30:21,683 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:30:21,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 22:30:21,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 22:30:21,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 22:30:21,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 22:30:21,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 22:30:21,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 22:30:21,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 22:30:21,783 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 22:30:21,783 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301821.78 2022-02-07 22:30:21,783 | infra | INFO | pool-3-thread-2 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301821.78' started 2022-02-07 22:30:21,786 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:21,788 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 22:30:21,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:30:22,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 22:30:22,148 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:30:22,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 22:30:22,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 22:30:22,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 22:30:22,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 22:30:22,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 22:30:22,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 22:30:22,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 22:30:22,236 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 22:30:22,236 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301822.24 2022-02-07 22:30:22,236 | infra | INFO | pool-3-thread-3 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301822.24' started 2022-02-07 22:30:22,240 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:22,242 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 22:30:22,599 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 22:30:22,601 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.116': 0} for bucket 'GleamBookUsers0' 2022-02-07 22:30:22,601 | infra | INFO | pool-3-thread-3 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301822.24' completed 2022-02-07 22:30:22,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:30:23,125 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 544712 == 0 2022-02-07 22:30:23,127 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 544712 == 0. Received: {'172.23.121.115': 544712} for bucket 'GleamBookUsers0' 2022-02-07 22:30:23,127 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:23,128 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:23,128 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:23,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 22:30:23,270 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:30:23,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 22:30:23,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 22:30:23,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 22:30:23,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 22:30:23,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 22:30:23,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 22:30:23,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 22:30:23,394 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 22:30:23,394 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301823.39 2022-02-07 22:30:23,395 | infra | INFO | pool-3-thread-4 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301823.39' started 2022-02-07 22:30:23,398 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:23,401 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 22:30:23,979 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 2064670 == 0 2022-02-07 22:30:23,979 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2064670 == 0. Received: {'172.23.120.170': 2064670} for bucket 'GleamBookUsers0' 2022-02-07 22:30:23,980 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:23,980 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:23,982 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:24,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:30:24,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 22:30:24,423 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:30:24,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 22:30:24,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 22:30:24,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 22:30:24,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 22:30:24,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 22:30:24,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 22:30:24,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 22:30:24,559 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 22:30:24,559 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301824.56 2022-02-07 22:30:24,559 | infra | INFO | pool-3-thread-5 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301824.56' started 2022-02-07 22:30:24,562 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 22:30:24,565 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:24,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:30:24,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 22:30:25,006 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:30:25,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 22:30:25,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 22:30:25,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 22:30:25,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 22:30:25,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 22:30:25,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 22:30:25,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 22:30:25,102 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 22:30:25,102 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301825.1 2022-02-07 22:30:25,102 | infra | INFO | pool-3-thread-6 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301825.1' started 2022-02-07 22:30:25,104 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:25,107 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 22:30:25,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:30:25,270 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1357994 == 0 2022-02-07 22:30:25,272 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1357994 == 0. Received: {'172.23.121.123': 1357994} for bucket 'GleamBookUsers0' 2022-02-07 22:30:25,272 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:25,273 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:25,273 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:25,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 22:30:25,497 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:30:25,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 22:30:25,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 22:30:25,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 22:30:25,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 22:30:25,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 22:30:25,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 22:30:25,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 22:30:25,602 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 22:30:25,602 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301825.6 2022-02-07 22:30:25,602 | infra | INFO | pool-3-thread-7 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301825.6' started 2022-02-07 22:30:25,605 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:25,607 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 22:30:25,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:30:25,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 22:30:25,960 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:30:25,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 22:30:25,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 22:30:25,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 22:30:26,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 22:30:26,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 22:30:26,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 22:30:26,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 22:30:26,069 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 22:30:26,069 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301826.07 2022-02-07 22:30:26,069 | infra | INFO | pool-3-thread-8 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301826.07' started 2022-02-07 22:30:26,072 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:26,075 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 22:30:26,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:30:26,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 22:30:26,539 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:30:26,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 22:30:26,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 22:30:26,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 22:30:26,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 22:30:26,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 22:30:26,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 22:30:26,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 22:30:26,660 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 22:30:26,660 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301826.66 2022-02-07 22:30:26,660 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301821.31 2022-02-07 22:30:26,660 | infra | INFO | pool-3-thread-9 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301826.66' started 2022-02-07 22:30:26,661 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:26,703 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1902814 == 0 2022-02-07 22:30:26,703 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1902814 == 0. Received: {'172.23.121.124': 1902814} for bucket 'GleamBookUsers0' 2022-02-07 22:30:26,703 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:26,707 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:26,707 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:26,957 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1417080 == 0 2022-02-07 22:30:26,957 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1417080 == 0. Received: {'172.23.121.126': 1417080} for bucket 'GleamBookUsers0' 2022-02-07 22:30:26,957 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:26,959 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:26,959 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:27,516 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1441269 == 0 2022-02-07 22:30:27,516 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1441269 == 0. Received: {'172.23.121.127': 1441269} for bucket 'GleamBookUsers0' 2022-02-07 22:30:27,516 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:27,519 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:27,519 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:28,072 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1623994 == 0 2022-02-07 22:30:28,072 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1623994 == 0. Received: {'172.23.121.128': 1623994} for bucket 'GleamBookUsers0' 2022-02-07 22:30:28,072 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:28,075 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:28,075 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:28,130 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:28,982 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:29,003 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 2038363 == 0 2022-02-07 22:30:29,003 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2038363 == 0. Received: {'172.23.121.129': 2038363} for bucket 'GleamBookUsers0' 2022-02-07 22:30:29,003 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:29,007 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:29,007 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:29,289 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 513637 == 0 2022-02-07 22:30:29,290 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 513637 == 0. Received: {'172.23.121.115': 513637} for bucket 'GleamBookUsers0' 2022-02-07 22:30:29,290 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:29,292 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:29,292 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:30,275 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:31,431 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 2028124 == 0 2022-02-07 22:30:31,431 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2028124 == 0. Received: {'172.23.120.170': 2028124} for bucket 'GleamBookUsers0' 2022-02-07 22:30:31,431 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:31,434 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:31,434 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:31,709 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:31,960 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:31,970 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1348916 == 0 2022-02-07 22:30:31,970 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1348916 == 0. Received: {'172.23.121.123': 1348916} for bucket 'GleamBookUsers0' 2022-02-07 22:30:31,970 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:31,971 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:31,971 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:32,519 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:33,075 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:33,950 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1404024 == 0 2022-02-07 22:30:33,951 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1404024 == 0. Received: {'172.23.121.126': 1404024} for bucket 'GleamBookUsers0' 2022-02-07 22:30:33,951 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:33,954 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:33,954 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:34,005 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1895592 == 0 2022-02-07 22:30:34,005 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1895592 == 0. Received: {'172.23.121.124': 1895592} for bucket 'GleamBookUsers0' 2022-02-07 22:30:34,006 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:34,007 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:34,007 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:34,009 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:34,292 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:34,322 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1419663 == 0 2022-02-07 22:30:34,322 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1419663 == 0. Received: {'172.23.121.127': 1419663} for bucket 'GleamBookUsers0' 2022-02-07 22:30:34,322 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:34,325 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:34,325 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:34,884 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1547305 == 0 2022-02-07 22:30:34,884 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1547305 == 0. Received: {'172.23.121.128': 1547305} for bucket 'GleamBookUsers0' 2022-02-07 22:30:34,884 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:34,887 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:34,887 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:35,453 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 491235 == 0 2022-02-07 22:30:35,453 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 491235 == 0. Received: {'172.23.121.115': 491235} for bucket 'GleamBookUsers0' 2022-02-07 22:30:35,453 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:35,456 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:35,456 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:36,232 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1990943 == 0 2022-02-07 22:30:36,232 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1990943 == 0. Received: {'172.23.121.129': 1990943} for bucket 'GleamBookUsers0' 2022-02-07 22:30:36,232 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:36,233 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:36,233 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:36,434 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:36,973 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:38,688 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1325463 == 0 2022-02-07 22:30:38,690 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1325463 == 0. Received: {'172.23.121.123': 1325463} for bucket 'GleamBookUsers0' 2022-02-07 22:30:38,690 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:38,691 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:38,691 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:38,954 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:39,009 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:39,022 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 2005586 == 0 2022-02-07 22:30:39,022 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2005586 == 0. Received: {'172.23.120.170': 2005586} for bucket 'GleamBookUsers0' 2022-02-07 22:30:39,023 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:39,023 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:39,025 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:39,325 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:39,888 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:40,456 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:40,513 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1392344 == 0 2022-02-07 22:30:40,513 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1392344 == 0. Received: {'172.23.121.126': 1392344} for bucket 'GleamBookUsers0' 2022-02-07 22:30:40,513 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:40,515 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:40,515 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:41,124 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1877184 == 0 2022-02-07 22:30:41,124 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1877184 == 0. Received: {'172.23.121.124': 1877184} for bucket 'GleamBookUsers0' 2022-02-07 22:30:41,124 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:41,127 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:41,128 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:41,184 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1374539 == 0 2022-02-07 22:30:41,184 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1374539 == 0. Received: {'172.23.121.127': 1374539} for bucket 'GleamBookUsers0' 2022-02-07 22:30:41,184 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:41,187 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:41,187 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:41,234 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:41,686 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 478377 == 0 2022-02-07 22:30:41,687 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 478377 == 0. Received: {'172.23.121.115': 478377} for bucket 'GleamBookUsers0' 2022-02-07 22:30:41,687 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:41,688 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:41,690 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:41,775 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1513815 == 0 2022-02-07 22:30:41,775 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1513815 == 0. Received: {'172.23.121.128': 1513815} for bucket 'GleamBookUsers0' 2022-02-07 22:30:41,775 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:41,776 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:41,776 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:43,457 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1990943 == 0 2022-02-07 22:30:43,457 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1990943 == 0. Received: {'172.23.121.129': 1990943} for bucket 'GleamBookUsers0' 2022-02-07 22:30:43,457 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:43,460 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:43,460 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:43,693 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:44,025 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:45,342 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1318828 == 0 2022-02-07 22:30:45,342 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1318828 == 0. Received: {'172.23.121.123': 1318828} for bucket 'GleamBookUsers0' 2022-02-07 22:30:45,342 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:45,345 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:45,345 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:45,516 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:46,128 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:46,188 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:46,503 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1954000 == 0 2022-02-07 22:30:46,503 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1954000 == 0. Received: {'172.23.120.170': 1954000} for bucket 'GleamBookUsers0' 2022-02-07 22:30:46,505 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:46,506 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:46,506 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:46,690 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:46,778 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:47,071 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1386250 == 0 2022-02-07 22:30:47,071 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1386250 == 0. Received: {'172.23.121.126': 1386250} for bucket 'GleamBookUsers0' 2022-02-07 22:30:47,071 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:47,073 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:47,073 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:48,000 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 462551 == 0 2022-02-07 22:30:48,000 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 462551 == 0. Received: {'172.23.121.115': 462551} for bucket 'GleamBookUsers0' 2022-02-07 22:30:48,000 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:48,003 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:48,003 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:48,323 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1355031 == 0 2022-02-07 22:30:48,325 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1355031 == 0. Received: {'172.23.121.127': 1355031} for bucket 'GleamBookUsers0' 2022-02-07 22:30:48,325 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:48,326 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:48,326 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:48,427 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1850353 == 0 2022-02-07 22:30:48,427 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1850353 == 0. Received: {'172.23.121.124': 1850353} for bucket 'GleamBookUsers0' 2022-02-07 22:30:48,427 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:48,430 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:48,430 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:48,460 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:48,608 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1491130 == 0 2022-02-07 22:30:48,608 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1491130 == 0. Received: {'172.23.121.128': 1491130} for bucket 'GleamBookUsers0' 2022-02-07 22:30:48,608 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:48,609 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:48,609 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:50,345 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:50,591 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1951605 == 0 2022-02-07 22:30:50,591 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1951605 == 0. Received: {'172.23.121.129': 1951605} for bucket 'GleamBookUsers0' 2022-02-07 22:30:50,591 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:50,592 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:50,592 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:51,507 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:51,917 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1294399 == 0 2022-02-07 22:30:51,917 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1294399 == 0. Received: {'172.23.121.123': 1294399} for bucket 'GleamBookUsers0' 2022-02-07 22:30:51,918 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:51,920 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:51,921 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:52,073 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:53,005 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:53,328 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:53,430 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:53,611 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:53,706 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1380303 == 0 2022-02-07 22:30:53,706 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1380303 == 0. Received: {'172.23.121.126': 1380303} for bucket 'GleamBookUsers0' 2022-02-07 22:30:53,706 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:53,707 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:53,709 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:54,105 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1923230 == 0 2022-02-07 22:30:54,105 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1923230 == 0. Received: {'172.23.120.170': 1923230} for bucket 'GleamBookUsers0' 2022-02-07 22:30:54,105 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:54,107 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:54,108 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:54,213 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 448293 == 0 2022-02-07 22:30:54,213 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 448293 == 0. Received: {'172.23.121.115': 448293} for bucket 'GleamBookUsers0' 2022-02-07 22:30:54,213 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:54,214 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:54,214 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:54,944 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1306443 == 0 2022-02-07 22:30:54,944 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1306443 == 0. Received: {'172.23.121.127': 1306443} for bucket 'GleamBookUsers0' 2022-02-07 22:30:54,944 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:54,947 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:54,947 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:55,467 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1484806 == 0 2022-02-07 22:30:55,467 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1484806 == 0. Received: {'172.23.121.128': 1484806} for bucket 'GleamBookUsers0' 2022-02-07 22:30:55,467 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:55,470 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:55,470 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:55,526 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1830775 == 0 2022-02-07 22:30:55,526 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1830775 == 0. Received: {'172.23.121.124': 1830775} for bucket 'GleamBookUsers0' 2022-02-07 22:30:55,526 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:55,529 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:55,529 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:55,594 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:56,921 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:57,849 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1926999 == 0 2022-02-07 22:30:57,849 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1926999 == 0. Received: {'172.23.121.129': 1926999} for bucket 'GleamBookUsers0' 2022-02-07 22:30:57,849 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:57,851 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:57,851 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:58,599 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1263063 == 0 2022-02-07 22:30:58,601 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1263063 == 0. Received: {'172.23.121.123': 1263063} for bucket 'GleamBookUsers0' 2022-02-07 22:30:58,601 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:30:58,604 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:30:58,604 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:30:58,710 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:59,108 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:59,216 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:30:59,947 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:00,296 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1349916 == 0 2022-02-07 22:31:00,296 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1349916 == 0. Received: {'172.23.121.126': 1349916} for bucket 'GleamBookUsers0' 2022-02-07 22:31:00,296 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:00,299 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:00,299 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:00,336 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 429521 == 0 2022-02-07 22:31:00,338 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 429521 == 0. Received: {'172.23.121.115': 429521} for bucket 'GleamBookUsers0' 2022-02-07 22:31:00,338 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:00,339 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:00,339 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:00,470 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:00,529 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:01,578 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1912168 == 0 2022-02-07 22:31:01,578 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1912168 == 0. Received: {'172.23.120.170': 1912168} for bucket 'GleamBookUsers0' 2022-02-07 22:31:01,578 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:01,582 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:01,582 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:01,637 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1262514 == 0 2022-02-07 22:31:01,637 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1262514 == 0. Received: {'172.23.121.127': 1262514} for bucket 'GleamBookUsers0' 2022-02-07 22:31:01,637 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:01,640 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:01,640 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:02,180 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1455087 == 0 2022-02-07 22:31:02,180 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1455087 == 0. Received: {'172.23.121.128': 1455087} for bucket 'GleamBookUsers0' 2022-02-07 22:31:02,180 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:02,184 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:02,184 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:02,482 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1758493 == 0 2022-02-07 22:31:02,482 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1758493 == 0. Received: {'172.23.121.124': 1758493} for bucket 'GleamBookUsers0' 2022-02-07 22:31:02,483 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:02,483 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:02,484 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:02,852 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:03,605 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:05,134 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1918937 == 0 2022-02-07 22:31:05,134 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1918937 == 0. Received: {'172.23.121.129': 1918937} for bucket 'GleamBookUsers0' 2022-02-07 22:31:05,134 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:05,137 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:05,137 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:05,193 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1238908 == 0 2022-02-07 22:31:05,193 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1238908 == 0. Received: {'172.23.121.123': 1238908} for bucket 'GleamBookUsers0' 2022-02-07 22:31:05,193 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:05,194 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:05,194 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:05,299 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:05,339 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:06,371 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 422191 == 0 2022-02-07 22:31:06,371 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 422191 == 0. Received: {'172.23.121.115': 422191} for bucket 'GleamBookUsers0' 2022-02-07 22:31:06,371 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:06,374 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:06,374 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:06,582 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:06,640 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:06,848 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1346374 == 0 2022-02-07 22:31:06,848 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1346374 == 0. Received: {'172.23.121.126': 1346374} for bucket 'GleamBookUsers0' 2022-02-07 22:31:06,848 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:06,851 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:06,851 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:07,184 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:07,484 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:08,153 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1240938 == 0 2022-02-07 22:31:08,154 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1240938 == 0. Received: {'172.23.121.127': 1240938} for bucket 'GleamBookUsers0' 2022-02-07 22:31:08,154 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:08,155 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:08,155 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:09,052 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1451565 == 0 2022-02-07 22:31:09,052 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1451565 == 0. Received: {'172.23.121.128': 1451565} for bucket 'GleamBookUsers0' 2022-02-07 22:31:09,052 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:09,055 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:09,055 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:09,104 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1901754 == 0 2022-02-07 22:31:09,104 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1901754 == 0. Received: {'172.23.120.170': 1901754} for bucket 'GleamBookUsers0' 2022-02-07 22:31:09,104 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:09,107 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:09,107 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:09,499 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1734986 == 0 2022-02-07 22:31:09,500 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1734986 == 0. Received: {'172.23.121.124': 1734986} for bucket 'GleamBookUsers0' 2022-02-07 22:31:09,500 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:09,502 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:09,502 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:10,138 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:10,196 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:11,378 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:11,674 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1226538 == 0 2022-02-07 22:31:11,674 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1226538 == 0. Received: {'172.23.121.123': 1226538} for bucket 'GleamBookUsers0' 2022-02-07 22:31:11,674 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:11,677 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:11,677 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:11,851 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:12,431 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1882599 == 0 2022-02-07 22:31:12,431 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1882599 == 0. Received: {'172.23.121.129': 1882599} for bucket 'GleamBookUsers0' 2022-02-07 22:31:12,433 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:12,434 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:12,434 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:12,470 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 403705 == 0 2022-02-07 22:31:12,470 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 403705 == 0. Received: {'172.23.121.115': 403705} for bucket 'GleamBookUsers0' 2022-02-07 22:31:12,471 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:12,473 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:12,473 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:13,155 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:13,305 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1296961 == 0 2022-02-07 22:31:13,306 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1296961 == 0. Received: {'172.23.121.126': 1296961} for bucket 'GleamBookUsers0' 2022-02-07 22:31:13,306 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:13,309 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:13,309 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:14,055 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:14,107 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:14,503 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:14,676 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1231877 == 0 2022-02-07 22:31:14,676 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1231877 == 0. Received: {'172.23.121.127': 1231877} for bucket 'GleamBookUsers0' 2022-02-07 22:31:14,677 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:14,678 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:14,678 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:15,717 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1444619 == 0 2022-02-07 22:31:15,719 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1444619 == 0. Received: {'172.23.121.128': 1444619} for bucket 'GleamBookUsers0' 2022-02-07 22:31:15,719 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:15,720 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:15,720 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:16,576 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1700636 == 0 2022-02-07 22:31:16,578 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1700636 == 0. Received: {'172.23.121.124': 1700636} for bucket 'GleamBookUsers0' 2022-02-07 22:31:16,578 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:16,579 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:16,581 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:16,596 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1863188 == 0 2022-02-07 22:31:16,596 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1863188 == 0. Received: {'172.23.120.170': 1863188} for bucket 'GleamBookUsers0' 2022-02-07 22:31:16,596 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:16,598 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:16,598 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:16,677 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:17,434 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:17,473 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:18,105 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1214444 == 0 2022-02-07 22:31:18,105 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1214444 == 0. Received: {'172.23.121.123': 1214444} for bucket 'GleamBookUsers0' 2022-02-07 22:31:18,105 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:18,107 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:18,108 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:18,309 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:18,513 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 391533 == 0 2022-02-07 22:31:18,513 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 391533 == 0. Received: {'172.23.121.115': 391533} for bucket 'GleamBookUsers0' 2022-02-07 22:31:18,513 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:18,516 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:18,516 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:19,680 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:19,730 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1868733 == 0 2022-02-07 22:31:19,730 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1868733 == 0. Received: {'172.23.121.129': 1868733} for bucket 'GleamBookUsers0' 2022-02-07 22:31:19,732 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:19,733 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:19,734 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:19,842 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1270213 == 0 2022-02-07 22:31:19,842 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1270213 == 0. Received: {'172.23.121.126': 1270213} for bucket 'GleamBookUsers0' 2022-02-07 22:31:19,842 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:19,844 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:19,844 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:20,721 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:21,147 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1218039 == 0 2022-02-07 22:31:21,148 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1218039 == 0. Received: {'172.23.121.127': 1218039} for bucket 'GleamBookUsers0' 2022-02-07 22:31:21,148 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:21,150 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:21,150 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:21,585 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:21,602 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:22,369 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1421315 == 0 2022-02-07 22:31:22,369 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1421315 == 0. Received: {'172.23.121.128': 1421315} for bucket 'GleamBookUsers0' 2022-02-07 22:31:22,369 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:22,371 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:22,371 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:23,108 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:23,417 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1673358 == 0 2022-02-07 22:31:23,418 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1673358 == 0. Received: {'172.23.121.124': 1673358} for bucket 'GleamBookUsers0' 2022-02-07 22:31:23,418 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:23,421 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:23,421 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:23,516 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:23,940 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1857790 == 0 2022-02-07 22:31:23,940 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1857790 == 0. Received: {'172.23.120.170': 1857790} for bucket 'GleamBookUsers0' 2022-02-07 22:31:23,940 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:23,943 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:23,943 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:24,342 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1152062 == 0 2022-02-07 22:31:24,342 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1152062 == 0. Received: {'172.23.121.123': 1152062} for bucket 'GleamBookUsers0' 2022-02-07 22:31:24,342 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:24,345 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:24,345 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:24,582 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 375203 == 0 2022-02-07 22:31:24,582 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 375203 == 0. Received: {'172.23.121.115': 375203} for bucket 'GleamBookUsers0' 2022-02-07 22:31:24,582 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:24,585 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:24,585 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:24,734 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:24,845 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:26,150 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:26,381 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1268927 == 0 2022-02-07 22:31:26,381 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1268927 == 0. Received: {'172.23.121.126': 1268927} for bucket 'GleamBookUsers0' 2022-02-07 22:31:26,382 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:26,384 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:26,385 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:26,927 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1831051 == 0 2022-02-07 22:31:26,927 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1831051 == 0. Received: {'172.23.121.129': 1831051} for bucket 'GleamBookUsers0' 2022-02-07 22:31:26,928 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:26,930 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:26,930 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:27,372 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:27,688 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1208591 == 0 2022-02-07 22:31:27,688 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1208591 == 0. Received: {'172.23.121.127': 1208591} for bucket 'GleamBookUsers0' 2022-02-07 22:31:27,690 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:27,691 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:27,691 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:28,421 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:28,944 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:29,062 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1414611 == 0 2022-02-07 22:31:29,062 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1414611 == 0. Received: {'172.23.121.128': 1414611} for bucket 'GleamBookUsers0' 2022-02-07 22:31:29,062 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:29,065 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:29,065 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:29,345 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:29,585 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:30,236 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1650095 == 0 2022-02-07 22:31:30,237 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1650095 == 0. Received: {'172.23.121.124': 1650095} for bucket 'GleamBookUsers0' 2022-02-07 22:31:30,237 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:30,240 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:30,240 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:30,598 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1134814 == 0 2022-02-07 22:31:30,599 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1134814 == 0. Received: {'172.23.121.123': 1134814} for bucket 'GleamBookUsers0' 2022-02-07 22:31:30,599 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:30,601 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:30,601 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:30,644 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 353463 == 0 2022-02-07 22:31:30,644 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 353463 == 0. Received: {'172.23.121.115': 353463} for bucket 'GleamBookUsers0' 2022-02-07 22:31:30,644 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:30,647 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:30,647 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:31,209 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1848474 == 0 2022-02-07 22:31:31,209 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1848474 == 0. Received: {'172.23.120.170': 1848474} for bucket 'GleamBookUsers0' 2022-02-07 22:31:31,209 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:31,210 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:31,210 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:31,385 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:31,930 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:32,693 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:32,884 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1268921 == 0 2022-02-07 22:31:32,884 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1268921 == 0. Received: {'172.23.121.126': 1268921} for bucket 'GleamBookUsers0' 2022-02-07 22:31:32,884 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:32,885 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:32,887 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:34,065 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:34,141 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1816727 == 0 2022-02-07 22:31:34,141 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1816727 == 0. Received: {'172.23.121.129': 1816727} for bucket 'GleamBookUsers0' 2022-02-07 22:31:34,141 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:34,144 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:34,144 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:34,262 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1191437 == 0 2022-02-07 22:31:34,263 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1191437 == 0. Received: {'172.23.121.127': 1191437} for bucket 'GleamBookUsers0' 2022-02-07 22:31:34,263 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:34,265 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:34,265 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:35,240 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:35,602 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:35,651 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:35,733 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1393780 == 0 2022-02-07 22:31:35,733 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1393780 == 0. Received: {'172.23.121.128': 1393780} for bucket 'GleamBookUsers0' 2022-02-07 22:31:35,734 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:35,736 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:35,736 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:36,211 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:36,688 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 330313 == 0 2022-02-07 22:31:36,690 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 330313 == 0. Received: {'172.23.121.115': 330313} for bucket 'GleamBookUsers0' 2022-02-07 22:31:36,690 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:36,691 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:36,693 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:36,940 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1121092 == 0 2022-02-07 22:31:36,941 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1121092 == 0. Received: {'172.23.121.123': 1121092} for bucket 'GleamBookUsers0' 2022-02-07 22:31:36,941 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:36,944 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:36,944 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:37,163 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1635011 == 0 2022-02-07 22:31:37,163 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1635011 == 0. Received: {'172.23.121.124': 1635011} for bucket 'GleamBookUsers0' 2022-02-07 22:31:37,163 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:37,171 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:37,171 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:37,887 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:38,161 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1767154 == 0 2022-02-07 22:31:38,163 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1767154 == 0. Received: {'172.23.120.170': 1767154} for bucket 'GleamBookUsers0' 2022-02-07 22:31:38,163 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:38,164 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:38,164 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:39,144 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:39,313 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:39,362 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1268921 == 0 2022-02-07 22:31:39,364 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1268921 == 0. Received: {'172.23.121.126': 1268921} for bucket 'GleamBookUsers0' 2022-02-07 22:31:39,364 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:39,365 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:39,365 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:40,737 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:40,834 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1177583 == 0 2022-02-07 22:31:40,834 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1177583 == 0. Received: {'172.23.121.127': 1177583} for bucket 'GleamBookUsers0' 2022-02-07 22:31:40,835 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:40,836 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:40,836 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:41,319 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1782337 == 0 2022-02-07 22:31:41,321 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1782337 == 0. Received: {'172.23.121.129': 1782337} for bucket 'GleamBookUsers0' 2022-02-07 22:31:41,321 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:41,322 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:41,323 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:41,693 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:41,944 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:42,171 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:42,433 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1364936 == 0 2022-02-07 22:31:42,434 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1364936 == 0. Received: {'172.23.121.128': 1364936} for bucket 'GleamBookUsers0' 2022-02-07 22:31:42,434 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:42,436 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:42,437 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:42,621 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 309733 == 0 2022-02-07 22:31:42,621 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 309733 == 0. Received: {'172.23.121.115': 309733} for bucket 'GleamBookUsers0' 2022-02-07 22:31:42,621 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:42,624 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:42,624 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:43,069 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1115958 == 0 2022-02-07 22:31:43,069 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1115958 == 0. Received: {'172.23.121.123': 1115958} for bucket 'GleamBookUsers0' 2022-02-07 22:31:43,069 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:43,071 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:43,071 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:43,164 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:44,028 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1628826 == 0 2022-02-07 22:31:44,028 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1628826 == 0. Received: {'172.23.121.124': 1628826} for bucket 'GleamBookUsers0' 2022-02-07 22:31:44,028 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:44,029 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:44,029 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:44,365 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:45,173 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1722092 == 0 2022-02-07 22:31:45,173 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1722092 == 0. Received: {'172.23.120.170': 1722092} for bucket 'GleamBookUsers0' 2022-02-07 22:31:45,173 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:45,174 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:45,174 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:45,776 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1268921 == 0 2022-02-07 22:31:45,776 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1268921 == 0. Received: {'172.23.121.126': 1268921} for bucket 'GleamBookUsers0' 2022-02-07 22:31:45,778 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:45,779 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:45,779 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:45,836 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:46,323 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:47,209 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1147417 == 0 2022-02-07 22:31:47,209 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1147417 == 0. Received: {'172.23.121.127': 1147417} for bucket 'GleamBookUsers0' 2022-02-07 22:31:47,209 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:47,210 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:47,211 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:47,437 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:47,625 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:48,071 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:48,487 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1726570 == 0 2022-02-07 22:31:48,489 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1726570 == 0. Received: {'172.23.121.129': 1726570} for bucket 'GleamBookUsers0' 2022-02-07 22:31:48,489 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:48,490 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:48,490 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:48,563 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 288789 == 0 2022-02-07 22:31:48,563 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 288789 == 0. Received: {'172.23.121.115': 288789} for bucket 'GleamBookUsers0' 2022-02-07 22:31:48,563 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:48,565 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:48,565 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:49,030 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:49,096 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1339581 == 0 2022-02-07 22:31:49,098 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1339581 == 0. Received: {'172.23.121.128': 1339581} for bucket 'GleamBookUsers0' 2022-02-07 22:31:49,098 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:49,099 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:49,099 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:49,250 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1115916 == 0 2022-02-07 22:31:49,250 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1115916 == 0. Received: {'172.23.121.123': 1115916} for bucket 'GleamBookUsers0' 2022-02-07 22:31:49,250 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:49,253 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:49,253 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:50,176 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:50,782 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:50,783 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1586656 == 0 2022-02-07 22:31:50,783 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1586656 == 0. Received: {'172.23.121.124': 1586656} for bucket 'GleamBookUsers0' 2022-02-07 22:31:50,785 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:50,786 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:50,786 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:52,177 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1689566 == 0 2022-02-07 22:31:52,178 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1689566 == 0. Received: {'172.23.120.170': 1689566} for bucket 'GleamBookUsers0' 2022-02-07 22:31:52,178 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:52,180 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:52,180 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:52,211 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:52,345 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1268921 == 0 2022-02-07 22:31:52,345 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1268921 == 0. Received: {'172.23.121.126': 1268921} for bucket 'GleamBookUsers0' 2022-02-07 22:31:52,345 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:52,348 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:52,348 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:53,492 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:53,572 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:53,601 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1112233 == 0 2022-02-07 22:31:53,601 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1112233 == 0. Received: {'172.23.121.127': 1112233} for bucket 'GleamBookUsers0' 2022-02-07 22:31:53,601 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:53,604 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:53,605 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:54,101 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:54,253 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:54,482 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 269357 == 0 2022-02-07 22:31:54,483 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 269357 == 0. Received: {'172.23.121.115': 269357} for bucket 'GleamBookUsers0' 2022-02-07 22:31:54,483 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:54,483 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:54,484 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:55,479 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1106444 == 0 2022-02-07 22:31:55,480 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1106444 == 0. Received: {'172.23.121.123': 1106444} for bucket 'GleamBookUsers0' 2022-02-07 22:31:55,480 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:55,482 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:55,483 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:55,788 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:55,801 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1702690 == 0 2022-02-07 22:31:55,802 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1702690 == 0. Received: {'172.23.121.129': 1702690} for bucket 'GleamBookUsers0' 2022-02-07 22:31:55,802 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:55,803 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:55,803 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:55,858 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1275060 == 0 2022-02-07 22:31:55,858 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1275060 == 0. Received: {'172.23.121.128': 1275060} for bucket 'GleamBookUsers0' 2022-02-07 22:31:55,858 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:55,861 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:55,861 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:57,183 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:57,348 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:57,549 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1576758 == 0 2022-02-07 22:31:57,551 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1576758 == 0. Received: {'172.23.121.124': 1576758} for bucket 'GleamBookUsers0' 2022-02-07 22:31:57,551 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:57,552 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:57,553 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:58,607 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:31:58,848 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1245479 == 0 2022-02-07 22:31:58,849 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1245479 == 0. Received: {'172.23.121.126': 1245479} for bucket 'GleamBookUsers0' 2022-02-07 22:31:58,849 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:58,851 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:58,851 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:59,191 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1665437 == 0 2022-02-07 22:31:59,191 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1665437 == 0. Received: {'172.23.120.170': 1665437} for bucket 'GleamBookUsers0' 2022-02-07 22:31:59,193 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:31:59,194 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:31:59,194 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:31:59,484 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:00,005 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1072860 == 0 2022-02-07 22:32:00,006 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1072860 == 0. Received: {'172.23.121.127': 1072860} for bucket 'GleamBookUsers0' 2022-02-07 22:32:00,006 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:00,007 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:00,007 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:00,331 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 241205 == 0 2022-02-07 22:32:00,331 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 241205 == 0. Received: {'172.23.121.115': 241205} for bucket 'GleamBookUsers0' 2022-02-07 22:32:00,331 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:00,332 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:00,332 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:00,483 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:00,809 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:00,861 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:01,592 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1097064 == 0 2022-02-07 22:32:01,592 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1097064 == 0. Received: {'172.23.121.123': 1097064} for bucket 'GleamBookUsers0' 2022-02-07 22:32:01,592 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:01,595 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:01,595 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:02,477 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1268718 == 0 2022-02-07 22:32:02,479 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1268718 == 0. Received: {'172.23.121.128': 1268718} for bucket 'GleamBookUsers0' 2022-02-07 22:32:02,479 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:02,480 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:02,480 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:02,553 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:03,023 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1690816 == 0 2022-02-07 22:32:03,023 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1690816 == 0. Received: {'172.23.121.129': 1690816} for bucket 'GleamBookUsers0' 2022-02-07 22:32:03,023 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:03,028 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:03,028 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:03,852 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:04,194 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:04,305 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1567176 == 0 2022-02-07 22:32:04,306 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1567176 == 0. Received: {'172.23.121.124': 1567176} for bucket 'GleamBookUsers0' 2022-02-07 22:32:04,306 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:04,309 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:04,309 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:05,009 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:05,334 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:05,341 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1245479 == 0 2022-02-07 22:32:05,342 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1245479 == 0. Received: {'172.23.121.126': 1245479} for bucket 'GleamBookUsers0' 2022-02-07 22:32:05,342 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:05,344 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:05,345 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:06,328 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 227453 == 0 2022-02-07 22:32:06,329 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 227453 == 0. Received: {'172.23.121.115': 227453} for bucket 'GleamBookUsers0' 2022-02-07 22:32:06,329 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:06,332 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:06,332 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:06,332 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1656875 == 0 2022-02-07 22:32:06,332 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1656875 == 0. Received: {'172.23.120.170': 1656875} for bucket 'GleamBookUsers0' 2022-02-07 22:32:06,334 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:06,335 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:06,335 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:06,555 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1072852 == 0 2022-02-07 22:32:06,555 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1072852 == 0. Received: {'172.23.121.127': 1072852} for bucket 'GleamBookUsers0' 2022-02-07 22:32:06,555 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:06,558 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:06,558 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:06,599 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:07,480 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:07,674 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1086072 == 0 2022-02-07 22:32:07,674 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1086072 == 0. Received: {'172.23.121.123': 1086072} for bucket 'GleamBookUsers0' 2022-02-07 22:32:07,674 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:07,677 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:07,677 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:08,029 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:09,035 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1265974 == 0 2022-02-07 22:32:09,036 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1265974 == 0. Received: {'172.23.121.128': 1265974} for bucket 'GleamBookUsers0' 2022-02-07 22:32:09,036 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:09,038 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:09,038 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:09,309 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:10,042 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1658360 == 0 2022-02-07 22:32:10,042 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1658360 == 0. Received: {'172.23.121.129': 1658360} for bucket 'GleamBookUsers0' 2022-02-07 22:32:10,043 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:10,045 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:10,046 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:10,345 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:11,096 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1544742 == 0 2022-02-07 22:32:11,098 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1544742 == 0. Received: {'172.23.121.124': 1544742} for bucket 'GleamBookUsers0' 2022-02-07 22:32:11,098 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:11,099 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:11,099 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:11,332 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:11,336 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:11,559 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:11,783 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1245479 == 0 2022-02-07 22:32:11,785 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1245479 == 0. Received: {'172.23.121.126': 1245479} for bucket 'GleamBookUsers0' 2022-02-07 22:32:11,785 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:11,786 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:11,786 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:12,128 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 216189 == 0 2022-02-07 22:32:12,128 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 216189 == 0. Received: {'172.23.121.115': 216189} for bucket 'GleamBookUsers0' 2022-02-07 22:32:12,128 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:12,131 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:12,131 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:12,677 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:13,069 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1069666 == 0 2022-02-07 22:32:13,069 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1069666 == 0. Received: {'172.23.121.127': 1069666} for bucket 'GleamBookUsers0' 2022-02-07 22:32:13,069 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:13,072 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:13,072 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:13,372 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1636865 == 0 2022-02-07 22:32:13,374 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1636865 == 0. Received: {'172.23.120.170': 1636865} for bucket 'GleamBookUsers0' 2022-02-07 22:32:13,374 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:13,375 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:13,375 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:13,776 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1079322 == 0 2022-02-07 22:32:13,778 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1079322 == 0. Received: {'172.23.121.123': 1079322} for bucket 'GleamBookUsers0' 2022-02-07 22:32:13,778 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:13,779 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:13,779 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:14,039 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:15,046 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:15,569 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1253374 == 0 2022-02-07 22:32:15,571 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1253374 == 0. Received: {'172.23.121.128': 1253374} for bucket 'GleamBookUsers0' 2022-02-07 22:32:15,571 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:15,572 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:15,572 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:16,101 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:16,788 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:17,055 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1646478 == 0 2022-02-07 22:32:17,056 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1646478 == 0. Received: {'172.23.121.129': 1646478} for bucket 'GleamBookUsers0' 2022-02-07 22:32:17,056 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:17,059 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:17,059 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:17,131 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:17,976 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1500542 == 0 2022-02-07 22:32:17,977 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1500542 == 0. Received: {'172.23.121.124': 1500542} for bucket 'GleamBookUsers0' 2022-02-07 22:32:17,977 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:17,980 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:17,980 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:18,068 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 201587 == 0 2022-02-07 22:32:18,069 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 201587 == 0. Received: {'172.23.121.115': 201587} for bucket 'GleamBookUsers0' 2022-02-07 22:32:18,069 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:18,072 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:18,072 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:18,072 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:18,306 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1224417 == 0 2022-02-07 22:32:18,306 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1224417 == 0. Received: {'172.23.121.126': 1224417} for bucket 'GleamBookUsers0' 2022-02-07 22:32:18,306 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:18,309 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:18,309 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:18,377 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:18,782 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:19,433 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1061074 == 0 2022-02-07 22:32:19,433 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1061074 == 0. Received: {'172.23.121.127': 1061074} for bucket 'GleamBookUsers0' 2022-02-07 22:32:19,433 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:19,434 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:19,434 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:19,871 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1049374 == 0 2022-02-07 22:32:19,871 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1049374 == 0. Received: {'172.23.121.123': 1049374} for bucket 'GleamBookUsers0' 2022-02-07 22:32:19,871 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:19,875 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:19,875 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:20,299 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1613527 == 0 2022-02-07 22:32:20,299 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1613527 == 0. Received: {'172.23.120.170': 1613527} for bucket 'GleamBookUsers0' 2022-02-07 22:32:20,299 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:20,301 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:20,301 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:20,573 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:22,059 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:22,111 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1236864 == 0 2022-02-07 22:32:22,111 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1236864 == 0. Received: {'172.23.121.128': 1236864} for bucket 'GleamBookUsers0' 2022-02-07 22:32:22,111 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:22,114 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:22,115 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:22,980 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:23,072 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:23,309 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:23,881 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 182238 == 0 2022-02-07 22:32:23,881 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 182238 == 0. Received: {'172.23.121.115': 182238} for bucket 'GleamBookUsers0' 2022-02-07 22:32:23,882 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:23,884 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:23,884 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:24,167 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1634752 == 0 2022-02-07 22:32:24,167 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1634752 == 0. Received: {'172.23.121.129': 1634752} for bucket 'GleamBookUsers0' 2022-02-07 22:32:24,167 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:24,170 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:24,170 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:24,434 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:24,816 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1434167 == 0 2022-02-07 22:32:24,818 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1434167 == 0. Received: {'172.23.121.124': 1434167} for bucket 'GleamBookUsers0' 2022-02-07 22:32:24,818 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:24,821 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:24,821 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:24,864 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1201847 == 0 2022-02-07 22:32:24,864 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1201847 == 0. Received: {'172.23.121.126': 1201847} for bucket 'GleamBookUsers0' 2022-02-07 22:32:24,864 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:24,867 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:24,867 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:24,875 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:25,302 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:25,805 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1053468 == 0 2022-02-07 22:32:25,806 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1053468 == 0. Received: {'172.23.121.127': 1053468} for bucket 'GleamBookUsers0' 2022-02-07 22:32:25,806 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:25,809 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:25,809 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:25,980 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1017860 == 0 2022-02-07 22:32:25,982 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1017860 == 0. Received: {'172.23.121.123': 1017860} for bucket 'GleamBookUsers0' 2022-02-07 22:32:25,982 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:25,983 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:25,983 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:27,115 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:27,239 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1608788 == 0 2022-02-07 22:32:27,239 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1608788 == 0. Received: {'172.23.120.170': 1608788} for bucket 'GleamBookUsers0' 2022-02-07 22:32:27,239 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:27,240 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:27,242 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:28,661 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1236864 == 0 2022-02-07 22:32:28,663 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1236864 == 0. Received: {'172.23.121.128': 1236864} for bucket 'GleamBookUsers0' 2022-02-07 22:32:28,663 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:28,664 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:28,664 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:28,884 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:29,171 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:29,641 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 168586 == 0 2022-02-07 22:32:29,642 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 168586 == 0. Received: {'172.23.121.115': 168586} for bucket 'GleamBookUsers0' 2022-02-07 22:32:29,642 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:29,645 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:29,645 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:29,821 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:29,867 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:30,809 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:30,983 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:31,467 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1594428 == 0 2022-02-07 22:32:31,467 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1594428 == 0. Received: {'172.23.121.129': 1594428} for bucket 'GleamBookUsers0' 2022-02-07 22:32:31,467 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:31,470 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:31,470 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:31,482 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1178259 == 0 2022-02-07 22:32:31,482 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1178259 == 0. Received: {'172.23.121.126': 1178259} for bucket 'GleamBookUsers0' 2022-02-07 22:32:31,482 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:31,483 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:31,484 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:31,676 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1396696 == 0 2022-02-07 22:32:31,676 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1396696 == 0. Received: {'172.23.121.124': 1396696} for bucket 'GleamBookUsers0' 2022-02-07 22:32:31,676 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:31,677 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:31,677 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:32,108 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 1014544 == 0 2022-02-07 22:32:32,108 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1014544 == 0. Received: {'172.23.121.123': 1014544} for bucket 'GleamBookUsers0' 2022-02-07 22:32:32,108 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:32,109 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:32,111 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:32,240 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1049230 == 0 2022-02-07 22:32:32,240 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1049230 == 0. Received: {'172.23.121.127': 1049230} for bucket 'GleamBookUsers0' 2022-02-07 22:32:32,242 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:32,242 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:32,243 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:32,243 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:33,665 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:34,158 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1602846 == 0 2022-02-07 22:32:34,160 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1602846 == 0. Received: {'172.23.120.170': 1602846} for bucket 'GleamBookUsers0' 2022-02-07 22:32:34,160 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:34,161 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:34,161 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:34,654 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:35,167 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1226486 == 0 2022-02-07 22:32:35,167 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1226486 == 0. Received: {'172.23.121.128': 1226486} for bucket 'GleamBookUsers0' 2022-02-07 22:32:35,167 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:35,170 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:35,170 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:35,375 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 152146 == 0 2022-02-07 22:32:35,375 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 152146 == 0. Received: {'172.23.121.115': 152146} for bucket 'GleamBookUsers0' 2022-02-07 22:32:35,375 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:35,378 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:35,378 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:36,471 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:36,484 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:36,678 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:37,111 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:37,243 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:37,928 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1178259 == 0 2022-02-07 22:32:37,928 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1178259 == 0. Received: {'172.23.121.126': 1178259} for bucket 'GleamBookUsers0' 2022-02-07 22:32:37,930 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:37,931 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:37,931 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:38,513 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 988702 == 0 2022-02-07 22:32:38,513 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 988702 == 0. Received: {'172.23.121.123': 988702} for bucket 'GleamBookUsers0' 2022-02-07 22:32:38,513 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:38,515 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:38,516 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:38,828 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1394572 == 0 2022-02-07 22:32:38,829 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1394572 == 0. Received: {'172.23.121.124': 1394572} for bucket 'GleamBookUsers0' 2022-02-07 22:32:38,829 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:38,832 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:38,832 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:38,921 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1042193 == 0 2022-02-07 22:32:38,921 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1042193 == 0. Received: {'172.23.121.127': 1042193} for bucket 'GleamBookUsers0' 2022-02-07 22:32:38,921 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:38,924 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:38,924 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:38,951 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1568781 == 0 2022-02-07 22:32:38,953 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1568781 == 0. Received: {'172.23.121.129': 1568781} for bucket 'GleamBookUsers0' 2022-02-07 22:32:38,953 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:38,954 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:38,954 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:39,161 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:40,170 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:40,378 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:41,109 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1571136 == 0 2022-02-07 22:32:41,109 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1571136 == 0. Received: {'172.23.120.170': 1571136} for bucket 'GleamBookUsers0' 2022-02-07 22:32:41,109 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:41,112 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:41,112 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:41,154 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 140963 == 0 2022-02-07 22:32:41,154 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 140963 == 0. Received: {'172.23.121.115': 140963} for bucket 'GleamBookUsers0' 2022-02-07 22:32:41,154 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:41,157 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:41,157 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:41,621 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1187422 == 0 2022-02-07 22:32:41,621 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1187422 == 0. Received: {'172.23.121.128': 1187422} for bucket 'GleamBookUsers0' 2022-02-07 22:32:41,621 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:41,624 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:41,624 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:42,933 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:43,516 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:43,834 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:43,926 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:43,956 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:44,305 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1178259 == 0 2022-02-07 22:32:44,305 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1178259 == 0. Received: {'172.23.121.126': 1178259} for bucket 'GleamBookUsers0' 2022-02-07 22:32:44,305 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:44,308 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:44,308 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:44,561 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 984650 == 0 2022-02-07 22:32:44,561 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 984650 == 0. Received: {'172.23.121.123': 984650} for bucket 'GleamBookUsers0' 2022-02-07 22:32:44,561 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:44,563 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:44,563 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:45,305 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 1011467 == 0 2022-02-07 22:32:45,305 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1011467 == 0. Received: {'172.23.121.127': 1011467} for bucket 'GleamBookUsers0' 2022-02-07 22:32:45,305 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:45,309 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:45,309 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:45,585 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1364903 == 0 2022-02-07 22:32:45,585 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1364903 == 0. Received: {'172.23.121.124': 1364903} for bucket 'GleamBookUsers0' 2022-02-07 22:32:45,585 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:45,588 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:45,588 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:46,025 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1510651 == 0 2022-02-07 22:32:46,026 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1510651 == 0. Received: {'172.23.121.129': 1510651} for bucket 'GleamBookUsers0' 2022-02-07 22:32:46,026 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:46,028 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:46,028 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:46,112 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:46,157 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:46,625 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:46,828 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 122737 == 0 2022-02-07 22:32:46,828 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 122737 == 0. Received: {'172.23.121.115': 122737} for bucket 'GleamBookUsers0' 2022-02-07 22:32:46,828 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:46,829 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:46,831 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:48,010 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1503400 == 0 2022-02-07 22:32:48,010 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1503400 == 0. Received: {'172.23.120.170': 1503400} for bucket 'GleamBookUsers0' 2022-02-07 22:32:48,010 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:48,013 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:48,013 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:48,160 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1165056 == 0 2022-02-07 22:32:48,161 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1165056 == 0. Received: {'172.23.121.128': 1165056} for bucket 'GleamBookUsers0' 2022-02-07 22:32:48,161 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:48,163 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:48,163 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:49,308 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:49,565 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:50,309 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:50,588 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:50,655 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 977390 == 0 2022-02-07 22:32:50,657 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 977390 == 0. Received: {'172.23.121.123': 977390} for bucket 'GleamBookUsers0' 2022-02-07 22:32:50,657 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:50,658 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:50,658 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:50,834 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1157575 == 0 2022-02-07 22:32:50,834 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1157575 == 0. Received: {'172.23.121.126': 1157575} for bucket 'GleamBookUsers0' 2022-02-07 22:32:50,834 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:50,835 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:50,835 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:51,029 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:51,578 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 958255 == 0 2022-02-07 22:32:51,579 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 958255 == 0. Received: {'172.23.121.127': 958255} for bucket 'GleamBookUsers0' 2022-02-07 22:32:51,579 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:51,582 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:51,582 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:51,831 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:52,226 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1349855 == 0 2022-02-07 22:32:52,226 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1349855 == 0. Received: {'172.23.121.124': 1349855} for bucket 'GleamBookUsers0' 2022-02-07 22:32:52,226 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:52,229 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:52,230 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:52,469 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 108750 == 0 2022-02-07 22:32:52,470 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 108750 == 0. Received: {'172.23.121.115': 108750} for bucket 'GleamBookUsers0' 2022-02-07 22:32:52,470 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:52,471 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:52,471 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:52,982 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1498061 == 0 2022-02-07 22:32:52,982 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1498061 == 0. Received: {'172.23.121.129': 1498061} for bucket 'GleamBookUsers0' 2022-02-07 22:32:52,982 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:52,984 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:52,984 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:53,013 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:53,163 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:54,703 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1133358 == 0 2022-02-07 22:32:54,704 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1133358 == 0. Received: {'172.23.121.128': 1133358} for bucket 'GleamBookUsers0' 2022-02-07 22:32:54,704 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:54,707 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:54,707 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:55,095 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1487613 == 0 2022-02-07 22:32:55,095 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1487613 == 0. Received: {'172.23.120.170': 1487613} for bucket 'GleamBookUsers0' 2022-02-07 22:32:55,095 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:55,096 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:55,096 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:55,661 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:55,836 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:56,582 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:56,627 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 926102 == 0 2022-02-07 22:32:56,627 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 926102 == 0. Received: {'172.23.121.123': 926102} for bucket 'GleamBookUsers0' 2022-02-07 22:32:56,628 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:56,630 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:56,630 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:57,230 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:57,249 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1113683 == 0 2022-02-07 22:32:57,250 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1113683 == 0. Received: {'172.23.121.126': 1113683} for bucket 'GleamBookUsers0' 2022-02-07 22:32:57,250 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:57,252 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:57,252 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:57,471 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:57,858 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 954973 == 0 2022-02-07 22:32:57,858 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 954973 == 0. Received: {'172.23.121.127': 954973} for bucket 'GleamBookUsers0' 2022-02-07 22:32:57,858 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:57,861 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:57,861 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:57,986 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:58,059 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 92597 == 0 2022-02-07 22:32:58,059 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 92597 == 0. Received: {'172.23.121.115': 92597} for bucket 'GleamBookUsers0' 2022-02-07 22:32:58,059 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:58,061 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:58,061 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:58,838 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1345431 == 0 2022-02-07 22:32:58,838 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1345431 == 0. Received: {'172.23.121.124': 1345431} for bucket 'GleamBookUsers0' 2022-02-07 22:32:58,838 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:58,841 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:58,841 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:32:59,707 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:32:59,915 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1482767 == 0 2022-02-07 22:32:59,915 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1482767 == 0. Received: {'172.23.121.129': 1482767} for bucket 'GleamBookUsers0' 2022-02-07 22:32:59,917 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:32:59,918 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:32:59,918 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:00,098 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:01,130 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1133358 == 0 2022-02-07 22:33:01,131 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1133358 == 0. Received: {'172.23.121.128': 1133358} for bucket 'GleamBookUsers0' 2022-02-07 22:33:01,131 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:01,132 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:01,132 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:01,631 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:01,904 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1453155 == 0 2022-02-07 22:33:01,904 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1453155 == 0. Received: {'172.23.120.170': 1453155} for bucket 'GleamBookUsers0' 2022-02-07 22:33:01,904 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:01,907 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:01,907 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:02,252 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:02,638 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 923624 == 0 2022-02-07 22:33:02,638 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 923624 == 0. Received: {'172.23.121.123': 923624} for bucket 'GleamBookUsers0' 2022-02-07 22:33:02,638 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:02,641 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:02,641 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:02,861 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:03,062 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:03,670 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1113683 == 0 2022-02-07 22:33:03,670 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1113683 == 0. Received: {'172.23.121.126': 1113683} for bucket 'GleamBookUsers0' 2022-02-07 22:33:03,671 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:03,673 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:03,673 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:03,757 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 81119 == 0 2022-02-07 22:33:03,757 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 81119 == 0. Received: {'172.23.121.115': 81119} for bucket 'GleamBookUsers0' 2022-02-07 22:33:03,759 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:03,762 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:03,762 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:03,841 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:04,102 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 949333 == 0 2022-02-07 22:33:04,104 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 949333 == 0. Received: {'172.23.121.127': 949333} for bucket 'GleamBookUsers0' 2022-02-07 22:33:04,104 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:04,105 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:04,105 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:04,918 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:05,447 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1319486 == 0 2022-02-07 22:33:05,447 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1319486 == 0. Received: {'172.23.121.124': 1319486} for bucket 'GleamBookUsers0' 2022-02-07 22:33:05,447 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:05,450 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:05,450 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:06,132 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:06,864 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1448385 == 0 2022-02-07 22:33:06,864 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1448385 == 0. Received: {'172.23.121.129': 1448385} for bucket 'GleamBookUsers0' 2022-02-07 22:33:06,864 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:06,865 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:06,865 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:06,907 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:07,533 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1109070 == 0 2022-02-07 22:33:07,535 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1109070 == 0. Received: {'172.23.121.128': 1109070} for bucket 'GleamBookUsers0' 2022-02-07 22:33:07,535 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:07,536 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:07,536 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:07,641 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:08,674 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:08,687 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 922016 == 0 2022-02-07 22:33:08,688 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 922016 == 0. Received: {'172.23.121.123': 922016} for bucket 'GleamBookUsers0' 2022-02-07 22:33:08,688 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:08,690 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:08,690 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:08,763 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:08,815 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1453155 == 0 2022-02-07 22:33:08,815 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1453155 == 0. Received: {'172.23.120.170': 1453155} for bucket 'GleamBookUsers0' 2022-02-07 22:33:08,815 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:08,819 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:08,819 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:09,108 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:09,267 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 62245 == 0 2022-02-07 22:33:09,269 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 62245 == 0. Received: {'172.23.121.115': 62245} for bucket 'GleamBookUsers0' 2022-02-07 22:33:09,269 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:09,270 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:09,270 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:10,046 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1092117 == 0 2022-02-07 22:33:10,046 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1092117 == 0. Received: {'172.23.121.126': 1092117} for bucket 'GleamBookUsers0' 2022-02-07 22:33:10,046 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:10,049 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:10,049 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:10,403 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 919091 == 0 2022-02-07 22:33:10,403 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 919091 == 0. Received: {'172.23.121.127': 919091} for bucket 'GleamBookUsers0' 2022-02-07 22:33:10,404 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:10,405 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:10,405 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:10,450 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:11,867 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:12,035 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1309574 == 0 2022-02-07 22:33:12,036 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1309574 == 0. Received: {'172.23.121.124': 1309574} for bucket 'GleamBookUsers0' 2022-02-07 22:33:12,036 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:12,038 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:12,038 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:12,539 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:13,701 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:13,819 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:13,848 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1439035 == 0 2022-02-07 22:33:13,849 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1439035 == 0. Received: {'172.23.121.129': 1439035} for bucket 'GleamBookUsers0' 2022-02-07 22:33:13,851 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:13,855 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:13,857 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:14,040 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1109070 == 0 2022-02-07 22:33:14,040 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1109070 == 0. Received: {'172.23.121.128': 1109070} for bucket 'GleamBookUsers0' 2022-02-07 22:33:14,040 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:14,042 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:14,042 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:14,270 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:14,683 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 922016 == 0 2022-02-07 22:33:14,683 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 922016 == 0. Received: {'172.23.121.123': 922016} for bucket 'GleamBookUsers0' 2022-02-07 22:33:14,683 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:14,686 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:14,686 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:14,779 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 50427 == 0 2022-02-07 22:33:14,779 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 50427 == 0. Received: {'172.23.121.115': 50427} for bucket 'GleamBookUsers0' 2022-02-07 22:33:14,779 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:14,782 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:14,782 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:15,049 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:15,405 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:15,605 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1426877 == 0 2022-02-07 22:33:15,605 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1426877 == 0. Received: {'172.23.120.170': 1426877} for bucket 'GleamBookUsers0' 2022-02-07 22:33:15,605 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:15,607 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:15,607 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:16,515 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1092117 == 0 2022-02-07 22:33:16,516 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1092117 == 0. Received: {'172.23.121.126': 1092117} for bucket 'GleamBookUsers0' 2022-02-07 22:33:16,516 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:16,519 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:16,519 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:16,681 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 916485 == 0 2022-02-07 22:33:16,683 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 916485 == 0. Received: {'172.23.121.127': 916485} for bucket 'GleamBookUsers0' 2022-02-07 22:33:16,683 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:16,684 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:16,684 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:17,039 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:18,563 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1307406 == 0 2022-02-07 22:33:18,563 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1307406 == 0. Received: {'172.23.121.124': 1307406} for bucket 'GleamBookUsers0' 2022-02-07 22:33:18,563 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:18,565 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:18,565 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:18,857 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:19,043 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:19,687 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:19,782 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:20,323 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 45940 == 0 2022-02-07 22:33:20,325 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 45940 == 0. Received: {'172.23.121.115': 45940} for bucket 'GleamBookUsers0' 2022-02-07 22:33:20,325 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:20,326 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:20,326 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:20,608 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:20,625 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1100284 == 0 2022-02-07 22:33:20,625 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1100284 == 0. Received: {'172.23.121.128': 1100284} for bucket 'GleamBookUsers0' 2022-02-07 22:33:20,625 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:20,628 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:20,628 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:20,891 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 897998 == 0 2022-02-07 22:33:20,891 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 897998 == 0. Received: {'172.23.121.123': 897998} for bucket 'GleamBookUsers0' 2022-02-07 22:33:20,892 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:20,894 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:20,894 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:21,015 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1432028 == 0 2022-02-07 22:33:21,016 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1432028 == 0. Received: {'172.23.121.129': 1432028} for bucket 'GleamBookUsers0' 2022-02-07 22:33:21,016 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:21,017 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:21,017 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:21,519 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:21,686 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:22,395 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1406127 == 0 2022-02-07 22:33:22,395 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1406127 == 0. Received: {'172.23.120.170': 1406127} for bucket 'GleamBookUsers0' 2022-02-07 22:33:22,397 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:22,398 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:22,398 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:22,970 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 893471 == 0 2022-02-07 22:33:22,970 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 893471 == 0. Received: {'172.23.121.127': 893471} for bucket 'GleamBookUsers0' 2022-02-07 22:33:22,970 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:22,973 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:22,973 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:23,039 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1092117 == 0 2022-02-07 22:33:23,039 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1092117 == 0. Received: {'172.23.121.126': 1092117} for bucket 'GleamBookUsers0' 2022-02-07 22:33:23,039 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:23,042 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:23,042 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:23,566 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:25,095 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1287624 == 0 2022-02-07 22:33:25,095 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1287624 == 0. Received: {'172.23.121.124': 1287624} for bucket 'GleamBookUsers0' 2022-02-07 22:33:25,095 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:25,098 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:25,098 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:25,328 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:25,630 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:25,773 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 41726 == 0 2022-02-07 22:33:25,773 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 41726 == 0. Received: {'172.23.121.115': 41726} for bucket 'GleamBookUsers0' 2022-02-07 22:33:25,773 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:25,775 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:25,775 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:25,894 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:26,019 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:26,934 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 873905 == 0 2022-02-07 22:33:26,934 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 873905 == 0. Received: {'172.23.121.123': 873905} for bucket 'GleamBookUsers0' 2022-02-07 22:33:26,934 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:26,937 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:26,937 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:27,062 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1100284 == 0 2022-02-07 22:33:27,062 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1100284 == 0. Received: {'172.23.121.128': 1100284} for bucket 'GleamBookUsers0' 2022-02-07 22:33:27,062 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:27,063 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:27,065 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:27,400 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:27,822 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1398516 == 0 2022-02-07 22:33:27,823 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1398516 == 0. Received: {'172.23.121.129': 1398516} for bucket 'GleamBookUsers0' 2022-02-07 22:33:27,823 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:27,825 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:27,825 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:27,973 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:28,042 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:29,253 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 870313 == 0 2022-02-07 22:33:29,253 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 870313 == 0. Received: {'172.23.121.127': 870313} for bucket 'GleamBookUsers0' 2022-02-07 22:33:29,253 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:29,256 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:29,256 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:29,257 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1392061 == 0 2022-02-07 22:33:29,257 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1392061 == 0. Received: {'172.23.120.170': 1392061} for bucket 'GleamBookUsers0' 2022-02-07 22:33:29,257 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:29,260 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:29,260 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:29,486 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1072610 == 0 2022-02-07 22:33:29,486 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1072610 == 0. Received: {'172.23.121.126': 1072610} for bucket 'GleamBookUsers0' 2022-02-07 22:33:29,486 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:29,489 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:29,489 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:30,098 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:30,776 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:31,213 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 34330 == 0 2022-02-07 22:33:31,213 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 34330 == 0. Received: {'172.23.121.115': 34330} for bucket 'GleamBookUsers0' 2022-02-07 22:33:31,213 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:31,216 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:31,217 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:31,519 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1273350 == 0 2022-02-07 22:33:31,519 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1273350 == 0. Received: {'172.23.121.124': 1273350} for bucket 'GleamBookUsers0' 2022-02-07 22:33:31,519 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:31,522 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:31,522 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:31,938 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:32,065 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:32,826 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:32,878 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 850389 == 0 2022-02-07 22:33:32,878 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 850389 == 0. Received: {'172.23.121.123': 850389} for bucket 'GleamBookUsers0' 2022-02-07 22:33:32,878 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:32,881 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:32,881 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:33,454 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1063438 == 0 2022-02-07 22:33:33,456 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1063438 == 0. Received: {'172.23.121.128': 1063438} for bucket 'GleamBookUsers0' 2022-02-07 22:33:33,456 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:33,457 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:33,457 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:34,256 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:34,260 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:34,490 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:34,588 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1367073 == 0 2022-02-07 22:33:34,588 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1367073 == 0. Received: {'172.23.121.129': 1367073} for bucket 'GleamBookUsers0' 2022-02-07 22:33:34,588 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:34,591 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:34,591 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:35,414 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 870313 == 0 2022-02-07 22:33:35,415 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 870313 == 0. Received: {'172.23.121.127': 870313} for bucket 'GleamBookUsers0' 2022-02-07 22:33:35,415 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:35,417 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:35,417 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:35,990 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1033249 == 0 2022-02-07 22:33:35,990 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1033249 == 0. Received: {'172.23.121.126': 1033249} for bucket 'GleamBookUsers0' 2022-02-07 22:33:35,990 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:35,993 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:35,993 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:36,111 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1381953 == 0 2022-02-07 22:33:36,112 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1381953 == 0. Received: {'172.23.120.170': 1381953} for bucket 'GleamBookUsers0' 2022-02-07 22:33:36,112 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:36,115 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:36,115 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:36,217 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:36,522 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:36,658 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 27300 == 0 2022-02-07 22:33:36,658 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 27300 == 0. Received: {'172.23.121.115': 27300} for bucket 'GleamBookUsers0' 2022-02-07 22:33:36,658 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:36,661 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:36,661 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:37,868 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1208808 == 0 2022-02-07 22:33:37,868 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1208808 == 0. Received: {'172.23.121.124': 1208808} for bucket 'GleamBookUsers0' 2022-02-07 22:33:37,869 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:37,871 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:37,871 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:37,881 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:38,457 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:38,832 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 850389 == 0 2022-02-07 22:33:38,834 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 850389 == 0. Received: {'172.23.121.123': 850389} for bucket 'GleamBookUsers0' 2022-02-07 22:33:38,834 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:38,835 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:38,835 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:39,594 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:39,835 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 1012328 == 0 2022-02-07 22:33:39,835 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1012328 == 0. Received: {'172.23.121.128': 1012328} for bucket 'GleamBookUsers0' 2022-02-07 22:33:39,835 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:39,836 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:39,836 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:40,417 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:40,993 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:41,115 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:41,357 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1315998 == 0 2022-02-07 22:33:41,357 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1315998 == 0. Received: {'172.23.121.129': 1315998} for bucket 'GleamBookUsers0' 2022-02-07 22:33:41,357 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:41,359 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:41,359 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:41,584 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 847396 == 0 2022-02-07 22:33:41,585 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 847396 == 0. Received: {'172.23.121.127': 847396} for bucket 'GleamBookUsers0' 2022-02-07 22:33:41,585 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:41,586 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:41,588 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:41,663 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:42,095 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 20016 == 0 2022-02-07 22:33:42,095 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 20016 == 0. Received: {'172.23.121.115': 20016} for bucket 'GleamBookUsers0' 2022-02-07 22:33:42,095 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:42,098 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:42,098 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:42,494 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1012443 == 0 2022-02-07 22:33:42,494 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1012443 == 0. Received: {'172.23.121.126': 1012443} for bucket 'GleamBookUsers0' 2022-02-07 22:33:42,494 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:42,497 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:42,497 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:42,871 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:42,990 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1381953 == 0 2022-02-07 22:33:42,990 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1381953 == 0. Received: {'172.23.120.170': 1381953} for bucket 'GleamBookUsers0' 2022-02-07 22:33:42,990 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:42,993 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:42,993 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:43,835 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:44,230 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1170090 == 0 2022-02-07 22:33:44,230 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1170090 == 0. Received: {'172.23.121.124': 1170090} for bucket 'GleamBookUsers0' 2022-02-07 22:33:44,230 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:44,233 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:44,233 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:44,835 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 828121 == 0 2022-02-07 22:33:44,835 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 828121 == 0. Received: {'172.23.121.123': 828121} for bucket 'GleamBookUsers0' 2022-02-07 22:33:44,835 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:44,836 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:44,836 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:44,838 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:46,224 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 990542 == 0 2022-02-07 22:33:46,224 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 990542 == 0. Received: {'172.23.121.128': 990542} for bucket 'GleamBookUsers0' 2022-02-07 22:33:46,224 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:46,227 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:46,227 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:46,361 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:46,588 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:47,098 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:47,497 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:47,532 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 15172 == 0 2022-02-07 22:33:47,532 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 15172 == 0. Received: {'172.23.121.115': 15172} for bucket 'GleamBookUsers0' 2022-02-07 22:33:47,532 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:47,535 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:47,536 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:47,759 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 847396 == 0 2022-02-07 22:33:47,759 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 847396 == 0. Received: {'172.23.121.127': 847396} for bucket 'GleamBookUsers0' 2022-02-07 22:33:47,759 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:47,763 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:47,763 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:47,993 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:48,082 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1307390 == 0 2022-02-07 22:33:48,082 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1307390 == 0. Received: {'172.23.121.129': 1307390} for bucket 'GleamBookUsers0' 2022-02-07 22:33:48,082 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:48,085 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:48,085 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:48,849 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 1012443 == 0 2022-02-07 22:33:48,851 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1012443 == 0. Received: {'172.23.121.126': 1012443} for bucket 'GleamBookUsers0' 2022-02-07 22:33:48,851 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:48,854 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:48,854 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:49,233 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:49,733 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1338469 == 0 2022-02-07 22:33:49,733 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1338469 == 0. Received: {'172.23.120.170': 1338469} for bucket 'GleamBookUsers0' 2022-02-07 22:33:49,733 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:49,736 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:49,736 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:49,838 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:50,562 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1158346 == 0 2022-02-07 22:33:50,562 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1158346 == 0. Received: {'172.23.121.124': 1158346} for bucket 'GleamBookUsers0' 2022-02-07 22:33:50,562 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:50,565 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:50,565 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:50,812 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 806651 == 0 2022-02-07 22:33:50,812 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 806651 == 0. Received: {'172.23.121.123': 806651} for bucket 'GleamBookUsers0' 2022-02-07 22:33:50,812 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:50,813 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:50,813 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:51,227 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:52,536 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:52,568 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 983600 == 0 2022-02-07 22:33:52,569 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 983600 == 0. Received: {'172.23.121.128': 983600} for bucket 'GleamBookUsers0' 2022-02-07 22:33:52,569 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:52,571 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:52,571 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:52,763 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:52,924 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 10576 == 0 2022-02-07 22:33:52,924 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 10576 == 0. Received: {'172.23.121.115': 10576} for bucket 'GleamBookUsers0' 2022-02-07 22:33:52,924 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:52,926 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:52,926 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:53,085 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:53,746 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 801421 == 0 2022-02-07 22:33:53,747 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 801421 == 0. Received: {'172.23.121.127': 801421} for bucket 'GleamBookUsers0' 2022-02-07 22:33:53,747 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:53,750 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:53,750 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:53,855 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:54,736 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:54,805 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1287212 == 0 2022-02-07 22:33:54,805 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1287212 == 0. Received: {'172.23.121.129': 1287212} for bucket 'GleamBookUsers0' 2022-02-07 22:33:54,805 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:54,808 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:54,808 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:55,186 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 974079 == 0 2022-02-07 22:33:55,186 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 974079 == 0. Received: {'172.23.121.126': 974079} for bucket 'GleamBookUsers0' 2022-02-07 22:33:55,186 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:55,187 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:55,187 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:55,565 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:55,815 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:56,404 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1296561 == 0 2022-02-07 22:33:56,404 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1296561 == 0. Received: {'172.23.120.170': 1296561} for bucket 'GleamBookUsers0' 2022-02-07 22:33:56,404 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:56,407 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:56,407 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:56,806 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 806651 == 0 2022-02-07 22:33:56,806 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 806651 == 0. Received: {'172.23.121.123': 806651} for bucket 'GleamBookUsers0' 2022-02-07 22:33:56,806 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:56,809 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:56,809 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:56,999 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1143040 == 0 2022-02-07 22:33:57,000 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1143040 == 0. Received: {'172.23.121.124': 1143040} for bucket 'GleamBookUsers0' 2022-02-07 22:33:57,000 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:57,002 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:57,002 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:57,572 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:57,927 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:58,319 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 5860 == 0 2022-02-07 22:33:58,319 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 5860 == 0. Received: {'172.23.121.115': 5860} for bucket 'GleamBookUsers0' 2022-02-07 22:33:58,319 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:58,321 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:58,321 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:58,750 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:33:58,907 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 977130 == 0 2022-02-07 22:33:58,907 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 977130 == 0. Received: {'172.23.121.128': 977130} for bucket 'GleamBookUsers0' 2022-02-07 22:33:58,907 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:58,908 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:58,908 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:59,769 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 801421 == 0 2022-02-07 22:33:59,770 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 801421 == 0. Received: {'172.23.121.127': 801421} for bucket 'GleamBookUsers0' 2022-02-07 22:33:59,770 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:33:59,773 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:33:59,773 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:33:59,809 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:00,188 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:01,407 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:01,571 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1278640 == 0 2022-02-07 22:34:01,571 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1278640 == 0. Received: {'172.23.121.129': 1278640} for bucket 'GleamBookUsers0' 2022-02-07 22:34:01,571 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:01,572 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:01,573 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:01,585 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 974079 == 0 2022-02-07 22:34:01,585 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 974079 == 0. Received: {'172.23.121.126': 974079} for bucket 'GleamBookUsers0' 2022-02-07 22:34:01,585 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:01,586 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:01,586 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:01,809 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:02,003 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:02,773 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 784643 == 0 2022-02-07 22:34:02,775 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 784643 == 0. Received: {'172.23.121.123': 784643} for bucket 'GleamBookUsers0' 2022-02-07 22:34:02,775 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:02,776 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:02,776 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:03,249 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1284053 == 0 2022-02-07 22:34:03,249 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1284053 == 0. Received: {'172.23.120.170': 1284053} for bucket 'GleamBookUsers0' 2022-02-07 22:34:03,249 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:03,250 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:03,250 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:03,322 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:03,430 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1132412 == 0 2022-02-07 22:34:03,430 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1132412 == 0. Received: {'172.23.121.124': 1132412} for bucket 'GleamBookUsers0' 2022-02-07 22:34:03,431 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:03,434 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:03,434 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:03,700 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 4590 == 0 2022-02-07 22:34:03,700 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 4590 == 0. Received: {'172.23.121.115': 4590} for bucket 'GleamBookUsers0' 2022-02-07 22:34:03,700 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:03,701 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:03,703 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:03,910 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:04,775 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:05,161 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 895230 == 0 2022-02-07 22:34:05,161 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 895230 == 0. Received: {'172.23.121.128': 895230} for bucket 'GleamBookUsers0' 2022-02-07 22:34:05,161 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:05,163 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:05,163 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:05,767 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 779313 == 0 2022-02-07 22:34:05,767 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 779313 == 0. Received: {'172.23.121.127': 779313} for bucket 'GleamBookUsers0' 2022-02-07 22:34:05,767 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:05,769 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:05,769 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:06,573 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:06,588 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:07,778 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:07,861 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 914815 == 0 2022-02-07 22:34:07,861 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 914815 == 0. Received: {'172.23.121.126': 914815} for bucket 'GleamBookUsers0' 2022-02-07 22:34:07,861 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:07,864 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:07,864 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:08,240 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1275012 == 0 2022-02-07 22:34:08,240 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1275012 == 0. Received: {'172.23.121.129': 1275012} for bucket 'GleamBookUsers0' 2022-02-07 22:34:08,240 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:08,243 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:08,243 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:08,252 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:08,434 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:08,696 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 784643 == 0 2022-02-07 22:34:08,696 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 784643 == 0. Received: {'172.23.121.123': 784643} for bucket 'GleamBookUsers0' 2022-02-07 22:34:08,696 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:08,697 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:08,698 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:08,703 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:09,071 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 4590 == 0 2022-02-07 22:34:09,071 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 4590 == 0. Received: {'172.23.121.115': 4590} for bucket 'GleamBookUsers0' 2022-02-07 22:34:09,071 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:09,072 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:09,072 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:09,772 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1119140 == 0 2022-02-07 22:34:09,772 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1119140 == 0. Received: {'172.23.121.124': 1119140} for bucket 'GleamBookUsers0' 2022-02-07 22:34:09,772 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:09,775 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:09,776 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:09,957 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1251349 == 0 2022-02-07 22:34:09,957 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1251349 == 0. Received: {'172.23.120.170': 1251349} for bucket 'GleamBookUsers0' 2022-02-07 22:34:09,957 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:09,959 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:09,960 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:10,164 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:10,770 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:11,450 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 890404 == 0 2022-02-07 22:34:11,450 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 890404 == 0. Received: {'172.23.121.128': 890404} for bucket 'GleamBookUsers0' 2022-02-07 22:34:11,450 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:11,453 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:11,453 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:11,782 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 779313 == 0 2022-02-07 22:34:11,782 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 779313 == 0. Received: {'172.23.121.127': 779313} for bucket 'GleamBookUsers0' 2022-02-07 22:34:11,782 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:11,786 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:11,786 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:12,865 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:13,243 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:13,698 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:14,073 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:14,078 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 895393 == 0 2022-02-07 22:34:14,078 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 895393 == 0. Received: {'172.23.121.126': 895393} for bucket 'GleamBookUsers0' 2022-02-07 22:34:14,078 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:14,081 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:14,081 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:14,505 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 2240 == 0 2022-02-07 22:34:14,505 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2240 == 0. Received: {'172.23.121.115': 2240} for bucket 'GleamBookUsers0' 2022-02-07 22:34:14,505 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:14,507 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:14,507 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:14,776 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:14,812 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 763175 == 0 2022-02-07 22:34:14,812 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 763175 == 0. Received: {'172.23.121.123': 763175} for bucket 'GleamBookUsers0' 2022-02-07 22:34:14,812 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:14,815 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:14,815 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:14,956 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1233658 == 0 2022-02-07 22:34:14,957 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1233658 == 0. Received: {'172.23.121.129': 1233658} for bucket 'GleamBookUsers0' 2022-02-07 22:34:14,957 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:14,959 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:14,959 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:14,960 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:16,042 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1114868 == 0 2022-02-07 22:34:16,043 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1114868 == 0. Received: {'172.23.121.124': 1114868} for bucket 'GleamBookUsers0' 2022-02-07 22:34:16,043 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:16,046 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:16,046 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:16,453 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:16,584 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1251349 == 0 2022-02-07 22:34:16,585 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1251349 == 0. Received: {'172.23.120.170': 1251349} for bucket 'GleamBookUsers0' 2022-02-07 22:34:16,585 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:16,586 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:16,586 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:16,786 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:17,838 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 885706 == 0 2022-02-07 22:34:17,838 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 885706 == 0. Received: {'172.23.121.128': 885706} for bucket 'GleamBookUsers0' 2022-02-07 22:34:17,838 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:17,841 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:17,841 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:17,849 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 735667 == 0 2022-02-07 22:34:17,849 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 735667 == 0. Received: {'172.23.121.127': 735667} for bucket 'GleamBookUsers0' 2022-02-07 22:34:17,849 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:17,851 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:17,851 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:19,081 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:19,509 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:19,815 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:19,880 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 1442 == 0 2022-02-07 22:34:19,880 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1442 == 0. Received: {'172.23.121.115': 1442} for bucket 'GleamBookUsers0' 2022-02-07 22:34:19,880 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:19,881 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:19,882 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:19,959 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:20,276 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 895393 == 0 2022-02-07 22:34:20,276 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 895393 == 0. Received: {'172.23.121.126': 895393} for bucket 'GleamBookUsers0' 2022-02-07 22:34:20,276 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:20,279 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:20,279 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:20,674 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 741343 == 0 2022-02-07 22:34:20,674 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 741343 == 0. Received: {'172.23.121.123': 741343} for bucket 'GleamBookUsers0' 2022-02-07 22:34:20,674 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:20,677 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:20,677 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:21,046 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:21,565 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1230568 == 0 2022-02-07 22:34:21,566 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1230568 == 0. Received: {'172.23.121.129': 1230568} for bucket 'GleamBookUsers0' 2022-02-07 22:34:21,566 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:21,569 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:21,569 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:21,586 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:22,305 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1105942 == 0 2022-02-07 22:34:22,305 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1105942 == 0. Received: {'172.23.121.124': 1105942} for bucket 'GleamBookUsers0' 2022-02-07 22:34:22,306 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:22,308 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:22,309 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:22,842 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:22,851 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:23,230 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1211803 == 0 2022-02-07 22:34:23,230 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1211803 == 0. Received: {'172.23.120.170': 1211803} for bucket 'GleamBookUsers0' 2022-02-07 22:34:23,230 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:23,233 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:23,233 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:23,835 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 735667 == 0 2022-02-07 22:34:23,835 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 735667 == 0. Received: {'172.23.121.127': 735667} for bucket 'GleamBookUsers0' 2022-02-07 22:34:23,835 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:23,838 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:23,838 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:24,098 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 869860 == 0 2022-02-07 22:34:24,099 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 869860 == 0. Received: {'172.23.121.128': 869860} for bucket 'GleamBookUsers0' 2022-02-07 22:34:24,099 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:24,101 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:24,101 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:24,882 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:25,252 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 1442 == 0 2022-02-07 22:34:25,252 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1442 == 0. Received: {'172.23.121.115': 1442} for bucket 'GleamBookUsers0' 2022-02-07 22:34:25,252 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:25,255 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:25,255 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:25,279 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:25,677 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:26,510 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 841449 == 0 2022-02-07 22:34:26,510 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 841449 == 0. Received: {'172.23.121.126': 841449} for bucket 'GleamBookUsers0' 2022-02-07 22:34:26,510 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:26,513 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:26,513 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:26,552 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 741343 == 0 2022-02-07 22:34:26,552 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 741343 == 0. Received: {'172.23.121.123': 741343} for bucket 'GleamBookUsers0' 2022-02-07 22:34:26,552 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:26,553 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:26,553 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:26,572 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:27,309 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:28,157 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1188357 == 0 2022-02-07 22:34:28,157 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1188357 == 0. Received: {'172.23.121.129': 1188357} for bucket 'GleamBookUsers0' 2022-02-07 22:34:28,157 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:28,158 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:28,158 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:28,239 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:28,542 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1067049 == 0 2022-02-07 22:34:28,542 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1067049 == 0. Received: {'172.23.121.124': 1067049} for bucket 'GleamBookUsers0' 2022-02-07 22:34:28,542 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:28,543 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:28,543 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:28,838 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:29,101 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:29,959 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 715313 == 0 2022-02-07 22:34:29,959 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 715313 == 0. Received: {'172.23.121.127': 715313} for bucket 'GleamBookUsers0' 2022-02-07 22:34:29,960 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:29,961 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:29,961 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:30,000 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1211803 == 0 2022-02-07 22:34:30,000 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1211803 == 0. Received: {'172.23.120.170': 1211803} for bucket 'GleamBookUsers0' 2022-02-07 22:34:30,000 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:30,003 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:30,003 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:30,255 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:30,316 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 846544 == 0 2022-02-07 22:34:30,316 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 846544 == 0. Received: {'172.23.121.128': 846544} for bucket 'GleamBookUsers0' 2022-02-07 22:34:30,318 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:30,321 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:30,321 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:30,615 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 6 == 0 2022-02-07 22:34:30,615 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 6 == 0. Received: {'172.23.121.115': 6} for bucket 'GleamBookUsers0' 2022-02-07 22:34:30,617 | infra | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:30,618 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:30,618 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:31,515 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:31,555 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:32,384 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 719676 == 0 2022-02-07 22:34:32,384 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 719676 == 0. Received: {'172.23.121.123': 719676} for bucket 'GleamBookUsers0' 2022-02-07 22:34:32,385 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:32,388 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:32,388 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:32,729 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 821641 == 0 2022-02-07 22:34:32,730 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 821641 == 0. Received: {'172.23.121.126': 821641} for bucket 'GleamBookUsers0' 2022-02-07 22:34:32,730 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:32,732 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:32,732 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:33,160 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:33,546 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:34,831 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 1034033 == 0 2022-02-07 22:34:34,832 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1034033 == 0. Received: {'172.23.121.124': 1034033} for bucket 'GleamBookUsers0' 2022-02-07 22:34:34,832 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:34,835 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:34,835 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:34,848 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1164980 == 0 2022-02-07 22:34:34,848 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1164980 == 0. Received: {'172.23.121.129': 1164980} for bucket 'GleamBookUsers0' 2022-02-07 22:34:34,848 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:34,849 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:34,849 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:34,963 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:35,003 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:35,322 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:35,618 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:35,973 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 693131 == 0 2022-02-07 22:34:35,973 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 693131 == 0. Received: {'172.23.121.127': 693131} for bucket 'GleamBookUsers0' 2022-02-07 22:34:35,973 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:35,976 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:35,976 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:35,994 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 22:34:35,994 | test | DEBUG | pool-3-thread-2 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.115': 0} for bucket 'GleamBookUsers0' 2022-02-07 22:34:35,996 | infra | INFO | pool-3-thread-2 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301821.78' completed 2022-02-07 22:34:36,476 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 799256 == 0 2022-02-07 22:34:36,476 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 799256 == 0. Received: {'172.23.121.128': 799256} for bucket 'GleamBookUsers0' 2022-02-07 22:34:36,476 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:36,480 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:36,480 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:36,640 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1192335 == 0 2022-02-07 22:34:36,640 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1192335 == 0. Received: {'172.23.120.170': 1192335} for bucket 'GleamBookUsers0' 2022-02-07 22:34:36,641 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:36,642 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:36,642 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:37,388 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:37,732 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:38,247 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 699906 == 0 2022-02-07 22:34:38,247 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 699906 == 0. Received: {'172.23.121.123': 699906} for bucket 'GleamBookUsers0' 2022-02-07 22:34:38,249 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:38,250 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:38,250 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:38,915 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 821641 == 0 2022-02-07 22:34:38,917 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 821641 == 0. Received: {'172.23.121.126': 821641} for bucket 'GleamBookUsers0' 2022-02-07 22:34:38,917 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:38,918 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:38,918 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:39,835 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:39,851 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:40,961 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 996933 == 0 2022-02-07 22:34:40,961 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 996933 == 0. Received: {'172.23.121.124': 996933} for bucket 'GleamBookUsers0' 2022-02-07 22:34:40,963 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:40,964 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:40,964 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:40,976 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:41,480 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:41,484 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1163734 == 0 2022-02-07 22:34:41,484 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1163734 == 0. Received: {'172.23.121.129': 1163734} for bucket 'GleamBookUsers0' 2022-02-07 22:34:41,484 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:41,486 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:41,487 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:41,644 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:41,964 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 693131 == 0 2022-02-07 22:34:41,964 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 693131 == 0. Received: {'172.23.121.127': 693131} for bucket 'GleamBookUsers0' 2022-02-07 22:34:41,966 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:41,967 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:41,967 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:42,532 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 784546 == 0 2022-02-07 22:34:42,532 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 784546 == 0. Received: {'172.23.121.128': 784546} for bucket 'GleamBookUsers0' 2022-02-07 22:34:42,532 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:42,536 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:42,536 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:43,250 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:43,279 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1173629 == 0 2022-02-07 22:34:43,279 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1173629 == 0. Received: {'172.23.120.170': 1173629} for bucket 'GleamBookUsers0' 2022-02-07 22:34:43,279 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:43,282 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:43,282 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:43,920 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:44,069 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 680100 == 0 2022-02-07 22:34:44,069 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 680100 == 0. Received: {'172.23.121.123': 680100} for bucket 'GleamBookUsers0' 2022-02-07 22:34:44,069 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:44,072 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:44,072 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:45,125 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 767856 == 0 2022-02-07 22:34:45,127 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 767856 == 0. Received: {'172.23.121.126': 767856} for bucket 'GleamBookUsers0' 2022-02-07 22:34:45,127 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:45,128 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:45,128 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:45,964 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:46,489 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:46,967 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:47,069 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 987018 == 0 2022-02-07 22:34:47,069 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 987018 == 0. Received: {'172.23.121.124': 987018} for bucket 'GleamBookUsers0' 2022-02-07 22:34:47,069 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:47,071 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:47,071 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:47,542 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:47,999 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 673029 == 0 2022-02-07 22:34:48,000 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 673029 == 0. Received: {'172.23.121.127': 673029} for bucket 'GleamBookUsers0' 2022-02-07 22:34:48,000 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:48,002 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:48,002 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:48,095 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1146476 == 0 2022-02-07 22:34:48,095 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1146476 == 0. Received: {'172.23.121.129': 1146476} for bucket 'GleamBookUsers0' 2022-02-07 22:34:48,095 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:48,096 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:48,098 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:48,283 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:48,586 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 763360 == 0 2022-02-07 22:34:48,588 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 763360 == 0. Received: {'172.23.121.128': 763360} for bucket 'GleamBookUsers0' 2022-02-07 22:34:48,588 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:48,589 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:48,589 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:49,072 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:49,938 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1113447 == 0 2022-02-07 22:34:49,940 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1113447 == 0. Received: {'172.23.120.170': 1113447} for bucket 'GleamBookUsers0' 2022-02-07 22:34:49,940 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:49,941 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:49,941 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:49,983 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 662100 == 0 2022-02-07 22:34:49,983 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 662100 == 0. Received: {'172.23.121.123': 662100} for bucket 'GleamBookUsers0' 2022-02-07 22:34:49,986 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:49,989 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:49,989 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:50,128 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:51,278 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 767856 == 0 2022-02-07 22:34:51,279 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 767856 == 0. Received: {'172.23.121.126': 767856} for bucket 'GleamBookUsers0' 2022-02-07 22:34:51,279 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:51,280 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:51,280 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:52,072 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:53,003 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:53,086 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 974354 == 0 2022-02-07 22:34:53,086 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 974354 == 0. Received: {'172.23.121.124': 974354} for bucket 'GleamBookUsers0' 2022-02-07 22:34:53,086 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:53,091 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:53,091 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:53,098 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:53,589 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:53,941 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 673029 == 0 2022-02-07 22:34:53,941 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 673029 == 0. Received: {'172.23.121.127': 673029} for bucket 'GleamBookUsers0' 2022-02-07 22:34:53,941 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:53,944 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:53,944 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:54,711 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 762492 == 0 2022-02-07 22:34:54,711 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 762492 == 0. Received: {'172.23.121.128': 762492} for bucket 'GleamBookUsers0' 2022-02-07 22:34:54,711 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:54,714 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:54,714 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:54,753 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1127004 == 0 2022-02-07 22:34:54,753 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1127004 == 0. Received: {'172.23.121.129': 1127004} for bucket 'GleamBookUsers0' 2022-02-07 22:34:54,753 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:54,756 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:54,756 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:54,943 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:54,990 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:55,825 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 642220 == 0 2022-02-07 22:34:55,826 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 642220 == 0. Received: {'172.23.121.123': 642220} for bucket 'GleamBookUsers0' 2022-02-07 22:34:55,826 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:55,828 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:55,828 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:56,280 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:56,444 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1095009 == 0 2022-02-07 22:34:56,444 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1095009 == 0. Received: {'172.23.120.170': 1095009} for bucket 'GleamBookUsers0' 2022-02-07 22:34:56,444 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:56,447 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:56,447 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:57,397 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 748954 == 0 2022-02-07 22:34:57,398 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 748954 == 0. Received: {'172.23.121.126': 748954} for bucket 'GleamBookUsers0' 2022-02-07 22:34:57,398 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:57,401 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:57,401 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:58,091 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:58,944 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:59,117 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 949768 == 0 2022-02-07 22:34:59,117 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 949768 == 0. Received: {'172.23.121.124': 949768} for bucket 'GleamBookUsers0' 2022-02-07 22:34:59,117 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:59,118 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:59,118 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:34:59,716 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:59,756 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:34:59,903 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 653329 == 0 2022-02-07 22:34:59,903 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 653329 == 0. Received: {'172.23.121.127': 653329} for bucket 'GleamBookUsers0' 2022-02-07 22:34:59,904 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:34:59,905 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:34:59,905 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:00,740 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 712948 == 0 2022-02-07 22:35:00,740 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 712948 == 0. Received: {'172.23.121.128': 712948} for bucket 'GleamBookUsers0' 2022-02-07 22:35:00,740 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:00,742 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:00,742 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:00,828 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:01,298 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1107094 == 0 2022-02-07 22:35:01,298 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1107094 == 0. Received: {'172.23.121.129': 1107094} for bucket 'GleamBookUsers0' 2022-02-07 22:35:01,298 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:01,299 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:01,299 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:01,447 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:01,690 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 622092 == 0 2022-02-07 22:35:01,690 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 622092 == 0. Received: {'172.23.121.123': 622092} for bucket 'GleamBookUsers0' 2022-02-07 22:35:01,690 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:01,693 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:01,694 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:02,401 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:02,982 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1087735 == 0 2022-02-07 22:35:02,983 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1087735 == 0. Received: {'172.23.120.170': 1087735} for bucket 'GleamBookUsers0' 2022-02-07 22:35:02,983 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:02,986 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:02,986 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:03,515 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 733136 == 0 2022-02-07 22:35:03,516 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 733136 == 0. Received: {'172.23.121.126': 733136} for bucket 'GleamBookUsers0' 2022-02-07 22:35:03,516 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:03,517 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:03,517 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:04,119 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:04,905 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:05,092 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 919710 == 0 2022-02-07 22:35:05,092 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 919710 == 0. Received: {'172.23.121.124': 919710} for bucket 'GleamBookUsers0' 2022-02-07 22:35:05,092 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:05,094 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:05,095 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:05,743 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:05,852 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 632903 == 0 2022-02-07 22:35:05,854 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 632903 == 0. Received: {'172.23.121.127': 632903} for bucket 'GleamBookUsers0' 2022-02-07 22:35:05,854 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:05,855 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:05,855 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:06,301 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:06,710 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:06,779 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 712948 == 0 2022-02-07 22:35:06,779 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 712948 == 0. Received: {'172.23.121.128': 712948} for bucket 'GleamBookUsers0' 2022-02-07 22:35:06,779 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:06,780 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:06,782 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:07,581 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 603290 == 0 2022-02-07 22:35:07,581 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 603290 == 0. Received: {'172.23.121.123': 603290} for bucket 'GleamBookUsers0' 2022-02-07 22:35:07,582 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:07,584 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:07,584 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:07,880 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1052623 == 0 2022-02-07 22:35:07,881 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1052623 == 0. Received: {'172.23.121.129': 1052623} for bucket 'GleamBookUsers0' 2022-02-07 22:35:07,881 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:07,882 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:07,882 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:07,986 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:08,519 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:09,341 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1057890 == 0 2022-02-07 22:35:09,341 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1057890 == 0. Received: {'172.23.120.170': 1057890} for bucket 'GleamBookUsers0' 2022-02-07 22:35:09,342 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:09,344 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:09,344 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:09,540 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 698257 == 0 2022-02-07 22:35:09,540 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 698257 == 0. Received: {'172.23.121.126': 698257} for bucket 'GleamBookUsers0' 2022-02-07 22:35:09,540 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:09,542 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:09,542 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:10,095 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:10,855 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:11,089 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 913316 == 0 2022-02-07 22:35:11,091 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 913316 == 0. Received: {'172.23.121.124': 913316} for bucket 'GleamBookUsers0' 2022-02-07 22:35:11,091 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:11,092 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:11,094 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:11,782 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:11,788 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 612225 == 0 2022-02-07 22:35:11,789 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 612225 == 0. Received: {'172.23.121.127': 612225} for bucket 'GleamBookUsers0' 2022-02-07 22:35:11,789 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:11,790 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:11,790 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:12,585 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:12,874 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 684748 == 0 2022-02-07 22:35:12,875 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 684748 == 0. Received: {'172.23.121.128': 684748} for bucket 'GleamBookUsers0' 2022-02-07 22:35:12,875 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:12,877 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:12,878 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:12,884 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:13,384 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 566908 == 0 2022-02-07 22:35:13,384 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 566908 == 0. Received: {'172.23.121.123': 566908} for bucket 'GleamBookUsers0' 2022-02-07 22:35:13,384 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:13,387 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:13,387 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:14,345 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:14,371 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1033567 == 0 2022-02-07 22:35:14,371 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1033567 == 0. Received: {'172.23.121.129': 1033567} for bucket 'GleamBookUsers0' 2022-02-07 22:35:14,371 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:14,374 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:14,374 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:14,543 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:15,657 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 678973 == 0 2022-02-07 22:35:15,657 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 678973 == 0. Received: {'172.23.121.126': 678973} for bucket 'GleamBookUsers0' 2022-02-07 22:35:15,657 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:15,658 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:15,660 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:15,779 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1035554 == 0 2022-02-07 22:35:15,779 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1035554 == 0. Received: {'172.23.120.170': 1035554} for bucket 'GleamBookUsers0' 2022-02-07 22:35:15,779 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:15,782 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:15,782 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:16,094 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:16,792 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:17,056 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 913316 == 0 2022-02-07 22:35:17,058 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 913316 == 0. Received: {'172.23.121.124': 913316} for bucket 'GleamBookUsers0' 2022-02-07 22:35:17,058 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:17,059 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:17,059 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:17,706 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 592287 == 0 2022-02-07 22:35:17,707 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 592287 == 0. Received: {'172.23.121.127': 592287} for bucket 'GleamBookUsers0' 2022-02-07 22:35:17,707 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:17,709 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:17,709 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:17,878 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:18,388 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:18,947 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 684748 == 0 2022-02-07 22:35:18,947 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 684748 == 0. Received: {'172.23.121.128': 684748} for bucket 'GleamBookUsers0' 2022-02-07 22:35:18,948 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:18,950 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:18,950 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:19,168 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 550256 == 0 2022-02-07 22:35:19,168 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 550256 == 0. Received: {'172.23.121.123': 550256} for bucket 'GleamBookUsers0' 2022-02-07 22:35:19,168 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:19,171 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:19,171 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:19,374 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:20,660 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:20,782 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:20,839 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 1015233 == 0 2022-02-07 22:35:20,839 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1015233 == 0. Received: {'172.23.121.129': 1015233} for bucket 'GleamBookUsers0' 2022-02-07 22:35:20,839 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:20,842 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:20,842 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:21,730 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 664291 == 0 2022-02-07 22:35:21,732 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 664291 == 0. Received: {'172.23.121.126': 664291} for bucket 'GleamBookUsers0' 2022-02-07 22:35:21,732 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:21,733 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:21,734 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:22,059 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:22,095 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1027454 == 0 2022-02-07 22:35:22,095 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1027454 == 0. Received: {'172.23.120.170': 1027454} for bucket 'GleamBookUsers0' 2022-02-07 22:35:22,095 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:22,098 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:22,098 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:22,710 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:23,105 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 889628 == 0 2022-02-07 22:35:23,105 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 889628 == 0. Received: {'172.23.121.124': 889628} for bucket 'GleamBookUsers0' 2022-02-07 22:35:23,105 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:23,108 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:23,108 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:23,709 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 571783 == 0 2022-02-07 22:35:23,709 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 571783 == 0. Received: {'172.23.121.127': 571783} for bucket 'GleamBookUsers0' 2022-02-07 22:35:23,709 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:23,710 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:23,710 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:23,951 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:24,171 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:24,989 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 550256 == 0 2022-02-07 22:35:24,989 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 550256 == 0. Received: {'172.23.121.123': 550256} for bucket 'GleamBookUsers0' 2022-02-07 22:35:24,989 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:24,992 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:24,992 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:25,028 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 656018 == 0 2022-02-07 22:35:25,028 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 656018 == 0. Received: {'172.23.121.128': 656018} for bucket 'GleamBookUsers0' 2022-02-07 22:35:25,028 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:25,029 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:25,029 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:25,844 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:26,742 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:27,098 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:27,190 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 960971 == 0 2022-02-07 22:35:27,190 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 960971 == 0. Received: {'172.23.121.129': 960971} for bucket 'GleamBookUsers0' 2022-02-07 22:35:27,190 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:27,193 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:27,193 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:27,766 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 632885 == 0 2022-02-07 22:35:27,766 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 632885 == 0. Received: {'172.23.121.126': 632885} for bucket 'GleamBookUsers0' 2022-02-07 22:35:27,766 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:27,769 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:27,769 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:28,108 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:28,410 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1027454 == 0 2022-02-07 22:35:28,410 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1027454 == 0. Received: {'172.23.120.170': 1027454} for bucket 'GleamBookUsers0' 2022-02-07 22:35:28,410 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:28,413 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:28,413 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:28,711 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:29,073 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 884652 == 0 2022-02-07 22:35:29,073 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 884652 == 0. Received: {'172.23.121.124': 884652} for bucket 'GleamBookUsers0' 2022-02-07 22:35:29,073 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:29,076 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:29,076 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:29,667 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 571783 == 0 2022-02-07 22:35:29,667 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 571783 == 0. Received: {'172.23.121.127': 571783} for bucket 'GleamBookUsers0' 2022-02-07 22:35:29,667 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:29,671 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:29,671 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:29,992 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:30,030 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:30,733 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 497920 == 0 2022-02-07 22:35:30,733 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 497920 == 0. Received: {'172.23.121.123': 497920} for bucket 'GleamBookUsers0' 2022-02-07 22:35:30,733 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:30,736 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:30,736 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:31,029 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 635554 == 0 2022-02-07 22:35:31,030 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 635554 == 0. Received: {'172.23.121.128': 635554} for bucket 'GleamBookUsers0' 2022-02-07 22:35:31,030 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:31,032 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:31,032 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:32,194 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:32,769 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:33,414 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:33,520 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 942473 == 0 2022-02-07 22:35:33,520 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 942473 == 0. Received: {'172.23.121.129': 942473} for bucket 'GleamBookUsers0' 2022-02-07 22:35:33,522 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:33,523 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:33,523 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:33,799 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 617423 == 0 2022-02-07 22:35:33,799 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 617423 == 0. Received: {'172.23.121.126': 617423} for bucket 'GleamBookUsers0' 2022-02-07 22:35:33,799 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:33,802 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:33,802 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:34,076 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:34,671 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:34,734 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 1021512 == 0 2022-02-07 22:35:34,734 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1021512 == 0. Received: {'172.23.120.170': 1021512} for bucket 'GleamBookUsers0' 2022-02-07 22:35:34,734 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:34,737 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:34,737 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:35,069 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 880302 == 0 2022-02-07 22:35:35,069 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 880302 == 0. Received: {'172.23.121.124': 880302} for bucket 'GleamBookUsers0' 2022-02-07 22:35:35,069 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:35,071 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:35,071 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:35,512 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 514047 == 0 2022-02-07 22:35:35,512 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 514047 == 0. Received: {'172.23.121.127': 514047} for bucket 'GleamBookUsers0' 2022-02-07 22:35:35,512 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:35,513 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:35,513 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:35,736 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:36,032 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:36,470 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 497920 == 0 2022-02-07 22:35:36,470 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 497920 == 0. Received: {'172.23.121.123': 497920} for bucket 'GleamBookUsers0' 2022-02-07 22:35:36,470 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:36,473 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:36,473 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:36,957 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 618326 == 0 2022-02-07 22:35:36,957 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 618326 == 0. Received: {'172.23.121.128': 618326} for bucket 'GleamBookUsers0' 2022-02-07 22:35:36,957 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:36,959 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:36,959 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:38,525 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:38,802 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:39,739 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:40,005 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 925845 == 0 2022-02-07 22:35:40,005 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 925845 == 0. Received: {'172.23.121.129': 925845} for bucket 'GleamBookUsers0' 2022-02-07 22:35:40,005 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:40,006 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 602427 == 0 2022-02-07 22:35:40,006 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 602427 == 0. Received: {'172.23.121.126': 602427} for bucket 'GleamBookUsers0' 2022-02-07 22:35:40,006 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:40,007 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:40,007 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:40,009 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:40,009 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:40,071 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:40,515 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:41,104 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 855792 == 0 2022-02-07 22:35:41,105 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 855792 == 0. Received: {'172.23.121.124': 855792} for bucket 'GleamBookUsers0' 2022-02-07 22:35:41,105 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:41,107 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:41,107 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:41,140 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 999169 == 0 2022-02-07 22:35:41,141 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 999169 == 0. Received: {'172.23.120.170': 999169} for bucket 'GleamBookUsers0' 2022-02-07 22:35:41,141 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:41,144 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:41,144 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:41,323 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 514047 == 0 2022-02-07 22:35:41,323 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 514047 == 0. Received: {'172.23.121.127': 514047} for bucket 'GleamBookUsers0' 2022-02-07 22:35:41,323 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:41,325 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:41,325 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:41,473 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:41,960 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:42,187 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 466110 == 0 2022-02-07 22:35:42,187 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 466110 == 0. Received: {'172.23.121.123': 466110} for bucket 'GleamBookUsers0' 2022-02-07 22:35:42,187 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:42,188 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:42,188 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:42,898 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 609062 == 0 2022-02-07 22:35:42,898 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 609062 == 0. Received: {'172.23.121.128': 609062} for bucket 'GleamBookUsers0' 2022-02-07 22:35:42,898 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:42,900 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:42,901 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:45,009 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:45,009 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:45,904 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 555601 == 0 2022-02-07 22:35:45,905 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 555601 == 0. Received: {'172.23.121.126': 555601} for bucket 'GleamBookUsers0' 2022-02-07 22:35:45,905 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:45,907 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:45,907 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:46,108 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:46,144 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:46,293 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 892365 == 0 2022-02-07 22:35:46,293 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 892365 == 0. Received: {'172.23.121.129': 892365} for bucket 'GleamBookUsers0' 2022-02-07 22:35:46,293 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:46,296 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:46,296 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:46,328 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:47,151 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 829998 == 0 2022-02-07 22:35:47,151 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 829998 == 0. Received: {'172.23.121.124': 829998} for bucket 'GleamBookUsers0' 2022-02-07 22:35:47,151 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:47,154 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:47,154 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:47,188 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:47,253 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 478137 == 0 2022-02-07 22:35:47,253 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 478137 == 0. Received: {'172.23.121.127': 478137} for bucket 'GleamBookUsers0' 2022-02-07 22:35:47,253 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:47,256 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:47,256 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:47,552 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 978113 == 0 2022-02-07 22:35:47,552 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 978113 == 0. Received: {'172.23.120.170': 978113} for bucket 'GleamBookUsers0' 2022-02-07 22:35:47,552 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:47,555 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:47,555 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:47,888 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 432221 == 0 2022-02-07 22:35:47,888 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 432221 == 0. Received: {'172.23.121.123': 432221} for bucket 'GleamBookUsers0' 2022-02-07 22:35:47,888 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:47,891 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:47,891 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:47,901 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:48,802 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 591110 == 0 2022-02-07 22:35:48,803 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 591110 == 0. Received: {'172.23.121.128': 591110} for bucket 'GleamBookUsers0' 2022-02-07 22:35:48,803 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:48,805 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:48,805 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:50,907 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:51,296 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:51,789 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 555601 == 0 2022-02-07 22:35:51,789 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 555601 == 0. Received: {'172.23.121.126': 555601} for bucket 'GleamBookUsers0' 2022-02-07 22:35:51,789 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:51,792 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:51,792 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:52,154 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:52,256 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:52,555 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:52,578 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 872723 == 0 2022-02-07 22:35:52,578 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 872723 == 0. Received: {'172.23.121.129': 872723} for bucket 'GleamBookUsers0' 2022-02-07 22:35:52,578 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:52,581 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:52,581 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:52,891 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:53,138 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 478137 == 0 2022-02-07 22:35:53,138 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 478137 == 0. Received: {'172.23.121.127': 478137} for bucket 'GleamBookUsers0' 2022-02-07 22:35:53,138 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:53,141 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:53,141 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:53,167 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 824830 == 0 2022-02-07 22:35:53,167 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 824830 == 0. Received: {'172.23.121.124': 824830} for bucket 'GleamBookUsers0' 2022-02-07 22:35:53,167 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:53,168 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:53,168 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:53,625 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 401285 == 0 2022-02-07 22:35:53,625 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 401285 == 0. Received: {'172.23.121.123': 401285} for bucket 'GleamBookUsers0' 2022-02-07 22:35:53,625 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:53,628 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:53,628 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:53,806 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:53,867 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 956405 == 0 2022-02-07 22:35:53,867 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 956405 == 0. Received: {'172.23.120.170': 956405} for bucket 'GleamBookUsers0' 2022-02-07 22:35:53,867 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:53,868 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:53,868 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:54,665 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 546538 == 0 2022-02-07 22:35:54,667 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 546538 == 0. Received: {'172.23.121.128': 546538} for bucket 'GleamBookUsers0' 2022-02-07 22:35:54,667 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:54,668 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:54,670 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:56,792 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:57,582 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:57,598 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 498561 == 0 2022-02-07 22:35:57,598 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 498561 == 0. Received: {'172.23.121.126': 498561} for bucket 'GleamBookUsers0' 2022-02-07 22:35:57,598 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:57,601 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:57,601 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:58,141 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:58,168 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:58,628 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:58,869 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:35:58,927 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 872723 == 0 2022-02-07 22:35:58,927 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 872723 == 0. Received: {'172.23.121.129': 872723} for bucket 'GleamBookUsers0' 2022-02-07 22:35:58,927 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:58,930 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:58,931 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:59,063 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 461537 == 0 2022-02-07 22:35:59,063 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 461537 == 0. Received: {'172.23.121.127': 461537} for bucket 'GleamBookUsers0' 2022-02-07 22:35:59,065 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:59,066 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:59,066 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:59,148 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 799688 == 0 2022-02-07 22:35:59,148 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 799688 == 0. Received: {'172.23.121.124': 799688} for bucket 'GleamBookUsers0' 2022-02-07 22:35:59,150 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:59,151 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:59,151 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:59,336 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 401285 == 0 2022-02-07 22:35:59,338 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 401285 == 0. Received: {'172.23.121.123': 401285} for bucket 'GleamBookUsers0' 2022-02-07 22:35:59,338 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:35:59,339 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:35:59,339 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:35:59,670 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:00,183 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 950155 == 0 2022-02-07 22:36:00,184 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 950155 == 0. Received: {'172.23.120.170': 950155} for bucket 'GleamBookUsers0' 2022-02-07 22:36:00,184 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:00,186 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:00,186 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:00,526 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 539484 == 0 2022-02-07 22:36:00,526 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 539484 == 0. Received: {'172.23.121.128': 539484} for bucket 'GleamBookUsers0' 2022-02-07 22:36:00,526 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:00,529 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:00,529 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:02,601 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:03,404 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 498561 == 0 2022-02-07 22:36:03,404 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 498561 == 0. Received: {'172.23.121.126': 498561} for bucket 'GleamBookUsers0' 2022-02-07 22:36:03,404 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:03,405 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:03,405 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:03,931 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:04,066 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:04,151 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:04,341 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:04,865 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 425865 == 0 2022-02-07 22:36:04,867 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 425865 == 0. Received: {'172.23.121.127': 425865} for bucket 'GleamBookUsers0' 2022-02-07 22:36:04,867 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:04,868 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:04,869 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:05,173 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 383382 == 0 2022-02-07 22:36:05,174 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 383382 == 0. Received: {'172.23.121.123': 383382} for bucket 'GleamBookUsers0' 2022-02-07 22:36:05,174 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:05,176 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:05,177 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:05,187 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:05,247 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 798138 == 0 2022-02-07 22:36:05,247 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 798138 == 0. Received: {'172.23.121.124': 798138} for bucket 'GleamBookUsers0' 2022-02-07 22:36:05,247 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:05,250 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:05,250 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:05,369 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 838711 == 0 2022-02-07 22:36:05,371 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 838711 == 0. Received: {'172.23.121.129': 838711} for bucket 'GleamBookUsers0' 2022-02-07 22:36:05,371 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:05,372 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:05,372 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:05,529 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:06,411 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 522844 == 0 2022-02-07 22:36:06,411 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 522844 == 0. Received: {'172.23.121.128': 522844} for bucket 'GleamBookUsers0' 2022-02-07 22:36:06,411 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:06,414 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:06,414 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:06,453 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 903566 == 0 2022-02-07 22:36:06,453 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 903566 == 0. Received: {'172.23.120.170': 903566} for bucket 'GleamBookUsers0' 2022-02-07 22:36:06,453 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:06,456 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:06,456 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:08,407 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:09,122 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 443499 == 0 2022-02-07 22:36:09,124 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 443499 == 0. Received: {'172.23.121.126': 443499} for bucket 'GleamBookUsers0' 2022-02-07 22:36:09,124 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:09,125 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:09,125 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:09,869 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:10,177 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:10,250 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:10,374 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:10,644 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 410833 == 0 2022-02-07 22:36:10,644 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 410833 == 0. Received: {'172.23.121.127': 410833} for bucket 'GleamBookUsers0' 2022-02-07 22:36:10,644 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:10,645 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:10,647 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:10,904 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 355476 == 0 2022-02-07 22:36:10,904 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 355476 == 0. Received: {'172.23.121.123': 355476} for bucket 'GleamBookUsers0' 2022-02-07 22:36:10,904 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:10,907 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:10,907 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:11,174 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 798138 == 0 2022-02-07 22:36:11,174 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 798138 == 0. Received: {'172.23.121.124': 798138} for bucket 'GleamBookUsers0' 2022-02-07 22:36:11,174 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:11,177 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:11,177 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:11,414 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:11,457 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:11,720 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 807945 == 0 2022-02-07 22:36:11,720 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 807945 == 0. Received: {'172.23.121.129': 807945} for bucket 'GleamBookUsers0' 2022-02-07 22:36:11,720 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:11,721 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:11,721 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:12,233 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 483323 == 0 2022-02-07 22:36:12,233 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 483323 == 0. Received: {'172.23.121.128': 483323} for bucket 'GleamBookUsers0' 2022-02-07 22:36:12,233 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:12,236 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:12,236 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:12,657 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 878816 == 0 2022-02-07 22:36:12,657 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 878816 == 0. Received: {'172.23.120.170': 878816} for bucket 'GleamBookUsers0' 2022-02-07 22:36:12,657 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:12,660 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:12,660 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:14,125 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:14,865 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 443499 == 0 2022-02-07 22:36:14,865 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 443499 == 0. Received: {'172.23.121.126': 443499} for bucket 'GleamBookUsers0' 2022-02-07 22:36:14,865 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:14,867 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:14,867 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:15,648 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:15,907 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:16,177 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:16,384 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 377381 == 0 2022-02-07 22:36:16,385 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 377381 == 0. Received: {'172.23.121.127': 377381} for bucket 'GleamBookUsers0' 2022-02-07 22:36:16,385 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:16,388 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:16,388 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:16,573 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 325838 == 0 2022-02-07 22:36:16,573 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 325838 == 0. Received: {'172.23.121.123': 325838} for bucket 'GleamBookUsers0' 2022-02-07 22:36:16,573 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:16,578 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:16,578 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:16,723 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:17,128 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 753630 == 0 2022-02-07 22:36:17,128 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 753630 == 0. Received: {'172.23.121.124': 753630} for bucket 'GleamBookUsers0' 2022-02-07 22:36:17,128 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:17,130 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:17,130 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:17,236 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:17,661 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:18,101 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 458775 == 0 2022-02-07 22:36:18,101 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 458775 == 0. Received: {'172.23.121.128': 458775} for bucket 'GleamBookUsers0' 2022-02-07 22:36:18,102 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:18,105 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:18,105 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:18,111 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 789917 == 0 2022-02-07 22:36:18,112 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 789917 == 0. Received: {'172.23.121.129': 789917} for bucket 'GleamBookUsers0' 2022-02-07 22:36:18,112 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:18,115 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:18,115 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:18,878 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 870256 == 0 2022-02-07 22:36:18,880 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 870256 == 0. Received: {'172.23.120.170': 870256} for bucket 'GleamBookUsers0' 2022-02-07 22:36:18,880 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:18,882 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:18,882 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:19,868 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:20,573 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 389939 == 0 2022-02-07 22:36:20,573 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 389939 == 0. Received: {'172.23.121.126': 389939} for bucket 'GleamBookUsers0' 2022-02-07 22:36:20,573 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:20,575 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:20,575 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:21,388 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:21,578 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:22,131 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:22,157 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 363413 == 0 2022-02-07 22:36:22,157 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 363413 == 0. Received: {'172.23.121.127': 363413} for bucket 'GleamBookUsers0' 2022-02-07 22:36:22,157 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:22,161 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:22,161 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:22,293 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 325838 == 0 2022-02-07 22:36:22,295 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 325838 == 0. Received: {'172.23.121.123': 325838} for bucket 'GleamBookUsers0' 2022-02-07 22:36:22,295 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:22,296 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:22,296 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:23,073 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 731987 == 0 2022-02-07 22:36:23,075 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 731987 == 0. Received: {'172.23.121.124': 731987} for bucket 'GleamBookUsers0' 2022-02-07 22:36:23,075 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:23,076 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:23,076 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:23,105 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:23,115 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:23,884 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:23,914 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 443175 == 0 2022-02-07 22:36:23,914 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 443175 == 0. Received: {'172.23.121.128': 443175} for bucket 'GleamBookUsers0' 2022-02-07 22:36:23,915 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:23,917 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:23,917 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:24,375 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 789917 == 0 2022-02-07 22:36:24,375 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 789917 == 0. Received: {'172.23.121.129': 789917} for bucket 'GleamBookUsers0' 2022-02-07 22:36:24,375 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:24,377 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:24,377 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:25,122 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 866982 == 0 2022-02-07 22:36:25,122 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 866982 == 0. Received: {'172.23.120.170': 866982} for bucket 'GleamBookUsers0' 2022-02-07 22:36:25,124 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:25,125 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:25,125 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:25,576 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:26,276 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 389939 == 0 2022-02-07 22:36:26,276 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 389939 == 0. Received: {'172.23.121.126': 389939} for bucket 'GleamBookUsers0' 2022-02-07 22:36:26,276 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:26,278 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:26,278 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:27,161 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:27,298 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:27,927 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 320119 == 0 2022-02-07 22:36:27,927 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 320119 == 0. Received: {'172.23.121.127': 320119} for bucket 'GleamBookUsers0' 2022-02-07 22:36:27,927 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:27,930 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:27,930 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:28,069 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 298914 == 0 2022-02-07 22:36:28,069 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 298914 == 0. Received: {'172.23.121.123': 298914} for bucket 'GleamBookUsers0' 2022-02-07 22:36:28,069 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:28,071 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:28,071 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:28,076 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:28,917 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:28,974 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 731987 == 0 2022-02-07 22:36:28,974 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 731987 == 0. Received: {'172.23.121.124': 731987} for bucket 'GleamBookUsers0' 2022-02-07 22:36:28,974 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:28,976 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:28,976 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:29,378 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:29,720 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 418923 == 0 2022-02-07 22:36:29,720 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 418923 == 0. Received: {'172.23.121.128': 418923} for bucket 'GleamBookUsers0' 2022-02-07 22:36:29,720 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:29,723 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:29,723 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:30,125 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:30,690 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 773647 == 0 2022-02-07 22:36:30,690 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 773647 == 0. Received: {'172.23.121.129': 773647} for bucket 'GleamBookUsers0' 2022-02-07 22:36:30,690 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:30,693 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:30,694 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:31,278 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:31,303 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 849440 == 0 2022-02-07 22:36:31,303 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 849440 == 0. Received: {'172.23.120.170': 849440} for bucket 'GleamBookUsers0' 2022-02-07 22:36:31,303 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:31,305 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:31,305 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:32,023 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 366035 == 0 2022-02-07 22:36:32,025 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 366035 == 0. Received: {'172.23.121.126': 366035} for bucket 'GleamBookUsers0' 2022-02-07 22:36:32,025 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:32,026 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:32,026 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:32,931 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:33,072 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:33,618 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 305503 == 0 2022-02-07 22:36:33,619 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 305503 == 0. Received: {'172.23.121.127': 305503} for bucket 'GleamBookUsers0' 2022-02-07 22:36:33,619 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:33,621 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:33,621 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:33,746 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 285056 == 0 2022-02-07 22:36:33,747 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 285056 == 0. Received: {'172.23.121.123': 285056} for bucket 'GleamBookUsers0' 2022-02-07 22:36:33,747 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:33,750 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:33,750 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:33,977 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:34,724 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:34,877 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 710807 == 0 2022-02-07 22:36:34,877 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 710807 == 0. Received: {'172.23.121.124': 710807} for bucket 'GleamBookUsers0' 2022-02-07 22:36:34,877 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:34,878 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:34,878 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:35,506 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 399455 == 0 2022-02-07 22:36:35,507 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 399455 == 0. Received: {'172.23.121.128': 399455} for bucket 'GleamBookUsers0' 2022-02-07 22:36:35,507 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:35,509 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:35,509 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:35,694 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:36,308 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:36,855 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 724897 == 0 2022-02-07 22:36:36,857 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 724897 == 0. Received: {'172.23.121.129': 724897} for bucket 'GleamBookUsers0' 2022-02-07 22:36:36,857 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:36,858 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:36,858 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:37,026 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:37,480 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 818984 == 0 2022-02-07 22:36:37,480 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 818984 == 0. Received: {'172.23.120.170': 818984} for bucket 'GleamBookUsers0' 2022-02-07 22:36:37,480 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:37,482 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:37,482 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:37,752 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 329845 == 0 2022-02-07 22:36:37,753 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 329845 == 0. Received: {'172.23.121.126': 329845} for bucket 'GleamBookUsers0' 2022-02-07 22:36:37,753 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:37,755 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:37,755 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:38,622 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:38,750 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:39,368 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 289673 == 0 2022-02-07 22:36:39,368 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 289673 == 0. Received: {'172.23.121.127': 289673} for bucket 'GleamBookUsers0' 2022-02-07 22:36:39,368 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:39,371 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:39,371 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:39,470 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 258566 == 0 2022-02-07 22:36:39,470 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 258566 == 0. Received: {'172.23.121.123': 258566} for bucket 'GleamBookUsers0' 2022-02-07 22:36:39,470 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:39,473 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:39,473 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:39,880 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:40,509 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:40,703 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 691181 == 0 2022-02-07 22:36:40,703 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 691181 == 0. Received: {'172.23.121.124': 691181} for bucket 'GleamBookUsers0' 2022-02-07 22:36:40,703 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:40,706 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:40,706 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:41,253 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 362663 == 0 2022-02-07 22:36:41,255 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 362663 == 0. Received: {'172.23.121.128': 362663} for bucket 'GleamBookUsers0' 2022-02-07 22:36:41,255 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:41,256 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:41,256 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:41,858 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:42,483 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:42,755 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:43,032 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 708627 == 0 2022-02-07 22:36:43,032 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 708627 == 0. Received: {'172.23.121.129': 708627} for bucket 'GleamBookUsers0' 2022-02-07 22:36:43,032 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:43,035 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:43,035 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:43,450 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 305173 == 0 2022-02-07 22:36:43,450 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 305173 == 0. Received: {'172.23.121.126': 305173} for bucket 'GleamBookUsers0' 2022-02-07 22:36:43,451 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:43,453 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:43,454 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:43,681 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 780150 == 0 2022-02-07 22:36:43,681 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 780150 == 0. Received: {'172.23.120.170': 780150} for bucket 'GleamBookUsers0' 2022-02-07 22:36:43,681 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:43,684 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:43,684 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:44,371 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:44,473 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:44,973 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 251199 == 0 2022-02-07 22:36:44,973 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 251199 == 0. Received: {'172.23.121.127': 251199} for bucket 'GleamBookUsers0' 2022-02-07 22:36:44,973 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:44,976 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:44,976 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:45,114 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 235948 == 0 2022-02-07 22:36:45,115 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 235948 == 0. Received: {'172.23.121.123': 235948} for bucket 'GleamBookUsers0' 2022-02-07 22:36:45,115 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:45,117 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:45,117 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:45,707 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:46,256 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:46,503 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 670950 == 0 2022-02-07 22:36:46,503 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 670950 == 0. Received: {'172.23.121.124': 670950} for bucket 'GleamBookUsers0' 2022-02-07 22:36:46,505 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:46,506 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:46,506 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:46,973 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 330511 == 0 2022-02-07 22:36:46,973 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 330511 == 0. Received: {'172.23.121.128': 330511} for bucket 'GleamBookUsers0' 2022-02-07 22:36:46,973 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:46,976 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:46,976 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:48,035 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:48,456 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:48,684 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:49,151 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 283196 == 0 2022-02-07 22:36:49,151 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 283196 == 0. Received: {'172.23.121.126': 283196} for bucket 'GleamBookUsers0' 2022-02-07 22:36:49,151 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:49,154 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:49,154 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:49,263 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 663803 == 0 2022-02-07 22:36:49,263 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 663803 == 0. Received: {'172.23.121.129': 663803} for bucket 'GleamBookUsers0' 2022-02-07 22:36:49,263 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:49,266 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:49,266 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:49,838 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 756810 == 0 2022-02-07 22:36:49,838 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 756810 == 0. Received: {'172.23.120.170': 756810} for bucket 'GleamBookUsers0' 2022-02-07 22:36:49,838 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:49,841 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:49,841 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:49,976 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:50,118 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:50,575 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 239571 == 0 2022-02-07 22:36:50,575 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 239571 == 0. Received: {'172.23.121.127': 239571} for bucket 'GleamBookUsers0' 2022-02-07 22:36:50,575 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:50,578 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:50,578 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:50,733 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 201363 == 0 2022-02-07 22:36:50,733 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 201363 == 0. Received: {'172.23.121.123': 201363} for bucket 'GleamBookUsers0' 2022-02-07 22:36:50,733 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:50,736 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:50,736 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:51,506 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:51,976 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:52,306 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 629742 == 0 2022-02-07 22:36:52,306 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 629742 == 0. Received: {'172.23.121.124': 629742} for bucket 'GleamBookUsers0' 2022-02-07 22:36:52,306 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:52,309 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:52,309 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:52,683 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 317615 == 0 2022-02-07 22:36:52,683 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 317615 == 0. Received: {'172.23.121.128': 317615} for bucket 'GleamBookUsers0' 2022-02-07 22:36:52,683 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:52,684 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:52,684 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:54,154 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:54,267 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:54,809 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 261740 == 0 2022-02-07 22:36:54,809 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 261740 == 0. Received: {'172.23.121.126': 261740} for bucket 'GleamBookUsers0' 2022-02-07 22:36:54,809 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:54,811 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:54,811 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:54,841 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:55,418 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 649301 == 0 2022-02-07 22:36:55,418 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 649301 == 0. Received: {'172.23.121.129': 649301} for bucket 'GleamBookUsers0' 2022-02-07 22:36:55,420 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:55,421 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:55,421 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:55,578 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:55,736 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:55,914 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 722737 == 0 2022-02-07 22:36:55,914 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 722737 == 0. Received: {'172.23.120.170': 722737} for bucket 'GleamBookUsers0' 2022-02-07 22:36:55,914 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:55,915 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:55,915 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:56,194 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 225029 == 0 2022-02-07 22:36:56,194 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 225029 == 0. Received: {'172.23.121.127': 225029} for bucket 'GleamBookUsers0' 2022-02-07 22:36:56,194 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:56,197 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:56,197 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:56,355 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 187645 == 0 2022-02-07 22:36:56,355 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 187645 == 0. Received: {'172.23.121.123': 187645} for bucket 'GleamBookUsers0' 2022-02-07 22:36:56,355 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:56,357 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:56,358 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:57,309 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:57,686 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:36:58,101 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 629742 == 0 2022-02-07 22:36:58,101 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 629742 == 0. Received: {'172.23.121.124': 629742} for bucket 'GleamBookUsers0' 2022-02-07 22:36:58,101 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:58,104 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:58,104 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:58,381 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 290301 == 0 2022-02-07 22:36:58,381 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 290301 == 0. Received: {'172.23.121.128': 290301} for bucket 'GleamBookUsers0' 2022-02-07 22:36:58,381 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:36:58,384 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:36:58,384 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:36:59,811 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:00,421 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:00,440 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 239616 == 0 2022-02-07 22:37:00,440 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 239616 == 0. Received: {'172.23.121.126': 239616} for bucket 'GleamBookUsers0' 2022-02-07 22:37:00,441 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:00,443 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:00,443 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:00,917 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:01,197 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:01,358 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:01,463 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 632051 == 0 2022-02-07 22:37:01,464 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 632051 == 0. Received: {'172.23.121.129': 632051} for bucket 'GleamBookUsers0' 2022-02-07 22:37:01,464 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:01,467 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:01,467 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:01,926 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 202318 == 0 2022-02-07 22:37:01,927 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 202318 == 0. Received: {'172.23.121.127': 202318} for bucket 'GleamBookUsers0' 2022-02-07 22:37:01,927 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:01,928 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:01,930 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:02,026 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 168733 == 0 2022-02-07 22:37:02,026 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 168733 == 0. Received: {'172.23.121.123': 168733} for bucket 'GleamBookUsers0' 2022-02-07 22:37:02,026 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:02,029 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:02,029 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:02,076 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 686428 == 0 2022-02-07 22:37:02,076 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 686428 == 0. Received: {'172.23.120.170': 686428} for bucket 'GleamBookUsers0' 2022-02-07 22:37:02,076 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:02,078 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:02,078 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:03,105 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:03,384 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:03,885 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 591538 == 0 2022-02-07 22:37:03,885 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 591538 == 0. Received: {'172.23.121.124': 591538} for bucket 'GleamBookUsers0' 2022-02-07 22:37:03,885 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:03,888 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:03,888 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:04,055 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 264409 == 0 2022-02-07 22:37:04,055 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 264409 == 0. Received: {'172.23.121.128': 264409} for bucket 'GleamBookUsers0' 2022-02-07 22:37:04,055 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:04,056 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:04,056 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:05,444 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:06,092 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 210320 == 0 2022-02-07 22:37:06,092 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 210320 == 0. Received: {'172.23.121.126': 210320} for bucket 'GleamBookUsers0' 2022-02-07 22:37:06,092 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:06,095 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:06,095 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:06,467 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:06,930 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:07,030 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:07,079 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:07,536 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 574833 == 0 2022-02-07 22:37:07,536 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 574833 == 0. Received: {'172.23.121.129': 574833} for bucket 'GleamBookUsers0' 2022-02-07 22:37:07,536 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:07,539 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:07,539 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:07,543 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 178990 == 0 2022-02-07 22:37:07,543 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 178990 == 0. Received: {'172.23.121.127': 178990} for bucket 'GleamBookUsers0' 2022-02-07 22:37:07,545 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:07,546 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:07,546 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:07,614 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 147449 == 0 2022-02-07 22:37:07,615 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 147449 == 0. Received: {'172.23.121.123': 147449} for bucket 'GleamBookUsers0' 2022-02-07 22:37:07,615 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:07,617 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:07,617 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:08,131 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 664102 == 0 2022-02-07 22:37:08,131 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 664102 == 0. Received: {'172.23.120.170': 664102} for bucket 'GleamBookUsers0' 2022-02-07 22:37:08,131 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:08,132 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:08,132 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:08,888 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:09,058 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:09,709 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 251367 == 0 2022-02-07 22:37:09,709 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 251367 == 0. Received: {'172.23.121.128': 251367} for bucket 'GleamBookUsers0' 2022-02-07 22:37:09,709 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:09,711 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:09,711 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:09,723 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 574252 == 0 2022-02-07 22:37:09,723 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 574252 == 0. Received: {'172.23.121.124': 574252} for bucket 'GleamBookUsers0' 2022-02-07 22:37:09,723 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:09,724 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:09,726 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:11,095 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:11,667 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 180972 == 0 2022-02-07 22:37:11,667 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 180972 == 0. Received: {'172.23.121.126': 180972} for bucket 'GleamBookUsers0' 2022-02-07 22:37:11,667 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:11,668 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:11,668 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:12,539 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:12,546 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:12,617 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:13,132 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 169068 == 0 2022-02-07 22:37:13,132 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 169068 == 0. Received: {'172.23.121.127': 169068} for bucket 'GleamBookUsers0' 2022-02-07 22:37:13,132 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:13,134 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:13,135 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:13,135 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:13,176 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 128795 == 0 2022-02-07 22:37:13,176 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 128795 == 0. Received: {'172.23.121.123': 128795} for bucket 'GleamBookUsers0' 2022-02-07 22:37:13,176 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:13,177 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:13,177 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:13,533 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 546605 == 0 2022-02-07 22:37:13,535 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 546605 == 0. Received: {'172.23.121.129': 546605} for bucket 'GleamBookUsers0' 2022-02-07 22:37:13,535 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:13,536 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:13,536 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:14,184 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 653740 == 0 2022-02-07 22:37:14,184 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 653740 == 0. Received: {'172.23.120.170': 653740} for bucket 'GleamBookUsers0' 2022-02-07 22:37:14,184 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:14,187 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:14,187 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:14,711 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:14,726 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:15,236 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 229467 == 0 2022-02-07 22:37:15,236 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 229467 == 0. Received: {'172.23.121.128': 229467} for bucket 'GleamBookUsers0' 2022-02-07 22:37:15,236 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:15,237 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:15,239 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:15,464 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 534780 == 0 2022-02-07 22:37:15,466 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 534780 == 0. Received: {'172.23.121.124': 534780} for bucket 'GleamBookUsers0' 2022-02-07 22:37:15,466 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:15,467 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:15,467 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:16,670 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:17,220 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 161462 == 0 2022-02-07 22:37:17,220 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 161462 == 0. Received: {'172.23.121.126': 161462} for bucket 'GleamBookUsers0' 2022-02-07 22:37:17,220 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:17,223 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:17,223 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:18,135 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:18,178 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:18,538 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:18,742 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 148014 == 0 2022-02-07 22:37:18,742 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 148014 == 0. Received: {'172.23.121.127': 148014} for bucket 'GleamBookUsers0' 2022-02-07 22:37:18,742 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:18,744 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:18,744 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:18,747 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 112983 == 0 2022-02-07 22:37:18,749 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 112983 == 0. Received: {'172.23.121.123': 112983} for bucket 'GleamBookUsers0' 2022-02-07 22:37:18,749 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:18,750 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:18,750 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:19,187 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:19,506 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 507195 == 0 2022-02-07 22:37:19,506 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 507195 == 0. Received: {'172.23.121.129': 507195} for bucket 'GleamBookUsers0' 2022-02-07 22:37:19,506 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:19,507 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:19,507 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:20,108 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 615492 == 0 2022-02-07 22:37:20,108 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 615492 == 0. Received: {'172.23.120.170': 615492} for bucket 'GleamBookUsers0' 2022-02-07 22:37:20,108 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:20,109 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:20,111 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:20,239 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:20,470 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:20,752 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 206298 == 0 2022-02-07 22:37:20,752 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 206298 == 0. Received: {'172.23.121.128': 206298} for bucket 'GleamBookUsers0' 2022-02-07 22:37:20,753 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:20,753 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:20,755 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:21,230 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 518670 == 0 2022-02-07 22:37:21,230 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 518670 == 0. Received: {'172.23.121.124': 518670} for bucket 'GleamBookUsers0' 2022-02-07 22:37:21,232 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:21,233 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:21,233 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:22,223 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:22,752 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 136562 == 0 2022-02-07 22:37:22,753 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 136562 == 0. Received: {'172.23.121.126': 136562} for bucket 'GleamBookUsers0' 2022-02-07 22:37:22,753 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:22,755 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:22,755 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:23,744 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:23,750 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:24,249 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 128549 == 0 2022-02-07 22:37:24,249 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 128549 == 0. Received: {'172.23.121.127': 128549} for bucket 'GleamBookUsers0' 2022-02-07 22:37:24,249 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:24,250 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:24,252 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:24,266 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 95837 == 0 2022-02-07 22:37:24,266 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 95837 == 0. Received: {'172.23.121.123': 95837} for bucket 'GleamBookUsers0' 2022-02-07 22:37:24,266 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:24,269 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:24,269 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:24,509 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:25,111 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:25,444 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 479733 == 0 2022-02-07 22:37:25,446 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 479733 == 0. Received: {'172.23.121.129': 479733} for bucket 'GleamBookUsers0' 2022-02-07 22:37:25,446 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:25,447 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:25,447 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:25,755 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:25,996 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 605689 == 0 2022-02-07 22:37:25,996 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 605689 == 0. Received: {'172.23.120.170': 605689} for bucket 'GleamBookUsers0' 2022-02-07 22:37:25,996 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:25,999 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:25,999 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:26,234 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:26,309 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 182982 == 0 2022-02-07 22:37:26,311 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 182982 == 0. Received: {'172.23.121.128': 182982} for bucket 'GleamBookUsers0' 2022-02-07 22:37:26,311 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:26,313 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:26,313 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:26,934 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 479514 == 0 2022-02-07 22:37:26,934 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 479514 == 0. Received: {'172.23.121.124': 479514} for bucket 'GleamBookUsers0' 2022-02-07 22:37:26,934 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:26,937 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:26,937 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:27,756 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:28,269 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 119641 == 0 2022-02-07 22:37:28,269 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 119641 == 0. Received: {'172.23.121.126': 119641} for bucket 'GleamBookUsers0' 2022-02-07 22:37:28,269 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:28,270 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:28,270 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:29,252 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:29,270 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:29,747 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 112013 == 0 2022-02-07 22:37:29,747 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 112013 == 0. Received: {'172.23.121.127': 112013} for bucket 'GleamBookUsers0' 2022-02-07 22:37:29,747 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:29,750 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:29,750 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:29,789 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 87677 == 0 2022-02-07 22:37:29,789 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 87677 == 0. Received: {'172.23.121.123': 87677} for bucket 'GleamBookUsers0' 2022-02-07 22:37:29,789 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:29,792 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:29,792 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:30,447 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:30,999 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:31,315 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:31,378 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 468621 == 0 2022-02-07 22:37:31,380 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 468621 == 0. Received: {'172.23.121.129': 468621} for bucket 'GleamBookUsers0' 2022-02-07 22:37:31,380 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:31,381 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:31,381 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:31,859 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 163210 == 0 2022-02-07 22:37:31,861 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 163210 == 0. Received: {'172.23.121.128': 163210} for bucket 'GleamBookUsers0' 2022-02-07 22:37:31,861 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:31,862 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:31,862 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:31,937 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:31,940 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 568565 == 0 2022-02-07 22:37:31,940 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 568565 == 0. Received: {'172.23.120.170': 568565} for bucket 'GleamBookUsers0' 2022-02-07 22:37:31,940 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:31,941 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:31,941 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:32,644 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 464158 == 0 2022-02-07 22:37:32,644 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 464158 == 0. Received: {'172.23.121.124': 464158} for bucket 'GleamBookUsers0' 2022-02-07 22:37:32,644 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:32,647 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:32,647 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:33,272 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:33,786 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 113235 == 0 2022-02-07 22:37:33,786 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 113235 == 0. Received: {'172.23.121.126': 113235} for bucket 'GleamBookUsers0' 2022-02-07 22:37:33,786 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:33,788 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:33,789 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:34,750 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:34,793 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:35,262 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 86591 == 0 2022-02-07 22:37:35,263 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 86591 == 0. Received: {'172.23.121.127': 86591} for bucket 'GleamBookUsers0' 2022-02-07 22:37:35,263 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:35,266 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:35,266 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:35,302 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 62335 == 0 2022-02-07 22:37:35,302 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 62335 == 0. Received: {'172.23.121.123': 62335} for bucket 'GleamBookUsers0' 2022-02-07 22:37:35,302 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:35,305 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:35,305 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:36,382 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:36,864 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:36,946 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:37,319 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 441183 == 0 2022-02-07 22:37:37,321 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 441183 == 0. Received: {'172.23.121.129': 441183} for bucket 'GleamBookUsers0' 2022-02-07 22:37:37,321 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:37,322 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:37,322 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:37,382 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 143380 == 0 2022-02-07 22:37:37,382 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 143380 == 0. Received: {'172.23.121.128': 143380} for bucket 'GleamBookUsers0' 2022-02-07 22:37:37,382 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:37,384 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:37,384 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:37,647 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:37,819 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 523236 == 0 2022-02-07 22:37:37,819 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 523236 == 0. Received: {'172.23.120.170': 523236} for bucket 'GleamBookUsers0' 2022-02-07 22:37:37,819 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:37,822 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:37,822 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:38,338 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 426658 == 0 2022-02-07 22:37:38,338 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 426658 == 0. Received: {'172.23.121.124': 426658} for bucket 'GleamBookUsers0' 2022-02-07 22:37:38,338 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:38,339 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:38,339 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:38,789 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:39,286 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 92581 == 0 2022-02-07 22:37:39,286 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 92581 == 0. Received: {'172.23.121.126': 92581} for bucket 'GleamBookUsers0' 2022-02-07 22:37:39,286 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:39,288 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:39,288 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:40,270 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:40,306 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:40,727 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 70821 == 0 2022-02-07 22:37:40,727 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 70821 == 0. Received: {'172.23.121.127': 70821} for bucket 'GleamBookUsers0' 2022-02-07 22:37:40,727 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:40,730 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:40,730 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:40,773 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 42845 == 0 2022-02-07 22:37:40,773 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 42845 == 0. Received: {'172.23.121.123': 42845} for bucket 'GleamBookUsers0' 2022-02-07 22:37:40,773 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:40,778 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:40,778 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:42,323 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:42,385 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:42,822 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:42,845 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 123988 == 0 2022-02-07 22:37:42,846 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 123988 == 0. Received: {'172.23.121.128': 123988} for bucket 'GleamBookUsers0' 2022-02-07 22:37:42,846 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:42,848 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:42,848 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:43,193 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 416691 == 0 2022-02-07 22:37:43,194 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 416691 == 0. Received: {'172.23.121.129': 416691} for bucket 'GleamBookUsers0' 2022-02-07 22:37:43,194 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:43,196 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:43,196 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:43,339 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:43,634 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 498784 == 0 2022-02-07 22:37:43,634 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 498784 == 0. Received: {'172.23.120.170': 498784} for bucket 'GleamBookUsers0' 2022-02-07 22:37:43,634 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:43,635 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:43,635 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:44,023 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 396098 == 0 2022-02-07 22:37:44,023 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 396098 == 0. Received: {'172.23.121.124': 396098} for bucket 'GleamBookUsers0' 2022-02-07 22:37:44,023 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:44,026 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:44,026 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:44,289 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:44,801 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 69909 == 0 2022-02-07 22:37:44,802 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 69909 == 0. Received: {'172.23.121.126': 69909} for bucket 'GleamBookUsers0' 2022-02-07 22:37:44,802 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:44,803 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:44,803 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:45,730 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:45,779 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:46,184 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 59306 == 0 2022-02-07 22:37:46,184 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 59306 == 0. Received: {'172.23.121.127': 59306} for bucket 'GleamBookUsers0' 2022-02-07 22:37:46,184 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:46,187 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:46,187 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:46,239 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 42843 == 0 2022-02-07 22:37:46,240 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 42843 == 0. Received: {'172.23.121.123': 42843} for bucket 'GleamBookUsers0' 2022-02-07 22:37:46,240 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:46,242 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:46,242 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:47,848 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:48,197 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:48,302 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 97492 == 0 2022-02-07 22:37:48,302 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 97492 == 0. Received: {'172.23.121.128': 97492} for bucket 'GleamBookUsers0' 2022-02-07 22:37:48,302 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:48,305 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:48,305 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:48,637 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:48,923 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 368489 == 0 2022-02-07 22:37:48,923 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 368489 == 0. Received: {'172.23.121.129': 368489} for bucket 'GleamBookUsers0' 2022-02-07 22:37:48,923 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:48,924 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:48,926 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:49,026 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:49,404 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 481382 == 0 2022-02-07 22:37:49,404 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 481382 == 0. Received: {'172.23.120.170': 481382} for bucket 'GleamBookUsers0' 2022-02-07 22:37:49,404 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:49,407 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:49,407 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:49,674 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 378002 == 0 2022-02-07 22:37:49,674 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 378002 == 0. Received: {'172.23.121.124': 378002} for bucket 'GleamBookUsers0' 2022-02-07 22:37:49,674 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:49,676 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:49,676 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:49,803 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:50,269 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 50126 == 0 2022-02-07 22:37:50,269 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 50126 == 0. Received: {'172.23.121.126': 50126} for bucket 'GleamBookUsers0' 2022-02-07 22:37:50,269 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:50,272 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:50,272 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:51,187 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:51,243 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:51,625 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 40974 == 0 2022-02-07 22:37:51,627 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 40974 == 0. Received: {'172.23.121.127': 40974} for bucket 'GleamBookUsers0' 2022-02-07 22:37:51,627 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:51,628 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:51,628 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:51,688 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 31631 == 0 2022-02-07 22:37:51,690 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 31631 == 0. Received: {'172.23.121.123': 31631} for bucket 'GleamBookUsers0' 2022-02-07 22:37:51,690 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:51,691 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:51,691 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:53,305 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:53,746 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 76902 == 0 2022-02-07 22:37:53,746 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 76902 == 0. Received: {'172.23.121.128': 76902} for bucket 'GleamBookUsers0' 2022-02-07 22:37:53,747 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:53,750 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:53,750 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:53,926 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:54,407 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:54,648 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 334887 == 0 2022-02-07 22:37:54,650 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 334887 == 0. Received: {'172.23.121.129': 334887} for bucket 'GleamBookUsers0' 2022-02-07 22:37:54,650 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:54,651 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:54,651 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:54,677 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:55,177 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 440646 == 0 2022-02-07 22:37:55,177 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 440646 == 0. Received: {'172.23.120.170': 440646} for bucket 'GleamBookUsers0' 2022-02-07 22:37:55,177 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:55,180 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:55,180 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:55,273 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:55,387 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 348532 == 0 2022-02-07 22:37:55,387 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 348532 == 0. Received: {'172.23.121.124': 348532} for bucket 'GleamBookUsers0' 2022-02-07 22:37:55,388 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:55,390 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:55,391 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:55,710 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 36824 == 0 2022-02-07 22:37:55,710 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 36824 == 0. Received: {'172.23.121.126': 36824} for bucket 'GleamBookUsers0' 2022-02-07 22:37:55,710 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:55,711 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:55,711 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:56,630 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:56,691 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:57,017 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 24792 == 0 2022-02-07 22:37:57,019 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 24792 == 0. Received: {'172.23.121.127': 24792} for bucket 'GleamBookUsers0' 2022-02-07 22:37:57,019 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:57,020 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:57,020 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:57,124 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 21803 == 0 2022-02-07 22:37:57,124 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 21803 == 0. Received: {'172.23.121.123': 21803} for bucket 'GleamBookUsers0' 2022-02-07 22:37:57,124 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:57,125 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:57,127 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:58,750 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:37:59,197 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 68248 == 0 2022-02-07 22:37:59,197 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 68248 == 0. Received: {'172.23.121.128': 68248} for bucket 'GleamBookUsers0' 2022-02-07 22:37:59,197 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:37:59,200 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:37:59,200 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:37:59,651 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:00,180 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:00,326 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 313709 == 0 2022-02-07 22:38:00,328 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 313709 == 0. Received: {'172.23.121.129': 313709} for bucket 'GleamBookUsers0' 2022-02-07 22:38:00,328 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:00,329 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:00,329 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:00,391 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:00,713 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:00,937 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 413212 == 0 2022-02-07 22:38:00,937 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 413212 == 0. Received: {'172.23.120.170': 413212} for bucket 'GleamBookUsers0' 2022-02-07 22:38:00,937 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:00,940 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:00,940 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:01,030 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 305847 == 0 2022-02-07 22:38:01,032 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 305847 == 0. Received: {'172.23.121.124': 305847} for bucket 'GleamBookUsers0' 2022-02-07 22:38:01,032 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:01,033 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:01,033 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:01,131 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 24956 == 0 2022-02-07 22:38:01,131 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 24956 == 0. Received: {'172.23.121.126': 24956} for bucket 'GleamBookUsers0' 2022-02-07 22:38:01,131 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:01,134 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:01,134 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:02,022 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:02,127 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:02,394 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 14740 == 0 2022-02-07 22:38:02,394 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 14740 == 0. Received: {'172.23.121.127': 14740} for bucket 'GleamBookUsers0' 2022-02-07 22:38:02,394 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:02,397 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:02,397 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:02,506 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 3856 == 0 2022-02-07 22:38:02,506 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 3856 == 0. Received: {'172.23.121.123': 3856} for bucket 'GleamBookUsers0' 2022-02-07 22:38:02,506 | infra | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:02,509 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:02,509 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:04,200 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:04,611 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 45709 == 0 2022-02-07 22:38:04,611 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 45709 == 0. Received: {'172.23.121.128': 45709} for bucket 'GleamBookUsers0' 2022-02-07 22:38:04,611 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:04,612 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:04,612 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:05,329 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:05,940 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:05,977 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 280999 == 0 2022-02-07 22:38:05,977 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 280999 == 0. Received: {'172.23.121.129': 280999} for bucket 'GleamBookUsers0' 2022-02-07 22:38:05,977 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:05,980 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:05,980 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:06,033 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:06,134 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:06,631 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 14826 == 0 2022-02-07 22:38:06,631 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 14826 == 0. Received: {'172.23.121.126': 14826} for bucket 'GleamBookUsers0' 2022-02-07 22:38:06,631 | infra | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:06,634 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:06,634 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:06,726 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 382404 == 0 2022-02-07 22:38:06,726 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 382404 == 0. Received: {'172.23.120.170': 382404} for bucket 'GleamBookUsers0' 2022-02-07 22:38:06,726 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:06,727 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:06,727 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:06,743 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 277713 == 0 2022-02-07 22:38:06,743 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 277713 == 0. Received: {'172.23.121.124': 277713} for bucket 'GleamBookUsers0' 2022-02-07 22:38:06,743 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:06,744 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:06,744 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:07,398 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:07,510 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:07,760 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 6960 == 0 2022-02-07 22:38:07,760 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 6960 == 0. Received: {'172.23.121.127': 6960} for bucket 'GleamBookUsers0' 2022-02-07 22:38:07,762 | infra | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:07,763 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:07,763 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:07,865 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 22:38:07,865 | test | DEBUG | pool-3-thread-4 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.123': 0} for bucket 'GleamBookUsers0' 2022-02-07 22:38:07,865 | infra | INFO | pool-3-thread-4 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301823.39' completed 2022-02-07 22:38:09,619 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:10,006 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 23154 == 0 2022-02-07 22:38:10,007 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 23154 == 0. Received: {'172.23.121.128': 23154} for bucket 'GleamBookUsers0' 2022-02-07 22:38:10,007 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:10,009 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:10,009 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:10,980 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:11,576 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 235101 == 0 2022-02-07 22:38:11,578 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 235101 == 0. Received: {'172.23.121.129': 235101} for bucket 'GleamBookUsers0' 2022-02-07 22:38:11,578 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:11,579 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:11,579 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:11,634 | infra | DEBUG | pool-3-thread-6 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:11,729 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:11,746 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:11,984 | test | DEBUG | pool-3-thread-6 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 22:38:11,984 | test | DEBUG | pool-3-thread-6 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.126': 0} for bucket 'GleamBookUsers0' 2022-02-07 22:38:11,984 | infra | INFO | pool-3-thread-6 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301825.1' completed 2022-02-07 22:38:12,391 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 249717 == 0 2022-02-07 22:38:12,391 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 249717 == 0. Received: {'172.23.121.124': 249717} for bucket 'GleamBookUsers0' 2022-02-07 22:38:12,391 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:12,394 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:12,394 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:12,447 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 341857 == 0 2022-02-07 22:38:12,447 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 341857 == 0. Received: {'172.23.120.170': 341857} for bucket 'GleamBookUsers0' 2022-02-07 22:38:12,447 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:12,448 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:12,448 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:12,763 | infra | DEBUG | pool-3-thread-7 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:13,111 | test | DEBUG | pool-3-thread-7 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 22:38:13,111 | test | DEBUG | pool-3-thread-7 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.127': 0} for bucket 'GleamBookUsers0' 2022-02-07 22:38:13,112 | infra | INFO | pool-3-thread-7 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301825.6' completed 2022-02-07 22:38:15,010 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:15,384 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 10720 == 0 2022-02-07 22:38:15,385 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 10720 == 0. Received: {'172.23.121.128': 10720} for bucket 'GleamBookUsers0' 2022-02-07 22:38:15,385 | infra | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:15,388 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:15,388 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:16,579 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:17,137 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 180561 == 0 2022-02-07 22:38:17,138 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 180561 == 0. Received: {'172.23.121.129': 180561} for bucket 'GleamBookUsers0' 2022-02-07 22:38:17,138 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:17,140 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:17,140 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:17,394 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:17,450 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:17,983 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 203558 == 0 2022-02-07 22:38:17,983 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 203558 == 0. Received: {'172.23.121.124': 203558} for bucket 'GleamBookUsers0' 2022-02-07 22:38:17,983 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:17,986 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:17,986 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:18,105 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 297937 == 0 2022-02-07 22:38:18,105 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 297937 == 0. Received: {'172.23.120.170': 297937} for bucket 'GleamBookUsers0' 2022-02-07 22:38:18,105 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:18,108 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:18,108 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:20,388 | infra | DEBUG | pool-3-thread-8 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:20,737 | test | DEBUG | pool-3-thread-8 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 22:38:20,737 | test | DEBUG | pool-3-thread-8 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.128': 0} for bucket 'GleamBookUsers0' 2022-02-07 22:38:20,737 | infra | INFO | pool-3-thread-8 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301826.07' completed 2022-02-07 22:38:22,140 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:22,694 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 144929 == 0 2022-02-07 22:38:22,694 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 144929 == 0. Received: {'172.23.121.129': 144929} for bucket 'GleamBookUsers0' 2022-02-07 22:38:22,694 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:22,697 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:22,697 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:22,986 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:23,108 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:23,548 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 170198 == 0 2022-02-07 22:38:23,548 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 170198 == 0. Received: {'172.23.121.124': 170198} for bucket 'GleamBookUsers0' 2022-02-07 22:38:23,549 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:23,552 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:23,552 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:23,667 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 257801 == 0 2022-02-07 22:38:23,667 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 257801 == 0. Received: {'172.23.120.170': 257801} for bucket 'GleamBookUsers0' 2022-02-07 22:38:23,667 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:23,670 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:23,670 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:27,698 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:28,141 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 66776 == 0 2022-02-07 22:38:28,141 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 66776 == 0. Received: {'172.23.121.129': 66776} for bucket 'GleamBookUsers0' 2022-02-07 22:38:28,141 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:28,144 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:28,144 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:28,552 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:28,670 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:29,042 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 100787 == 0 2022-02-07 22:38:29,042 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 100787 == 0. Received: {'172.23.121.124': 100787} for bucket 'GleamBookUsers0' 2022-02-07 22:38:29,042 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:29,045 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:29,045 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:29,180 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 197973 == 0 2022-02-07 22:38:29,180 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 197973 == 0. Received: {'172.23.120.170': 197973} for bucket 'GleamBookUsers0' 2022-02-07 22:38:29,180 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:29,181 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:29,181 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:33,144 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:33,535 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 17428 == 0 2022-02-07 22:38:33,536 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 17428 == 0. Received: {'172.23.121.129': 17428} for bucket 'GleamBookUsers0' 2022-02-07 22:38:33,536 | infra | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:33,538 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:33,538 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:34,046 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:34,181 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:34,490 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 45246 == 0 2022-02-07 22:38:34,490 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 45246 == 0. Received: {'172.23.121.124': 45246} for bucket 'GleamBookUsers0' 2022-02-07 22:38:34,490 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:34,493 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:34,493 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:34,658 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 148015 == 0 2022-02-07 22:38:34,660 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 148015 == 0. Received: {'172.23.120.170': 148015} for bucket 'GleamBookUsers0' 2022-02-07 22:38:34,660 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:34,663 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:34,663 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:38,539 | infra | DEBUG | pool-3-thread-9 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:38,888 | test | DEBUG | pool-3-thread-9 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 22:38:38,888 | test | DEBUG | pool-3-thread-9 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.129': 0} for bucket 'GleamBookUsers0' 2022-02-07 22:38:38,888 | infra | INFO | pool-3-thread-9 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301826.66' completed 2022-02-07 22:38:39,494 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:39,664 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:39,878 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 4044 == 0 2022-02-07 22:38:39,878 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 4044 == 0. Received: {'172.23.121.124': 4044} for bucket 'GleamBookUsers0' 2022-02-07 22:38:39,878 | infra | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:39,881 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:39,881 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:40,091 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 63296 == 0 2022-02-07 22:38:40,091 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 63296 == 0. Received: {'172.23.120.170': 63296} for bucket 'GleamBookUsers0' 2022-02-07 22:38:40,092 | infra | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 22:38:40,092 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 22:38:40,092 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 22:38:44,881 | infra | DEBUG | pool-3-thread-5 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:45,094 | infra | DEBUG | pool-3-thread-1 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 22:38:45,237 | test | DEBUG | pool-3-thread-5 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 22:38:45,237 | test | DEBUG | pool-3-thread-5 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.124': 0} for bucket 'GleamBookUsers0' 2022-02-07 22:38:45,237 | infra | INFO | pool-3-thread-5 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301824.56' completed 2022-02-07 22:38:45,433 | test | DEBUG | pool-3-thread-1 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 22:38:45,433 | test | DEBUG | pool-3-thread-1 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.120.170': 0} for bucket 'GleamBookUsers0' 2022-02-07 22:38:45,433 | infra | INFO | pool-3-thread-1 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301821.31' completed 2022-02-07 22:38:45,436 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 22:38:45,437 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301821.78 2022-02-07 22:38:45,437 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 22:38:45,437 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301822.24 2022-02-07 22:38:45,437 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 22:38:45,437 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301823.39 2022-02-07 22:38:45,438 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 22:38:45,438 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301824.56 2022-02-07 22:38:45,440 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 22:38:45,440 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301825.1 2022-02-07 22:38:45,440 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 22:38:45,440 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301825.6 2022-02-07 22:38:45,440 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 22:38:45,440 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301826.07 2022-02-07 22:38:45,441 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 22:38:45,441 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644301826.66 2022-02-07 22:38:45,441 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 22:38:45,519 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:651] All nodes in cluster: [ip:172.23.121.128 port:8091 ssh_username:root, ip:172.23.121.115 port:8091 ssh_username:root, ip:172.23.120.170 port:8091 ssh_username:root, ip:172.23.121.129 port:8091 ssh_username:root, ip:172.23.121.124 port:8091 ssh_username:root, ip:172.23.121.126 port:8091 ssh_username:root, ip:172.23.121.116 port:8091 ssh_username:root, ip:172.23.121.127 port:8091 ssh_username:root, ip:172.23.121.123 port:8091 ssh_username:root] 2022-02-07 22:38:45,522 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 22:38:45,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:45,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 22:38:45,865 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:45,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 22:38:45,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 22:38:45,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 22:38:45,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 22:38:45,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 22:38:45,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 22:38:45,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 22:38:45,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 22:38:46,157 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 22:38:46,160 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 22:38:46,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:46,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 22:38:46,507 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:46,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 22:38:46,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 22:38:46,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 22:38:46,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 22:38:46,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 22:38:46,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 22:38:46,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 22:38:46,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 22:38:46,790 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 22:38:46,793 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 22:38:46,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:47,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 22:38:47,134 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:47,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 22:38:47,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 22:38:47,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 22:38:47,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 22:38:47,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 22:38:47,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 22:38:47,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 22:38:47,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 22:38:47,421 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 22:38:47,426 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 22:38:47,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:47,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 22:38:47,805 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:47,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 22:38:47,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 22:38:47,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 22:38:47,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 22:38:47,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 22:38:47,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 22:38:47,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 22:38:47,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 22:38:48,088 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 22:38:48,092 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 22:38:48,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:48,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 22:38:48,453 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:48,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 22:38:48,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 22:38:48,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 22:38:48,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 22:38:48,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 22:38:48,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 22:38:48,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 22:38:48,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 22:38:48,750 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 22:38:48,753 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 22:38:48,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:49,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 22:38:49,124 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:49,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 22:38:49,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 22:38:49,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 22:38:49,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 22:38:49,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 22:38:49,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 22:38:49,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 22:38:49,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 22:38:49,411 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 22:38:49,413 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 22:38:49,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:49,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 22:38:49,776 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:49,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 22:38:49,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 22:38:49,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 22:38:49,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 22:38:49,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 22:38:49,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 22:38:49,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 22:38:49,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 22:38:50,055 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 22:38:50,058 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 22:38:50,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:50,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 22:38:50,382 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:50,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 22:38:50,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 22:38:50,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 22:38:50,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 22:38:50,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 22:38:50,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 22:38:50,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 22:38:50,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 22:38:50,661 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 22:38:50,663 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 22:38:50,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:50,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 22:38:51,002 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:51,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 22:38:51,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 22:38:51,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 22:38:51,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 22:38:51,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 22:38:51,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 22:38:51,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 22:38:51,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 22:38:51,282 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 22:38:51,282 | test | DEBUG | MainThread | [bucket_ready_functions:verify_stats_for_bucket:2318] Verifying stats for bucket GleamBookUsers0 2022-02-07 22:38:51,348 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:651] All nodes in cluster: [ip:172.23.121.128 port:8091 ssh_username:root, ip:172.23.121.115 port:8091 ssh_username:root, ip:172.23.120.170 port:8091 ssh_username:root, ip:172.23.121.129 port:8091 ssh_username:root, ip:172.23.121.124 port:8091 ssh_username:root, ip:172.23.121.126 port:8091 ssh_username:root, ip:172.23.121.116 port:8091 ssh_username:root, ip:172.23.121.127 port:8091 ssh_username:root, ip:172.23.121.123 port:8091 ssh_username:root] 2022-02-07 22:38:51,351 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 22:38:51,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:51,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 22:38:51,694 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:51,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 22:38:51,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 22:38:51,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 22:38:51,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 22:38:51,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 22:38:51,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 22:38:51,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 22:38:51,782 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 22:38:51,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:52,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 22:38:52,142 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:52,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 22:38:52,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 22:38:52,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 22:38:52,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 22:38:52,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 22:38:52,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 22:38:52,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 22:38:52,233 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 22:38:52,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:52,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 22:38:52,569 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:52,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 22:38:52,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 22:38:52,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 22:38:52,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 22:38:52,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 22:38:52,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 22:38:52,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 22:38:52,663 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 22:38:52,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:52,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 22:38:53,003 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:53,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 22:38:53,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 22:38:53,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 22:38:53,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 22:38:53,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 22:38:53,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 22:38:53,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 22:38:53,095 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 22:38:53,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:53,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 22:38:53,414 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:53,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 22:38:53,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 22:38:53,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 22:38:53,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 22:38:53,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 22:38:53,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 22:38:53,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 22:38:53,526 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 22:38:53,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:53,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 22:38:53,851 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:53,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 22:38:53,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 22:38:53,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 22:38:53,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 22:38:53,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 22:38:53,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 22:38:53,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 22:38:53,937 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 22:38:54,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:54,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 22:38:54,288 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:54,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 22:38:54,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 22:38:54,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 22:38:54,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 22:38:54,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 22:38:54,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 22:38:54,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 22:38:54,380 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 22:38:54,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:54,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 22:38:54,707 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:54,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 22:38:54,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 22:38:54,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 22:38:54,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 22:38:54,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 22:38:54,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 22:38:54,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 22:38:54,799 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 22:38:54,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:55,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 22:38:55,157 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:55,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 22:38:55,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 22:38:55,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 22:38:55,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 22:38:55,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 22:38:55,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 22:38:55,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 22:38:55,265 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 22:38:55,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:55,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 22:38:55,594 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:55,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 22:38:55,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 22:38:55,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 22:38:55,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 22:38:55,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 22:38:55,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 22:38:55,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 22:38:55,700 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 22:38:55,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:56,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 22:38:56,104 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:56,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 22:38:56,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 22:38:56,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 22:38:56,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 22:38:56,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 22:38:56,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 22:38:56,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 22:38:56,194 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 22:38:56,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:56,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 22:38:56,542 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:56,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 22:38:56,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 22:38:56,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 22:38:56,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 22:38:56,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 22:38:56,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 22:38:56,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 22:38:56,664 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 22:38:56,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:56,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 22:38:56,983 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:56,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 22:38:56,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 22:38:57,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 22:38:57,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 22:38:57,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 22:38:57,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 22:38:57,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 22:38:57,075 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 22:38:57,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:57,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 22:38:57,384 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:57,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 22:38:57,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 22:38:57,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 22:38:57,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 22:38:57,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 22:38:57,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 22:38:57,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 22:38:57,470 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 22:38:57,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:57,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 22:38:57,789 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:57,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 22:38:57,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 22:38:57,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 22:38:57,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 22:38:57,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 22:38:57,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 22:38:57,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 22:38:57,887 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 22:38:57,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:58,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 22:38:58,204 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:58,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 22:38:58,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 22:38:58,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 22:38:58,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 22:38:58,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 22:38:58,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 22:38:58,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 22:38:58,296 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 22:38:58,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:58,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 22:38:58,647 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:58,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 22:38:58,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 22:38:58,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 22:38:58,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 22:38:58,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 22:38:58,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 22:38:58,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 22:38:58,736 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 22:38:58,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:38:59,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 22:38:59,062 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:38:59,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 22:38:59,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 22:38:59,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 22:38:59,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 22:38:59,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 22:38:59,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 22:38:59,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 22:38:59,150 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for vb_replica_curr_items on bucket GleamBookUsers0 2022-02-07 22:38:59,150 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_vb_replica_curr_items_1644302339.15 2022-02-07 22:38:59,151 | infra | INFO | pool-3-thread-10 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_vb_replica_curr_items_1644302339.15' started 2022-02-07 22:38:59,151 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for curr_items_tot on bucket GleamBookUsers0 2022-02-07 22:38:59,153 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_curr_items_tot_1644302339.15 2022-02-07 22:38:59,153 | infra | DEBUG | pool-3-thread-10 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:38:59,153 | infra | INFO | pool-3-thread-11 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_curr_items_tot_1644302339.15' started 2022-02-07 22:38:59,153 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_vb_replica_curr_items_1644302339.15 2022-02-07 22:38:59,154 | infra | DEBUG | pool-3-thread-11 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:38:59,295 | infra | DEBUG | pool-3-thread-10 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:38:59,296 | infra | DEBUG | pool-3-thread-11 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:38:59,486 | infra | DEBUG | pool-3-thread-11 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:38:59,489 | infra | DEBUG | pool-3-thread-10 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:38:59,627 | infra | DEBUG | pool-3-thread-11 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:38:59,628 | infra | DEBUG | pool-3-thread-10 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:38:59,767 | infra | DEBUG | pool-3-thread-10 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:38:59,773 | infra | DEBUG | pool-3-thread-11 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:38:59,914 | infra | DEBUG | pool-3-thread-11 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:38:59,917 | infra | DEBUG | pool-3-thread-10 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:39:00,055 | infra | DEBUG | pool-3-thread-10 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:39:00,059 | infra | DEBUG | pool-3-thread-11 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:39:00,191 | infra | DEBUG | pool-3-thread-10 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:39:00,194 | infra | DEBUG | pool-3-thread-11 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:39:00,331 | infra | DEBUG | pool-3-thread-10 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:39:00,331 | infra | DEBUG | pool-3-thread-11 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 22:39:00,467 | test | DEBUG | pool-3-thread-10 | [task:_compare:2755] Comparing 20000000 == 20000000 2022-02-07 22:39:00,467 | test | DEBUG | pool-3-thread-10 | [task:_get_all_stats_and_compare:2707] Ready: vb_replica_curr_items 20000000 == 20000000. Received: {'172.23.121.123': '2222460', '172.23.121.124': '2235940', '172.23.121.115': '2221300', '172.23.121.126': '2220600', '172.23.121.116': '2226240', '172.23.121.127': '2245920', '172.23.121.128': '2197500', '172.23.121.129': '2203020', '172.23.120.170': '2227020'} for bucket 'GleamBookUsers0' 2022-02-07 22:39:00,467 | infra | INFO | pool-3-thread-10 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_vb_replica_curr_items_1644302339.15' completed 2022-02-07 22:39:00,467 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_curr_items_tot_1644302339.15 2022-02-07 22:39:00,469 | test | DEBUG | pool-3-thread-11 | [task:_compare:2755] Comparing 40000000 == 40000000 2022-02-07 22:39:00,469 | test | DEBUG | pool-3-thread-11 | [task:_get_all_stats_and_compare:2707] Ready: curr_items_tot 40000000 == 40000000. Received: {'172.23.121.123': '4462920', '172.23.121.124': '4452900', '172.23.121.115': '4445440', '172.23.121.126': '4441460', '172.23.121.116': '4461840', '172.23.121.127': '4470780', '172.23.121.128': '4400860', '172.23.121.129': '4405480', '172.23.120.170': '4458320'} for bucket 'GleamBookUsers0' 2022-02-07 22:39:00,469 | infra | INFO | pool-3-thread-11 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_curr_items_tot_1644302339.15' completed 2022-02-07 22:39:00,470 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 22:39:00,470 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 22:39:00,470 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 22:39:00,470 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 22:39:00,470 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 22:39:00,470 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 22:39:00,470 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 22:39:00,470 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 22:39:00,470 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 22:39:00,470 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 22:39:00,470 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 22:39:00,471 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 22:39:00,471 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 22:39:00,471 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 22:39:00,471 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 22:39:00,471 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 22:39:00,473 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 22:39:00,473 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 22:39:00,533 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3970] Stats: {u'opsPerSec': 0, u'diskFetches': 0, u'dataUsed': 32020543632L, u'itemCount': 20000000, u'quotaPercentUsed': 58.98535623184274, u'storageTotals': {u'hdd': {u'usedByData': 55418701953L, u'used': 150068618150L, u'quotaTotal': 1168799956992L, u'total': 1168799956992L, u'free': 892306910457L}, u'ram': {u'usedByData': 45929658512L, u'used': 78925766656L, u'quotaTotal': 78809923584L, u'total': 110647066624L, u'quotaUsed': 77866205184L, u'quotaUsedPerNode': 8651800576L, u'quotaTotalPerNode': 8756658176L}}, u'diskUsed': 55418701953L, u'vbActiveNumNonResident': 0, u'memUsed': 45929658512L} 2022-02-07 22:39:00,812 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+-----+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 20000000 | 72.52 GiB | 42.78 GiB | 51.61 GiB | 100 | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+-----+ 2022-02-07 22:39:00,813 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+---------------+---------------+---------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+---------------+---------------+---------------+---------------+ | 20000000 | 20000000 | 0-1000000 | 0-0 | 0-0 | 0-0 | +---------------+---------------+---------------+---------------+---------------+---------------+ 2022-02-07 22:39:01,059 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 100 2022-02-07 22:39:01,062 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 22:39:01,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:01,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 22:39:01,441 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:01,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 22:39:01,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 22:39:01,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 22:39:01,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 22:39:01,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 22:39:01,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 22:39:01,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 22:39:01,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 22:39:01,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 22:39:01,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 22:39:01,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 22:39:01,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 22:39:01,859 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 22:39:01,862 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 22:39:01,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:02,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 22:39:02,203 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:02,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 22:39:02,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 22:39:02,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 22:39:02,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 22:39:02,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 22:39:02,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 22:39:02,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 22:39:02,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 22:39:02,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 22:39:02,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 22:39:02,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 22:39:02,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 22:39:02,655 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 22:39:02,658 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 22:39:02,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:02,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 22:39:02,992 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:02,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 22:39:03,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 22:39:03,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 22:39:03,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 22:39:03,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 22:39:03,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 22:39:03,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 22:39:03,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 22:39:03,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 22:39:03,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 22:39:03,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 22:39:03,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 22:39:03,523 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 22:39:03,525 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 22:39:03,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:03,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 22:39:03,868 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:03,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 22:39:03,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 22:39:03,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 22:39:03,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 22:39:03,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 22:39:03,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 22:39:03,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 22:39:03,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 22:39:04,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 22:39:04,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 22:39:04,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 22:39:04,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 22:39:04,311 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 22:39:04,312 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 22:39:04,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:04,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 22:39:04,668 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:04,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 22:39:04,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 22:39:04,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 22:39:04,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 22:39:04,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 22:39:04,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 22:39:04,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 22:39:04,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 22:39:04,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 22:39:04,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 22:39:04,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 22:39:04,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 22:39:05,122 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 22:39:05,125 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 22:39:05,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:05,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 22:39:05,467 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:05,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 22:39:05,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 22:39:05,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 22:39:05,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 22:39:05,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 22:39:05,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 22:39:05,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 22:39:05,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 22:39:05,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 22:39:05,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 22:39:05,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 22:39:05,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 22:39:05,923 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 22:39:05,926 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 22:39:06,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:06,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 22:39:06,267 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:06,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 22:39:06,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 22:39:06,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 22:39:06,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 22:39:06,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 22:39:06,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 22:39:06,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 22:39:06,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 22:39:06,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 22:39:06,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 22:39:06,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 22:39:06,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 22:39:06,703 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 22:39:06,706 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 22:39:06,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:06,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 22:39:07,026 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:07,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 22:39:07,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 22:39:07,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 22:39:07,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 22:39:07,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 22:39:07,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 22:39:07,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 22:39:07,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 22:39:07,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 22:39:07,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 22:39:07,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 22:39:07,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 22:39:07,451 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 22:39:07,453 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 22:39:07,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:07,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 22:39:07,795 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:07,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 22:39:07,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 22:39:07,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 22:39:07,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 22:39:07,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 22:39:07,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 22:39:07,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 22:39:07,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 22:39:08,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 22:39:08,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 22:39:08,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 22:39:08,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 22:39:08,230 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 22:39:08,233 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-07 22:39:08,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:08,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-07 22:39:08,612 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:08,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-07 22:39:08,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-07 22:39:08,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-07 22:39:08,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-07 22:39:08,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-07 22:39:08,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-07 22:39:08,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-07 22:39:08,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 22:39:08,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 22:39:08,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 22:39:08,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 22:39:08,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 22:39:08,815 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-07 22:39:08,822 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-07 22:39:08,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:09,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-07 22:39:09,181 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:09,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-07 22:39:09,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-07 22:39:09,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-07 22:39:09,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-07 22:39:09,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-07 22:39:09,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-07 22:39:09,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-07 22:39:09,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 22:39:09,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 22:39:09,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 22:39:09,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 22:39:09,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 22:39:09,362 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-07 22:39:09,365 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-07 22:39:09,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:09,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-07 22:39:09,710 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:09,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-07 22:39:09,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-07 22:39:09,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-07 22:39:09,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-07 22:39:09,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-07 22:39:09,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-07 22:39:09,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-07 22:39:09,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 22:39:09,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 22:39:09,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 22:39:09,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 22:39:09,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 22:39:09,904 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-07 22:39:09,907 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-07 22:39:09,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:10,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-07 22:39:10,253 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:10,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-07 22:39:10,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-07 22:39:10,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-07 22:39:10,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-07 22:39:10,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-07 22:39:10,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-07 22:39:10,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-07 22:39:10,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 22:39:10,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 22:39:10,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 22:39:10,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 22:39:10,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 22:39:10,433 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-07 22:39:10,434 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-07 22:39:10,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:10,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-07 22:39:10,790 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:10,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-07 22:39:10,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-07 22:39:10,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-07 22:39:10,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-07 22:39:10,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-07 22:39:10,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-07 22:39:10,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-07 22:39:10,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 22:39:10,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 22:39:10,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 22:39:10,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 22:39:10,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 22:39:11,002 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-07 22:39:11,002 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for kvstore is 49273MB 2022-02-07 22:39:11,003 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for wal is 4259MB 2022-02-07 22:39:11,003 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:642] Total Disk usage for keyTree is 4008MB 2022-02-07 22:39:11,003 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:643] Total Disk usage for seqTree is 45119MB 2022-02-07 22:39:11,006 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 22:39:11,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:11,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 22:39:11,354 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:11,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 22:39:11,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 22:39:11,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 22:39:11,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 22:39:11,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 22:39:11,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 22:39:11,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 22:39:11,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 22:39:11,463 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.115 - core(s): 8 2022-02-07 22:39:11,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 22:39:11,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 22:39:11,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 22:39:11,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 22:39:12,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 22:39:12,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 22:39:12,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 22:39:12,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 22:39:12,528 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 22:39:12,530 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 22:39:12,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:12,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 22:39:12,898 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:12,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 22:39:12,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 22:39:12,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 22:39:12,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 22:39:12,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 22:39:12,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 22:39:12,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 22:39:12,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 22:39:13,009 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.116 - core(s): 8 2022-02-07 22:39:13,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 22:39:13,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 22:39:13,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 22:39:13,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 22:39:13,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 22:39:13,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 22:39:13,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 22:39:13,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 22:39:14,088 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 22:39:14,091 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 22:39:14,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:14,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 22:39:14,450 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:14,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 22:39:14,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 22:39:14,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 22:39:14,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 22:39:14,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 22:39:14,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 22:39:14,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 22:39:14,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 22:39:14,559 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.123 - core(s): 8 2022-02-07 22:39:14,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 22:39:14,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 22:39:14,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 22:39:14,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 22:39:15,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 22:39:15,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 22:39:15,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 22:39:15,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 22:39:15,630 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 22:39:15,631 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 22:39:15,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:15,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 22:39:15,979 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:15,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 22:39:15,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 22:39:16,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 22:39:16,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 22:39:16,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 22:39:16,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 22:39:16,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 22:39:16,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 22:39:16,137 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.124 - core(s): 8 2022-02-07 22:39:16,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 22:39:16,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 22:39:16,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 22:39:16,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 22:39:16,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 22:39:16,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 22:39:16,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 22:39:17,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 22:39:17,197 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 22:39:17,200 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 22:39:17,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:17,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 22:39:17,536 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:17,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 22:39:17,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 22:39:17,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 22:39:17,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 22:39:17,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 22:39:17,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 22:39:17,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 22:39:17,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 22:39:17,640 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.126 - core(s): 8 2022-02-07 22:39:17,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 22:39:17,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 22:39:17,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 22:39:18,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 22:39:18,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 22:39:18,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 22:39:18,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 22:39:18,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 22:39:18,698 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 22:39:18,700 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 22:39:18,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:18,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 22:39:19,042 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:19,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 22:39:19,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 22:39:19,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 22:39:19,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 22:39:19,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 22:39:19,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 22:39:19,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 22:39:19,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 22:39:19,167 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.127 - core(s): 8 2022-02-07 22:39:19,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 22:39:19,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 22:39:19,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 22:39:19,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 22:39:19,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 22:39:19,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 22:39:19,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 22:39:20,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 22:39:20,220 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 22:39:20,223 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 22:39:20,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:20,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 22:39:20,562 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:20,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 22:39:20,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 22:39:20,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 22:39:20,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 22:39:20,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 22:39:20,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 22:39:20,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 22:39:20,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 22:39:20,665 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.128 - core(s): 8 2022-02-07 22:39:20,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 22:39:20,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 22:39:20,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 22:39:21,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 22:39:21,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 22:39:21,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 22:39:21,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 22:39:21,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 22:39:21,721 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 22:39:21,723 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 22:39:21,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:21,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 22:39:22,051 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:22,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 22:39:22,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 22:39:22,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 22:39:22,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 22:39:22,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 22:39:22,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 22:39:22,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 22:39:22,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 22:39:22,171 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.129 - core(s): 8 2022-02-07 22:39:22,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 22:39:22,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 22:39:22,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 22:39:22,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 22:39:22,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 22:39:22,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 22:39:22,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 22:39:23,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 22:39:23,220 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 22:39:23,223 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 22:39:23,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:23,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 22:39:23,555 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:23,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 22:39:23,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 22:39:23,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 22:39:23,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 22:39:23,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 22:39:23,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 22:39:23,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 22:39:23,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 22:39:23,676 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.120.170 - core(s): 8 2022-02-07 22:39:23,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 22:39:23,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 22:39:23,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 22:39:24,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 22:39:24,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 22:39:24,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 22:39:24,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 22:39:24,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 22:39:24,716 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 22:39:24,717 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:723] {'172.23.121.115': {'rw_0:magma': {u'NReadBytesGetPerSec': 0.0, u'ReplayWALOffset': u'125:8249344', u'NTablesCreated': 3553, u'RecentReadIOAmpSet': 0.0, u'ReadIOAmp': 233.4444580078125, u'BlockCacheMisses': 41538, u'NFlushes': 1666, u'RecentWriteAmp': 0.0, u'NumSyncFlushes': 0, u'TotalMemUsed': 88325686, u'NBloomFilterHits': 13159288, u'Fragmentation': 0.37798023223876953, u'NumNormalFlushes': 392, u'RecentBlockCacheHitRatio': 0.0, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 1885, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 24360, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 7314228, u'CheckpointOverhead': 4734976, u'NItemsGet': 0, u'NWriteIOs': 5544, u'NReadBytes': 661450397, u'LevelStats': [{u'TargetSize': 121634816, u'LogicalSize': 50069406, u'SizeRatio': 0.411637125344112, u'NWriteBytes': 27762688, u'BloomFilterMemUsed': 0, u'NumTables': 19, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 1331200}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 86887851, u'SizeRatio': 0.7143337233313198, u'NWriteBytes': 25174016, u'BloomFilterMemUsed': 0, u'NumTables': 57, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 1622016}, {u'TargetSize': 415823726, u'LogicalSize': 325317999, u'SizeRatio': 0.7823459284764332, u'NWriteBytes': 19034112, u'BloomFilterMemUsed': 0, u'NumTables': 155, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 5636096}, {u'TargetSize': 1655351476, u'LogicalSize': 831647460, u'SizeRatio': 0.5023993224747637, u'NWriteBytes': 1461817344, u'BloomFilterMemUsed': 0, u'NumTables': 250, u'WastedSpace': 3971722, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 831623168}], u'NItemsGarbageCollected': 674078, u'NInternalFragmentationCompacts': 6, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 6957829, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 50976, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 245244, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 149742, u'CompactWriter': {u'NDataBlockWrites': 103950, u'NWriteBytes': 312131584, u'NIndexBlockWrites': 1556, u'NBytesIncoming': 484083110, u'NSyncs': 0, u'NInserts': 1709573, u'NWriteIOs': 1306}, u'NonResidentBloomFilterSize': 0, u'IteratorMemUsed': 0, u'NWriteBytes': 1510583160, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 674078, u'LogicalDiskSize': 831647460, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.707311811116749, u'BufferMemUsed': 407552, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 506329461, u'NTableFiles': 532, u'NFileCountCompacts': 0, u'TableObjectMemUsed': 128160, u'TreeSnapshotMemoryUsed': 173088, u'TotalDiskUsage': 885518336, u'BytesIncoming': 2135667942, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 615, u'NItemsPut': 3139274, u'NSyncs': 1274, u'CompactReader': {u'NReadIOs': 245244, u'NReadBytes': 661450397, u'NDataBlockReads': 242563, u'NBytesOutgoing': 0, u'NIndexBlockReads': 2681}, u'ActiveDiskUsage': 840212480, u'NTablesDeleted': 924, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 461273, u'NWriteBytes': 1198451576, u'NIndexBlockWrites': 4866, u'NBytesIncoming': 2170577882L, u'NSyncs': 1274, u'NInserts': 3441841, u'NWriteIOs': 4238}}, u'NReadBytes': 223204703551L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 0.0, u'NInternalFragmentationCompacts': 6, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 8267425, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 11122338, u'NTxns': 56940, u'NWriteBytes': 2342006784L, u'NSegments': 125, u'NSyncs': 56940, u'NLoggedBytes': 2342006784L, u'NActiveSegments': 1, u'NWrites': 69529, u'PreallocateSupported': True, u'NInserts': 2667056, u'DiskUsed': 8249344}, u'NGets': 275020, u'NReadBytesSet': 105165591, u'NReadBytesCompact': 684502232, u'IndexResidentRatio': 2.1768200397491455, u'CompactQueueSize': 0, u'NSyncsPerSec': 0.6666666865348816, u'BlockCacheHitRatio': 98.58253479003906, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 222520201319L, u'BloomFilterMemUsedRatio': 0.020475588738918304, u'NonResidentBloomFilterSize': 0, u'NWriteBytes': 4046364778L, u'BytesOutgoing': 0, u'RecentReadIOAmp': 0.0, u'WriteAmp': 1.9001507759094238, u'NBlocksCached': 18371, u'LogicalDataSize': 535356633, u'ActiveBloomFilterMemUsed': 2214384, u'NFileCountCompacts': 62, u'NTableFiles': 1142, u'FlushQueueSize': 8, u'TableObjectMemUsed': 274560, u'BlockCacheQuota': 79966823, u'TotalDiskUsage': 1002328064, u'NPurgerRuns': 29, u'ReadIOAmpSet': 0.0, u'NSyncs': 3822, u'NReadIOsPerSec': 0.0, u'NReadIOsSetPerSec': 0.0, u'NTablesDeleted': 1982, u'BloomFilterFPR': 0.012569929473102093, u'NBloomFilterHitsPerSec': 0.0, u'WriteCacheMemUsed': 73080, u'RecentBloomFilterFPR': 0.0, u'BytesPerRead': 808723.125, u'BytesIncomingPerSec': 0.0, u'NReadIOsGetPerSec': 0.0, u'NumKVStores': 29, u'NGetStats': 15421, u'TotalIndexBlocksSize': 8623824, u'NGetStatsPerSec': 0.8333333134651184, u'NWriteBytesCompact': 342724608, u'NReadIOsSet': 29266, u'ActiveDataSize': 535356633, u'NWriteBytesCompactPerSec': 0.0, u'NBloomFilterMissesPerSec': 0.0, u'NCommitBatches': 56940, u'NGetStatsComputed': 15421, u'NWriteIOs': 79401, u'NReadIOsGet': 64231160, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 19991706, u'LSMTreeObjectMemUsed': 500192, u'NSetsPerSec': 0.0, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.03333333507180214, u'ManagedFileObjectMemUsed': 109536, u'ReadAmpGet': 0.0, u'BlockCacheMemUsed': 78476968, u'NWriterCompacts': 0, u'NReadIOs': 64483154, u'WALBufferMemUsed': 5242880, u'ReadAmp': 0.0, u'keyStats': {u'NItemsTransformed': 671350, u'NTablesCreated': 834, u'ReadIOAmp': 25.20608733871221, u'MemtablesMemUsed': 24360, u'NBloomFilterHits': 13159288, u'TotalIndexBlocksSize': 1294666, u'CheckpointOverhead': 0, u'NItemsGet': 2548240, u'NWriteIOs': 2220, u'NReadBytes': 222542731899L, u'LevelStats': [{u'TargetSize': 121634816, u'LogicalSize': 85891785, u'SizeRatio': 0.7061447357309276, u'NWriteBytes': 155389952, u'BloomFilterMemUsed': 1669976, u'NumTables': 276, u'WastedSpace': 34168, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 85536768}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 28736968, u'SizeRatio': 0.2362561061464507, u'NWriteBytes': 30339072, u'BloomFilterMemUsed': 538624, u'NumTables': 29, u'WastedSpace': 42453, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 28774400}], u'NItemsGarbageCollected': 367200, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 1294666, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 29280, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 64236852, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 302567, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 92851, u'CompactWriter': {u'NDataBlockWrites': 10401, u'NWriteBytes': 30339072, u'NIndexBlockWrites': 167, u'NBytesIncoming': 39303366, u'NSyncs': 0, u'NInserts': 671350, u'NWriteIOs': 74}, u'NonResidentBloomFilterSize': 0, u'IteratorMemUsed': 0, u'NWriteBytes': 186425684, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 64231160, u'NReadBytes': 222520201319L, u'NDataBlockReads': 3287447573L, u'NBytesOutgoing': 0, u'NIndexBlockReads': 6224931882L}, u'NBloomFilterMisses': 167517, u'NItemsMerged': 0, u'LogicalDiskSize': 28736968, u'TotalBloomFilterMemUsed': 2208600, u'WriteAmp': 1.6402406457575354, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 2208600, u'LogicalDataSize': 28736968, u'NTableFiles': 305, u'NFileCountCompacts': 31, u'TableObjectMemUsed': 73200, u'TreeSnapshotMemoryUsed': 156576, u'TotalDiskUsage': 114311168, u'BytesIncoming': 113657520, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 31, u'NItemsPut': 2273220, u'NSyncs': 1274, u'CompactReader': {u'NReadIOs': 5692, u'NReadBytes': 22530580, u'NDataBlockReads': 16470, u'NBytesOutgoing': 0, u'NIndexBlockReads': 580}, u'ActiveDiskUsage': 114311168, u'NTablesDeleted': 529, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.01256992955175678, u'Writer': {u'NDataBlockWrites': 35826, u'NWriteBytes': 156086612, u'NIndexBlockWrites': 1035, u'NBytesIncoming': 134116500, u'NSyncs': 1274, u'NInserts': 2273220, u'NWriteIOs': 2146}}, u'NReadBytesCompactPerSec': 0.0, u'TableMetaMemUsed': 329950, u'NDeletes': 0, u'BlockCacheHitsPerSec': 0.0, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 167517, u'LogicalDiskSize': 860674632, u'NSets': 2273220, u'TotalBloomFilterMemUsed': 2214384, u'BufferMemUsed': 407552, u'RecentReadAmp': 0.0, u'NGetsPerSec': 0.0, u'NInserts': 0, u'NCommitBatchesPerSec': 0.0, u'FragmentationWithOverhead': 0.46588680148124695, u'NCompactsPerSec': 0.0, u'WALMemUsed': 5308496, u'MemoryQuota': 108147512, u'NReadBytesPerSec': 0.0, u'TreeSnapshotMemoryUsed': 447264, u'BytesIncoming': 2129496525, u'NCompacts': 677, u'NumPersistentFlushes': 1274, u'NWriteIOsPerSec': 0.6666666865348816, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 0.0, u'ActiveDiskUsage': 957022208, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 0.8333333134651184, u'NReadBytesSetPerSec': 0.0, u'BlockCacheHits': 2888907, u'NWriteBytesPerSec': 396.8666687011719, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 834, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 24360, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 14930, u'CheckpointOverhead': 0, u'NItemsGet': 642, u'NWriteIOs': 2108, u'NReadBytes': 521255, u'LevelStats': [{u'TargetSize': 121634816, u'LogicalSize': 2578774, u'SizeRatio': 0.02120095285876044, u'NWriteBytes': 6578176, u'BloomFilterMemUsed': 4856, u'NumTables': 276, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2260992}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 290204, u'SizeRatio': 0.00238586294239965, u'NWriteBytes': 253952, u'BloomFilterMemUsed': 928, u'NumTables': 29, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 237568}], u'NItemsGarbageCollected': 219182, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 14930, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 29280, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 1058, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 87357, u'CompactWriter': {u'NDataBlockWrites': 31, u'NWriteBytes': 253952, u'NIndexBlockWrites': 31, u'NBytesIncoming': 68575, u'NSyncs': 0, u'NInserts': 713, u'NWriteIOs': 31}, u'NonResidentBloomFilterSize': 0, u'IteratorMemUsed': 0, u'NWriteBytes': 7349150, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 290204, u'TotalBloomFilterMemUsed': 5784, u'WriteAmp': 0.22706442037420527, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 5784, u'LogicalDataSize': 290204, u'NTableFiles': 305, u'NFileCountCompacts': 31, u'TableObjectMemUsed': 73200, u'TreeSnapshotMemoryUsed': 117600, u'TotalDiskUsage': 2498560, u'BytesIncoming': 32365925, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 31, u'NItemsPut': 223058, u'NSyncs': 1274, u'CompactReader': {u'NReadIOs': 1058, u'NReadBytes': 521255, u'NDataBlockReads': 529, u'NBytesOutgoing': 0, u'NIndexBlockReads': 529}, u'ActiveDiskUsage': 2498560, u'NTablesDeleted': 529, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 803, u'NWriteBytes': 7095198, u'NIndexBlockWrites': 803, u'NBytesIncoming': 652567, u'NSyncs': 1274, u'NInserts': 9507, u'NWriteIOs': 2077}}}}} 2022-02-07 22:39:24,720 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:728] magma stats fragmentation result {'172.23.121.123': [0.32933133840560913, 0.3493538498878479, 0.291084885597229, 0.30036115646362305, 0.3399454653263092, 0.3011471629142761, 0.3362497389316559, 0.33005255460739136], '172.23.121.124': [0.3132867217063904, 0.31672343611717224, 0.33466681838035583, 0.3159733712673187, 0.29349884390830994, 0.2928997874259949, 0.3379680812358856, 0.3205021023750305], '172.23.121.115': [0.37798023223876953, 0.3533810079097748, 0.31484314799308777, 0.3736836910247803, 0.3172468841075897, 0.2990865111351013, 0.32007017731666565, 0.33850693702697754], '172.23.121.126': [0.30866849422454834, 0.36313292384147644, 0.2863989770412445, 0.3538433015346527, 0.2999805808067322, 0.3620122969150543, 0.30452337861061096, 0.33587074279785156], '172.23.121.116': [0.3517601490020752, 0.35124266147613525, 0.3516284227371216, 0.3592567443847656, 0.34516194462776184, 0.35650649666786194, 0.3629343509674072, 0.32373854517936707], '172.23.121.127': [0.2989168167114258, 0.2942350208759308, 0.31354522705078125, 0.3737896680831909, 0.3311765491962433, 0.3634409010410309, 0.32257500290870667, 0.3204660713672638], '172.23.121.128': [0.30795571208000183, 0.3144696056842804, 0.30083638429641724, 0.29420873522758484, 0.2996004521846771, 0.34049510955810547, 0.3486867845058441, 0.3405545651912689], '172.23.121.129': [0.3244149088859558, 0.31810370087623596, 0.29718926548957825, 0.2881484031677246, 0.31235793232917786, 0.3109337091445923, 0.3227093517780304, 0.27960047125816345], '172.23.120.170': [0.31118613481521606, 0.3128248155117035, 0.3167731463909149, 0.2780396342277527, 0.25284644961357117, 0.3051188886165619, 0.3217419683933258, 0.28899097442626953]} 2022-02-07 22:39:24,964 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.115 is 36.0 2022-02-07 22:39:25,220 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.116 is 36.0 2022-02-07 22:39:25,470 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.123 is 36.0 2022-02-07 22:39:25,716 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.124 is 36.0 2022-02-07 22:39:25,982 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.126 is 36.0 2022-02-07 22:39:26,226 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.127 is 36.0 2022-02-07 22:39:26,483 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.128 is 36.0 2022-02-07 22:39:26,726 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.129 is 36.0 2022-02-07 22:39:27,003 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.120.170 is 36.0 2022-02-07 22:39:27,003 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:677] KV stats fragmentation values {'172.23.121.123': 36.0, '172.23.121.124': 36.0, '172.23.121.115': 36.0, '172.23.121.126': 36.0, '172.23.121.116': 36.0, '172.23.121.127': 36.0, '172.23.121.128': 36.0, '172.23.121.129': 36.0, '172.23.120.170': 36.0} 2022-02-07 22:39:27,006 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 22:39:27,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:27,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 22:39:27,345 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:27,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 22:39:27,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 22:39:27,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 22:39:27,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 22:39:27,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 22:39:27,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 22:39:27,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 22:39:27,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 22:39:27,451 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-07 22:39:27,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 22:39:27,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:27,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 22:39:27,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:27,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:27,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 22:39:27,710 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-07 22:39:27,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 22:39:27,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:27,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 22:39:27,749 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-07 22:39:27,750 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 22:39:27,752 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 22:39:27,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:28,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 22:39:28,102 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:28,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 22:39:28,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 22:39:28,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 22:39:28,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 22:39:28,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 22:39:28,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 22:39:28,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 22:39:28,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 22:39:28,203 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-07 22:39:28,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 22:39:28,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:28,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 22:39:28,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:28,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:28,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 22:39:28,450 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-07 22:39:28,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 22:39:28,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:28,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 22:39:28,486 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-07 22:39:28,486 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 22:39:28,489 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 22:39:28,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:28,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 22:39:28,828 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:28,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 22:39:28,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 22:39:28,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 22:39:28,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 22:39:28,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 22:39:28,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 22:39:28,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 22:39:28,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 22:39:28,934 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-07 22:39:28,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 22:39:28,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:29,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 22:39:29,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:29,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:29,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 22:39:29,111 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-07 22:39:29,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 22:39:29,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:29,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 22:39:29,148 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-07 22:39:29,150 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 22:39:29,153 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 22:39:29,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:29,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 22:39:29,500 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:29,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 22:39:29,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 22:39:29,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 22:39:29,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 22:39:29,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 22:39:29,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 22:39:29,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 22:39:29,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 22:39:29,605 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-07 22:39:29,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 22:39:29,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:29,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 22:39:29,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:29,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:29,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 22:39:29,864 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-07 22:39:29,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 22:39:29,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:29,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 22:39:29,910 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-07 22:39:29,910 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 22:39:29,913 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 22:39:29,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:30,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 22:39:30,260 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:30,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 22:39:30,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 22:39:30,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 22:39:30,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 22:39:30,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 22:39:30,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 22:39:30,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 22:39:30,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 22:39:30,384 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-07 22:39:30,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 22:39:30,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:30,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 22:39:30,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:30,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:30,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 22:39:30,730 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-07 22:39:30,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 22:39:30,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:30,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 22:39:30,773 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-07 22:39:30,773 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 22:39:30,776 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 22:39:30,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:31,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 22:39:31,122 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:31,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 22:39:31,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 22:39:31,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 22:39:31,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 22:39:31,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 22:39:31,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 22:39:31,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 22:39:31,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 22:39:31,226 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-07 22:39:31,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 22:39:31,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:31,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 22:39:31,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:31,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:31,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 22:39:31,487 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-07 22:39:31,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 22:39:31,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:31,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 22:39:31,526 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-07 22:39:31,526 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 22:39:31,529 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 22:39:31,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:31,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 22:39:31,851 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:31,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 22:39:31,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 22:39:31,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 22:39:31,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 22:39:31,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 22:39:31,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 22:39:31,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 22:39:31,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 22:39:31,960 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-07 22:39:32,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 22:39:32,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:32,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 22:39:32,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:32,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:32,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 22:39:32,280 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-07 22:39:32,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 22:39:32,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:32,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 22:39:32,321 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-07 22:39:32,321 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 22:39:32,323 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 22:39:32,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:32,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 22:39:32,665 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:32,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 22:39:32,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 22:39:32,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 22:39:32,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 22:39:32,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 22:39:32,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 22:39:32,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 22:39:32,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 22:39:32,785 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-07 22:39:32,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 22:39:32,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:32,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 22:39:32,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:32,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:32,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 22:39:33,003 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-07 22:39:33,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 22:39:33,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:33,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 22:39:33,039 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-07 22:39:33,040 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 22:39:33,042 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 22:39:33,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:33,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 22:39:33,388 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:33,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 22:39:33,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 22:39:33,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 22:39:33,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 22:39:33,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 22:39:33,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 22:39:33,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 22:39:33,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 22:39:33,493 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-07 22:39:33,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 22:39:33,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:33,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 22:39:33,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:33,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:33,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 22:39:33,767 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-07 22:39:33,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 22:39:33,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:33,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 22:39:33,806 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-07 22:39:33,808 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 22:39:33,809 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-07 22:39:33,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:34,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-07 22:39:34,167 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:34,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-07 22:39:34,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-07 22:39:34,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-07 22:39:34,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-07 22:39:34,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-07 22:39:34,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-07 22:39:34,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-07 22:39:34,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 22:39:34,269 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-07 22:39:34,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 22:39:34,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:34,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 22:39:34,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:34,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:34,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 22:39:34,384 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-07 22:39:34,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 22:39:34,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:34,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 22:39:34,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:34,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-07 22:39:34,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:34,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:34,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:34,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:34,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:34,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:34,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:34,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:34,549 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-07 22:39:34,552 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-07 22:39:34,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:34,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-07 22:39:34,898 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:34,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-07 22:39:34,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-07 22:39:34,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-07 22:39:34,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-07 22:39:34,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-07 22:39:34,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-07 22:39:34,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-07 22:39:34,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 22:39:35,003 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-07 22:39:35,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 22:39:35,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:35,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 22:39:35,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:35,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:35,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 22:39:35,118 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-07 22:39:35,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 22:39:35,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:35,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 22:39:35,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:35,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-07 22:39:35,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:35,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:35,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:35,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:35,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:35,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:35,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:35,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:35,283 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-07 22:39:35,286 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-07 22:39:35,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:35,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-07 22:39:35,635 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:35,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-07 22:39:35,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-07 22:39:35,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-07 22:39:35,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-07 22:39:35,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-07 22:39:35,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-07 22:39:35,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-07 22:39:35,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 22:39:35,749 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-07 22:39:35,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 22:39:35,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:35,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 22:39:35,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:35,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:35,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 22:39:35,858 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-07 22:39:35,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 22:39:35,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:35,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 22:39:35,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:35,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-07 22:39:35,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:35,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:35,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:35,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:35,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:35,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:35,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:35,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:36,009 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-07 22:39:36,012 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-07 22:39:36,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:36,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-07 22:39:36,339 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:36,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-07 22:39:36,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-07 22:39:36,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-07 22:39:36,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-07 22:39:36,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-07 22:39:36,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-07 22:39:36,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-07 22:39:36,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 22:39:36,441 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-07 22:39:36,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 22:39:36,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:36,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 22:39:36,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:36,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:36,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 22:39:36,562 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-07 22:39:36,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 22:39:36,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:36,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 22:39:36,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:36,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-07 22:39:36,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:36,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:36,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:36,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:36,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:36,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:36,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:36,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:36,724 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-07 22:39:36,726 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-07 22:39:36,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 22:39:37,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-07 22:39:37,089 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 22:39:37,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-07 22:39:37,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-07 22:39:37,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-07 22:39:37,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-07 22:39:37,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-07 22:39:37,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-07 22:39:37,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-07 22:39:37,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 22:39:37,184 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-07 22:39:37,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 22:39:37,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:37,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 22:39:37,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:37,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 22:39:37,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 22:39:37,289 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-07 22:39:37,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 22:39:37,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 22:39:37,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 22:39:37,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:37,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-07 22:39:37,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:37,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:37,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:37,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:37,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:37,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:37,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:37,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 22:39:37,450 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 ############################################################ # # Step 3: Create 1000000 items sequentially # ############################################################ ############################################################ # # RAM FootPrint: 3446.36328125 # ############################################################ Read Start: 0 Read End: 0 Update Start: 0 Update End: 0 Expiry Start: 0 Expiry End: 0 Delete Start: 0 Delete End: 0 Create Start: 1000000 Create End: 2000000 Final Start: 1000000 Final End: 2000000 2022-02-07 22:39:37,512 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:37,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection9_10_1644302378.51 2022-02-07 22:39:38,520 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:38,522 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection8_9_1644302379.52 2022-02-07 22:39:39,530 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:39,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection19_8_1644302380.53 2022-02-07 22:39:40,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:40,556 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection13_7_1644302381.56 2022-02-07 22:39:41,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:41,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection14_6_1644302382.57 2022-02-07 22:39:42,576 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:42,576 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection11_5_1644302383.58 2022-02-07 22:39:43,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:43,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection12_4_1644302384.59 2022-02-07 22:39:44,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:44,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection17_3_1644302385.6 2022-02-07 22:39:45,607 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:45,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection18_2_1644302386.61 2022-02-07 22:39:46,615 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:46,615 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection15_1_1644302387.62 2022-02-07 22:39:47,625 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:47,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 22:39:48,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:48,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 22:39:49,642 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:49,642 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 22:39:50,655 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:50,655 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 22:39:51,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:51,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 22:39:52,676 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:52,676 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 22:39:53,686 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:53,686 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 22:39:54,696 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:54,696 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 22:39:55,704 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:55,704 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 22:39:56,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 22:39:56,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Loader_GleamBookUsers0__default_VolumeCollection13_7_1644302381.56 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection16_0_1644302388.63 Loader_GleamBookUsers0__default_VolumeCollection9_10_1644302378.51 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection7_-1_1644302389.63 Loader_GleamBookUsers0__default_VolumeCollection11_5_1644302383.58 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection6_-2_1644302390.65 Loader_GleamBookUsers0__default_VolumeCollection8_9_1644302379.52 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection5_-3_1644302391.66 Loader_GleamBookUsers0__default_VolumeCollection15_1_1644302387.62 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection4_-4_1644302392.67 Loader_GleamBookUsers0__default_VolumeCollection14_6_1644302382.57 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection3_-5_1644302393.68 Loader_GleamBookUsers0__default_VolumeCollection18_2_1644302386.61 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection2_-6_1644302394.69 Loader_GleamBookUsers0__default_VolumeCollection17_3_1644302385.6 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection10_-7_1644302395.7 Loader_GleamBookUsers0__default_VolumeCollection19_8_1644302380.53 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection1_-8_1644302396.71 Loader_GleamBookUsers0__default_VolumeCollection12_4_1644302384.59 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection0_-9_1644302397.72 Loader_GleamBookUsers0__default_VolumeCollection16_0_1644302388.63 is completed! Loader_GleamBookUsers0__default_VolumeCollection6_-2_1644302390.65 is completed! Loader_GleamBookUsers0__default_VolumeCollection3_-5_1644302393.68 is completed! Loader_GleamBookUsers0__default_VolumeCollection5_-3_1644302391.66 is completed! Loader_GleamBookUsers0__default_VolumeCollection7_-1_1644302389.63 is completed! Loader_GleamBookUsers0__default_VolumeCollection4_-4_1644302392.67 is completed! Loader_GleamBookUsers0__default_VolumeCollection10_-7_1644302395.7 is completed! Loader_GleamBookUsers0__default_VolumeCollection0_-9_1644302397.72 is completed! Loader_GleamBookUsers0__default_VolumeCollection1_-8_1644302396.71 is completed! Loader_GleamBookUsers0__default_VolumeCollection2_-6_1644302394.69 is completed! 2022-02-07 23:09:20,017 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:651] All nodes in cluster: [ip:172.23.121.128 port:8091 ssh_username:root, ip:172.23.121.115 port:8091 ssh_username:root, ip:172.23.120.170 port:8091 ssh_username:root, ip:172.23.121.129 port:8091 ssh_username:root, ip:172.23.121.124 port:8091 ssh_username:root, ip:172.23.121.126 port:8091 ssh_username:root, ip:172.23.121.116 port:8091 ssh_username:root, ip:172.23.121.127 port:8091 ssh_username:root, ip:172.23.121.123 port:8091 ssh_username:root] 2022-02-07 23:09:20,020 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 23:09:20,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:09:20,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 23:09:20,378 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:09:20,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 23:09:20,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 23:09:20,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 23:09:20,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 23:09:20,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 23:09:20,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 23:09:20,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 23:09:20,473 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 23:09:20,473 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304160.47 2022-02-07 23:09:20,473 | infra | INFO | pool-3-thread-12 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304160.47' started 2022-02-07 23:09:20,476 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:20,479 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 23:09:20,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:09:20,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 23:09:20,851 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:09:20,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 23:09:20,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 23:09:20,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 23:09:20,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 23:09:20,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 23:09:20,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 23:09:20,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 23:09:20,944 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 23:09:20,944 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304160.94 2022-02-07 23:09:20,944 | infra | INFO | pool-3-thread-13 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304160.94' started 2022-02-07 23:09:20,946 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:20,950 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 23:09:21,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:09:21,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 23:09:21,299 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:09:21,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 23:09:21,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 23:09:21,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 23:09:21,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 23:09:21,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 23:09:21,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 23:09:21,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 23:09:21,391 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 23:09:21,391 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304161.39 2022-02-07 23:09:21,392 | infra | INFO | pool-3-thread-14 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304161.39' started 2022-02-07 23:09:21,394 | infra | DEBUG | pool-3-thread-14 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:21,395 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 23:09:21,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:09:21,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 23:09:21,720 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 552182 == 0 2022-02-07 23:09:21,720 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 552182 == 0. Received: {'172.23.120.170': 552182} for bucket 'GleamBookUsers0' 2022-02-07 23:09:21,720 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:21,723 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:21,723 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:21,766 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:09:21,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 23:09:21,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 23:09:21,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 23:09:21,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 23:09:21,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 23:09:21,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 23:09:21,845 | test | DEBUG | pool-3-thread-14 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 23:09:21,845 | test | DEBUG | pool-3-thread-14 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.116': 0} for bucket 'GleamBookUsers0' 2022-02-07 23:09:21,845 | infra | INFO | pool-3-thread-14 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304161.39' completed 2022-02-07 23:09:21,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 23:09:21,861 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 23:09:21,861 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304161.86 2022-02-07 23:09:21,861 | infra | INFO | pool-3-thread-15 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304161.86' started 2022-02-07 23:09:21,862 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:21,865 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 23:09:21,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:09:22,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 23:09:22,229 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:09:22,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 23:09:22,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 23:09:22,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 23:09:22,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 23:09:22,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 23:09:22,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 23:09:22,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 23:09:22,335 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 23:09:22,335 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304162.34 2022-02-07 23:09:22,335 | infra | INFO | pool-3-thread-16 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304162.34' started 2022-02-07 23:09:22,336 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:22,341 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 23:09:22,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:09:22,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 23:09:22,694 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:09:22,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 23:09:22,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 23:09:22,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 23:09:22,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 23:09:22,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 23:09:22,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 23:09:22,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 23:09:22,790 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 23:09:22,790 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304162.79 2022-02-07 23:09:22,792 | infra | INFO | pool-3-thread-17 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304162.79' started 2022-02-07 23:09:22,792 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:22,796 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 23:09:22,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:09:23,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 23:09:23,141 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:09:23,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 23:09:23,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 23:09:23,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 23:09:23,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 23:09:23,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 23:09:23,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 23:09:23,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 23:09:23,239 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 23:09:23,239 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304163.24 2022-02-07 23:09:23,240 | infra | INFO | pool-3-thread-18 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304163.24' started 2022-02-07 23:09:23,240 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:23,244 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 23:09:23,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:09:23,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 23:09:23,576 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:09:23,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 23:09:23,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 23:09:23,598 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 1020176 == 0 2022-02-07 23:09:23,598 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1020176 == 0. Received: {'172.23.121.123': 1020176} for bucket 'GleamBookUsers0' 2022-02-07 23:09:23,598 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:23,601 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:23,601 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:23,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 23:09:23,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 23:09:23,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 23:09:23,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 23:09:23,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 23:09:23,668 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 23:09:23,668 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304163.67 2022-02-07 23:09:23,668 | infra | INFO | pool-3-thread-19 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304163.67' started 2022-02-07 23:09:23,670 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:23,673 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 23:09:23,727 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 2080800 == 0 2022-02-07 23:09:23,729 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2080800 == 0. Received: {'172.23.121.115': 2080800} for bucket 'GleamBookUsers0' 2022-02-07 23:09:23,729 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:23,730 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:23,730 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:23,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:09:23,766 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 629760 == 0 2022-02-07 23:09:23,766 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 629760 == 0. Received: {'172.23.121.124': 629760} for bucket 'GleamBookUsers0' 2022-02-07 23:09:23,766 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:23,770 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:23,770 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:23,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 23:09:24,019 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:09:24,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 23:09:24,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 23:09:24,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 23:09:24,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 23:09:24,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 23:09:24,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 23:09:24,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 23:09:24,111 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-07 23:09:24,111 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304164.11 2022-02-07 23:09:24,111 | infra | INFO | pool-3-thread-22 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304164.11' started 2022-02-07 23:09:24,111 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304160.47 2022-02-07 23:09:24,112 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:24,352 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 952707 == 0 2022-02-07 23:09:24,352 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 952707 == 0. Received: {'172.23.121.126': 952707} for bucket 'GleamBookUsers0' 2022-02-07 23:09:24,352 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:24,355 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:24,355 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:24,614 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 676125 == 0 2022-02-07 23:09:24,615 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 676125 == 0. Received: {'172.23.121.127': 676125} for bucket 'GleamBookUsers0' 2022-02-07 23:09:24,615 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:24,617 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:24,617 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:24,821 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 504583 == 0 2022-02-07 23:09:24,822 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 504583 == 0. Received: {'172.23.121.128': 504583} for bucket 'GleamBookUsers0' 2022-02-07 23:09:24,822 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:24,823 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:24,823 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:25,219 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 455616 == 0 2022-02-07 23:09:25,219 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 455616 == 0. Received: {'172.23.121.129': 455616} for bucket 'GleamBookUsers0' 2022-02-07 23:09:25,220 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:25,221 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:25,221 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:26,723 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:27,904 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 525340 == 0 2022-02-07 23:09:27,904 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 525340 == 0. Received: {'172.23.120.170': 525340} for bucket 'GleamBookUsers0' 2022-02-07 23:09:27,904 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:27,907 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:27,907 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:28,601 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:28,730 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:28,770 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:29,355 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:29,618 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:29,823 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:30,061 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 611060 == 0 2022-02-07 23:09:30,062 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 611060 == 0. Received: {'172.23.121.124': 611060} for bucket 'GleamBookUsers0' 2022-02-07 23:09:30,062 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:30,063 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:30,063 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:30,157 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 983056 == 0 2022-02-07 23:09:30,157 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 983056 == 0. Received: {'172.23.121.123': 983056} for bucket 'GleamBookUsers0' 2022-02-07 23:09:30,157 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:30,158 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:30,158 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:30,221 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:31,289 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 923422 == 0 2022-02-07 23:09:31,289 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 923422 == 0. Received: {'172.23.121.126': 923422} for bucket 'GleamBookUsers0' 2022-02-07 23:09:31,289 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:31,292 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:31,292 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:31,313 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 665971 == 0 2022-02-07 23:09:31,313 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 665971 == 0. Received: {'172.23.121.127': 665971} for bucket 'GleamBookUsers0' 2022-02-07 23:09:31,313 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:31,316 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:31,316 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:31,358 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 484334 == 0 2022-02-07 23:09:31,358 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 484334 == 0. Received: {'172.23.121.128': 484334} for bucket 'GleamBookUsers0' 2022-02-07 23:09:31,358 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:31,361 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:31,361 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:31,601 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 440553 == 0 2022-02-07 23:09:31,602 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 440553 == 0. Received: {'172.23.121.129': 440553} for bucket 'GleamBookUsers0' 2022-02-07 23:09:31,602 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:31,605 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:31,605 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:31,760 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 2071116 == 0 2022-02-07 23:09:31,760 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2071116 == 0. Received: {'172.23.121.115': 2071116} for bucket 'GleamBookUsers0' 2022-02-07 23:09:31,760 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:31,763 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:31,763 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:32,907 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:34,038 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 512506 == 0 2022-02-07 23:09:34,039 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 512506 == 0. Received: {'172.23.120.170': 512506} for bucket 'GleamBookUsers0' 2022-02-07 23:09:34,039 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:34,039 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:34,040 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:35,065 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:35,158 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:36,293 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:36,298 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 586777 == 0 2022-02-07 23:09:36,298 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 586777 == 0. Received: {'172.23.121.124': 586777} for bucket 'GleamBookUsers0' 2022-02-07 23:09:36,298 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:36,299 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:36,299 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:36,316 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:36,361 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:36,605 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:36,690 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 965044 == 0 2022-02-07 23:09:36,690 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 965044 == 0. Received: {'172.23.121.123': 965044} for bucket 'GleamBookUsers0' 2022-02-07 23:09:36,690 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:36,691 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:36,693 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:36,763 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:37,589 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 449332 == 0 2022-02-07 23:09:37,591 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 449332 == 0. Received: {'172.23.121.128': 449332} for bucket 'GleamBookUsers0' 2022-02-07 23:09:37,591 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:37,594 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:37,594 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:37,848 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 646140 == 0 2022-02-07 23:09:37,848 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 646140 == 0. Received: {'172.23.121.127': 646140} for bucket 'GleamBookUsers0' 2022-02-07 23:09:37,848 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:37,851 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:37,851 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:37,892 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 407571 == 0 2022-02-07 23:09:37,892 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 407571 == 0. Received: {'172.23.121.129': 407571} for bucket 'GleamBookUsers0' 2022-02-07 23:09:37,892 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:37,895 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:37,895 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:38,003 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 895396 == 0 2022-02-07 23:09:38,003 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 895396 == 0. Received: {'172.23.121.126': 895396} for bucket 'GleamBookUsers0' 2022-02-07 23:09:38,003 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:38,006 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:38,006 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:39,040 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:39,240 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1952040 == 0 2022-02-07 23:09:39,240 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1952040 == 0. Received: {'172.23.121.115': 1952040} for bucket 'GleamBookUsers0' 2022-02-07 23:09:39,240 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:39,243 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:39,243 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:40,131 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 491765 == 0 2022-02-07 23:09:40,131 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 491765 == 0. Received: {'172.23.120.170': 491765} for bucket 'GleamBookUsers0' 2022-02-07 23:09:40,132 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:40,134 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:40,134 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:41,301 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:41,693 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:42,431 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 571557 == 0 2022-02-07 23:09:42,431 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 571557 == 0. Received: {'172.23.121.124': 571557} for bucket 'GleamBookUsers0' 2022-02-07 23:09:42,433 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:42,434 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:42,434 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:42,594 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:42,851 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:42,895 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:43,006 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:43,073 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 941722 == 0 2022-02-07 23:09:43,073 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 941722 == 0. Received: {'172.23.121.123': 941722} for bucket 'GleamBookUsers0' 2022-02-07 23:09:43,073 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:43,075 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:43,075 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:43,628 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 429050 == 0 2022-02-07 23:09:43,628 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 429050 == 0. Received: {'172.23.121.128': 429050} for bucket 'GleamBookUsers0' 2022-02-07 23:09:43,628 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:43,630 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:43,631 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:43,983 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 390815 == 0 2022-02-07 23:09:43,983 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 390815 == 0. Received: {'172.23.121.129': 390815} for bucket 'GleamBookUsers0' 2022-02-07 23:09:43,983 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:43,986 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:43,986 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:44,206 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 631214 == 0 2022-02-07 23:09:44,206 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 631214 == 0. Received: {'172.23.121.127': 631214} for bucket 'GleamBookUsers0' 2022-02-07 23:09:44,207 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:44,209 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:44,209 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:44,243 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:44,506 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 876627 == 0 2022-02-07 23:09:44,507 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 876627 == 0. Received: {'172.23.121.126': 876627} for bucket 'GleamBookUsers0' 2022-02-07 23:09:44,507 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:44,510 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:44,510 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:45,134 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:46,259 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 468601 == 0 2022-02-07 23:09:46,260 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 468601 == 0. Received: {'172.23.120.170': 468601} for bucket 'GleamBookUsers0' 2022-02-07 23:09:46,260 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:46,263 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:46,263 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:46,759 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1948560 == 0 2022-02-07 23:09:46,759 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1948560 == 0. Received: {'172.23.121.115': 1948560} for bucket 'GleamBookUsers0' 2022-02-07 23:09:46,759 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:46,762 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:46,762 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:47,434 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:48,076 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:48,538 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 548672 == 0 2022-02-07 23:09:48,538 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 548672 == 0. Received: {'172.23.121.124': 548672} for bucket 'GleamBookUsers0' 2022-02-07 23:09:48,539 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:48,539 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:48,540 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:48,631 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:48,986 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:49,210 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:49,512 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:49,516 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 917142 == 0 2022-02-07 23:09:49,516 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 917142 == 0. Received: {'172.23.121.123': 917142} for bucket 'GleamBookUsers0' 2022-02-07 23:09:49,516 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:49,519 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:49,519 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:49,707 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 409824 == 0 2022-02-07 23:09:49,707 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 409824 == 0. Received: {'172.23.121.128': 409824} for bucket 'GleamBookUsers0' 2022-02-07 23:09:49,707 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:49,710 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:49,710 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:50,036 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 369606 == 0 2022-02-07 23:09:50,036 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 369606 == 0. Received: {'172.23.121.129': 369606} for bucket 'GleamBookUsers0' 2022-02-07 23:09:50,036 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:50,039 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:50,039 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:50,440 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 611881 == 0 2022-02-07 23:09:50,440 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 611881 == 0. Received: {'172.23.121.127': 611881} for bucket 'GleamBookUsers0' 2022-02-07 23:09:50,440 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:50,443 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:50,443 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:50,923 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 853321 == 0 2022-02-07 23:09:50,923 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 853321 == 0. Received: {'172.23.121.126': 853321} for bucket 'GleamBookUsers0' 2022-02-07 23:09:50,923 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:50,924 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:50,924 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:51,263 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:51,763 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:52,272 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 456261 == 0 2022-02-07 23:09:52,273 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 456261 == 0. Received: {'172.23.120.170': 456261} for bucket 'GleamBookUsers0' 2022-02-07 23:09:52,273 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:52,275 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:52,275 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:53,540 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:54,153 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1921672 == 0 2022-02-07 23:09:54,153 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1921672 == 0. Received: {'172.23.121.115': 1921672} for bucket 'GleamBookUsers0' 2022-02-07 23:09:54,153 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:54,154 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:54,154 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:54,519 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:54,660 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 534754 == 0 2022-02-07 23:09:54,660 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 534754 == 0. Received: {'172.23.121.124': 534754} for bucket 'GleamBookUsers0' 2022-02-07 23:09:54,660 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:54,661 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:54,661 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:54,713 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:55,039 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:55,444 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:55,785 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 386134 == 0 2022-02-07 23:09:55,785 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 386134 == 0. Received: {'172.23.121.128': 386134} for bucket 'GleamBookUsers0' 2022-02-07 23:09:55,785 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:55,788 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:55,788 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:55,926 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:55,984 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 897648 == 0 2022-02-07 23:09:55,984 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 897648 == 0. Received: {'172.23.121.123': 897648} for bucket 'GleamBookUsers0' 2022-02-07 23:09:55,984 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:55,986 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:55,986 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:56,062 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 346140 == 0 2022-02-07 23:09:56,062 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 346140 == 0. Received: {'172.23.121.129': 346140} for bucket 'GleamBookUsers0' 2022-02-07 23:09:56,062 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:56,065 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:56,065 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:56,663 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 584175 == 0 2022-02-07 23:09:56,663 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 584175 == 0. Received: {'172.23.121.127': 584175} for bucket 'GleamBookUsers0' 2022-02-07 23:09:56,663 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:56,665 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:56,665 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:57,276 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:57,286 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 819426 == 0 2022-02-07 23:09:57,286 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 819426 == 0. Received: {'172.23.121.126': 819426} for bucket 'GleamBookUsers0' 2022-02-07 23:09:57,286 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:57,289 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:57,289 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:58,224 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 423125 == 0 2022-02-07 23:09:58,224 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 423125 == 0. Received: {'172.23.120.170': 423125} for bucket 'GleamBookUsers0' 2022-02-07 23:09:58,224 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:09:58,226 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:09:58,226 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:09:59,155 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:09:59,663 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:00,783 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 520856 == 0 2022-02-07 23:10:00,783 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 520856 == 0. Received: {'172.23.121.124': 520856} for bucket 'GleamBookUsers0' 2022-02-07 23:10:00,785 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:00,786 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:00,786 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:00,789 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:00,987 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:01,065 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:01,543 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1918512 == 0 2022-02-07 23:10:01,543 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1918512 == 0. Received: {'172.23.121.115': 1918512} for bucket 'GleamBookUsers0' 2022-02-07 23:10:01,545 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:01,546 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:01,546 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:01,665 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:01,747 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 363965 == 0 2022-02-07 23:10:01,747 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 363965 == 0. Received: {'172.23.121.128': 363965} for bucket 'GleamBookUsers0' 2022-02-07 23:10:01,747 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:01,750 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:01,750 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:02,006 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 327064 == 0 2022-02-07 23:10:02,006 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 327064 == 0. Received: {'172.23.121.129': 327064} for bucket 'GleamBookUsers0' 2022-02-07 23:10:02,006 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:02,007 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:02,009 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:02,289 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:02,289 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 860024 == 0 2022-02-07 23:10:02,289 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 860024 == 0. Received: {'172.23.121.123': 860024} for bucket 'GleamBookUsers0' 2022-02-07 23:10:02,289 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:02,292 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:02,292 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:02,904 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 569028 == 0 2022-02-07 23:10:02,904 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 569028 == 0. Received: {'172.23.121.127': 569028} for bucket 'GleamBookUsers0' 2022-02-07 23:10:02,904 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:02,907 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:02,907 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:03,227 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:03,654 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 798896 == 0 2022-02-07 23:10:03,654 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 798896 == 0. Received: {'172.23.121.126': 798896} for bucket 'GleamBookUsers0' 2022-02-07 23:10:03,654 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:03,657 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:03,657 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:04,168 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 398891 == 0 2022-02-07 23:10:04,168 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 398891 == 0. Received: {'172.23.120.170': 398891} for bucket 'GleamBookUsers0' 2022-02-07 23:10:04,168 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:04,171 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:04,171 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:05,788 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:06,552 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:06,750 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:06,920 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 508594 == 0 2022-02-07 23:10:06,920 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 508594 == 0. Received: {'172.23.121.124': 508594} for bucket 'GleamBookUsers0' 2022-02-07 23:10:06,920 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:06,921 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:06,921 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:07,009 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:07,292 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:07,651 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 331362 == 0 2022-02-07 23:10:07,651 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 331362 == 0. Received: {'172.23.121.128': 331362} for bucket 'GleamBookUsers0' 2022-02-07 23:10:07,651 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:07,653 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:07,654 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:07,907 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:07,934 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 313734 == 0 2022-02-07 23:10:07,934 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 313734 == 0. Received: {'172.23.121.129': 313734} for bucket 'GleamBookUsers0' 2022-02-07 23:10:07,934 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:07,937 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:07,937 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:08,657 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:08,694 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 837287 == 0 2022-02-07 23:10:08,694 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 837287 == 0. Received: {'172.23.121.123': 837287} for bucket 'GleamBookUsers0' 2022-02-07 23:10:08,694 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:08,696 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:08,697 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:08,927 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1874078 == 0 2022-02-07 23:10:08,927 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1874078 == 0. Received: {'172.23.121.115': 1874078} for bucket 'GleamBookUsers0' 2022-02-07 23:10:08,928 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:08,930 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:08,930 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:09,119 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 542784 == 0 2022-02-07 23:10:09,119 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 542784 == 0. Received: {'172.23.121.127': 542784} for bucket 'GleamBookUsers0' 2022-02-07 23:10:09,119 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:09,121 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:09,121 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:09,171 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:10,058 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 767134 == 0 2022-02-07 23:10:10,059 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 767134 == 0. Received: {'172.23.121.126': 767134} for bucket 'GleamBookUsers0' 2022-02-07 23:10:10,059 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:10,061 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:10,061 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:10,137 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 381513 == 0 2022-02-07 23:10:10,138 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 381513 == 0. Received: {'172.23.120.170': 381513} for bucket 'GleamBookUsers0' 2022-02-07 23:10:10,138 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:10,140 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:10,140 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:11,923 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:12,654 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:12,937 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:12,976 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 485139 == 0 2022-02-07 23:10:12,976 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 485139 == 0. Received: {'172.23.121.124': 485139} for bucket 'GleamBookUsers0' 2022-02-07 23:10:12,976 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:12,977 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:12,977 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:13,500 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 318068 == 0 2022-02-07 23:10:13,500 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 318068 == 0. Received: {'172.23.121.128': 318068} for bucket 'GleamBookUsers0' 2022-02-07 23:10:13,500 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:13,502 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:13,502 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:13,697 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:13,894 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 300981 == 0 2022-02-07 23:10:13,894 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 300981 == 0. Received: {'172.23.121.129': 300981} for bucket 'GleamBookUsers0' 2022-02-07 23:10:13,894 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:13,898 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:13,898 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:13,931 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:14,122 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:15,016 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 808856 == 0 2022-02-07 23:10:15,017 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 808856 == 0. Received: {'172.23.121.123': 808856} for bucket 'GleamBookUsers0' 2022-02-07 23:10:15,017 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:15,019 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:15,019 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:15,062 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:15,140 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:15,250 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 526935 == 0 2022-02-07 23:10:15,250 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 526935 == 0. Received: {'172.23.121.127': 526935} for bucket 'GleamBookUsers0' 2022-02-07 23:10:15,250 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:15,252 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:15,253 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:16,121 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 359330 == 0 2022-02-07 23:10:16,121 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 359330 == 0. Received: {'172.23.120.170': 359330} for bucket 'GleamBookUsers0' 2022-02-07 23:10:16,121 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:16,124 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:16,124 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:16,322 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1811980 == 0 2022-02-07 23:10:16,322 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1811980 == 0. Received: {'172.23.121.115': 1811980} for bucket 'GleamBookUsers0' 2022-02-07 23:10:16,322 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:16,325 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:16,325 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:16,433 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 726532 == 0 2022-02-07 23:10:16,433 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 726532 == 0. Received: {'172.23.121.126': 726532} for bucket 'GleamBookUsers0' 2022-02-07 23:10:16,433 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:16,434 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:16,434 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:17,979 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:18,503 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:18,898 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:19,002 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 468353 == 0 2022-02-07 23:10:19,003 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 468353 == 0. Received: {'172.23.121.124': 468353} for bucket 'GleamBookUsers0' 2022-02-07 23:10:19,003 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:19,005 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:19,005 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:19,339 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 297938 == 0 2022-02-07 23:10:19,339 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 297938 == 0. Received: {'172.23.121.128': 297938} for bucket 'GleamBookUsers0' 2022-02-07 23:10:19,339 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:19,342 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:19,342 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:19,852 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 278245 == 0 2022-02-07 23:10:19,852 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 278245 == 0. Received: {'172.23.121.129': 278245} for bucket 'GleamBookUsers0' 2022-02-07 23:10:19,852 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:19,855 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:19,855 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:20,019 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:20,253 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:21,124 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:21,325 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:21,361 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 789614 == 0 2022-02-07 23:10:21,362 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 789614 == 0. Received: {'172.23.121.123': 789614} for bucket 'GleamBookUsers0' 2022-02-07 23:10:21,362 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:21,365 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:21,365 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:21,437 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:21,460 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 500745 == 0 2022-02-07 23:10:21,460 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 500745 == 0. Received: {'172.23.121.127': 500745} for bucket 'GleamBookUsers0' 2022-02-07 23:10:21,460 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:21,463 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:21,463 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:22,009 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 345090 == 0 2022-02-07 23:10:22,009 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 345090 == 0. Received: {'172.23.120.170': 345090} for bucket 'GleamBookUsers0' 2022-02-07 23:10:22,009 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:22,012 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:22,012 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:22,709 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 701202 == 0 2022-02-07 23:10:22,709 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 701202 == 0. Received: {'172.23.121.126': 701202} for bucket 'GleamBookUsers0' 2022-02-07 23:10:22,709 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:22,710 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:22,710 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:23,519 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1787982 == 0 2022-02-07 23:10:23,519 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1787982 == 0. Received: {'172.23.121.115': 1787982} for bucket 'GleamBookUsers0' 2022-02-07 23:10:23,519 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:23,522 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:23,522 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:24,005 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:24,342 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:24,855 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:25,072 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 441877 == 0 2022-02-07 23:10:25,072 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 441877 == 0. Received: {'172.23.121.124': 441877} for bucket 'GleamBookUsers0' 2022-02-07 23:10:25,072 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:25,075 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:25,075 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:25,170 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 275928 == 0 2022-02-07 23:10:25,170 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 275928 == 0. Received: {'172.23.121.128': 275928} for bucket 'GleamBookUsers0' 2022-02-07 23:10:25,170 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:25,171 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:25,173 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:25,713 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 258262 == 0 2022-02-07 23:10:25,713 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 258262 == 0. Received: {'172.23.121.129': 258262} for bucket 'GleamBookUsers0' 2022-02-07 23:10:25,713 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:25,716 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:25,716 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:26,365 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:26,463 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:27,013 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:27,595 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 487455 == 0 2022-02-07 23:10:27,595 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 487455 == 0. Received: {'172.23.121.127': 487455} for bucket 'GleamBookUsers0' 2022-02-07 23:10:27,595 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:27,596 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:27,598 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:27,651 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 778242 == 0 2022-02-07 23:10:27,653 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 778242 == 0. Received: {'172.23.121.123': 778242} for bucket 'GleamBookUsers0' 2022-02-07 23:10:27,653 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:27,654 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:27,654 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:27,711 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:27,924 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 328448 == 0 2022-02-07 23:10:27,924 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 328448 == 0. Received: {'172.23.120.170': 328448} for bucket 'GleamBookUsers0' 2022-02-07 23:10:27,924 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:27,927 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:27,927 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:28,523 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:28,969 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 668592 == 0 2022-02-07 23:10:28,969 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 668592 == 0. Received: {'172.23.121.126': 668592} for bucket 'GleamBookUsers0' 2022-02-07 23:10:28,969 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:28,970 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:28,971 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:30,075 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:30,173 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:30,621 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1726867 == 0 2022-02-07 23:10:30,621 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1726867 == 0. Received: {'172.23.121.115': 1726867} for bucket 'GleamBookUsers0' 2022-02-07 23:10:30,621 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:30,622 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:30,622 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:30,716 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:31,005 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 256435 == 0 2022-02-07 23:10:31,006 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 256435 == 0. Received: {'172.23.121.128': 256435} for bucket 'GleamBookUsers0' 2022-02-07 23:10:31,006 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:31,007 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:31,007 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:31,111 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 420025 == 0 2022-02-07 23:10:31,112 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 420025 == 0. Received: {'172.23.121.124': 420025} for bucket 'GleamBookUsers0' 2022-02-07 23:10:31,112 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:31,115 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:31,115 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:31,507 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 233944 == 0 2022-02-07 23:10:31,507 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 233944 == 0. Received: {'172.23.121.129': 233944} for bucket 'GleamBookUsers0' 2022-02-07 23:10:31,509 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:31,509 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:31,509 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:32,598 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:32,655 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:32,927 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:33,674 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 466303 == 0 2022-02-07 23:10:33,674 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 466303 == 0. Received: {'172.23.121.127': 466303} for bucket 'GleamBookUsers0' 2022-02-07 23:10:33,674 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:33,677 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:33,677 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:33,915 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 312818 == 0 2022-02-07 23:10:33,915 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 312818 == 0. Received: {'172.23.120.170': 312818} for bucket 'GleamBookUsers0' 2022-02-07 23:10:33,915 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:33,918 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:33,918 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:33,963 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 758578 == 0 2022-02-07 23:10:33,963 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 758578 == 0. Received: {'172.23.121.123': 758578} for bucket 'GleamBookUsers0' 2022-02-07 23:10:33,963 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:33,966 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:33,966 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:33,971 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:35,187 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 645770 == 0 2022-02-07 23:10:35,187 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 645770 == 0. Received: {'172.23.121.126': 645770} for bucket 'GleamBookUsers0' 2022-02-07 23:10:35,187 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:35,188 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:35,188 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:35,624 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:36,009 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:36,115 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:36,510 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:36,765 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 232177 == 0 2022-02-07 23:10:36,765 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 232177 == 0. Received: {'172.23.121.128': 232177} for bucket 'GleamBookUsers0' 2022-02-07 23:10:36,766 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:36,766 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:36,767 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:37,095 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 407915 == 0 2022-02-07 23:10:37,095 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 407915 == 0. Received: {'172.23.121.124': 407915} for bucket 'GleamBookUsers0' 2022-02-07 23:10:37,095 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:37,098 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:37,098 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:37,296 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 206120 == 0 2022-02-07 23:10:37,298 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 206120 == 0. Received: {'172.23.121.129': 206120} for bucket 'GleamBookUsers0' 2022-02-07 23:10:37,298 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:37,299 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:37,299 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:37,782 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1717287 == 0 2022-02-07 23:10:37,782 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1717287 == 0. Received: {'172.23.121.115': 1717287} for bucket 'GleamBookUsers0' 2022-02-07 23:10:37,782 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:37,785 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:37,785 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:38,678 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:38,918 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:38,966 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:39,753 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 442241 == 0 2022-02-07 23:10:39,755 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 442241 == 0. Received: {'172.23.121.127': 442241} for bucket 'GleamBookUsers0' 2022-02-07 23:10:39,755 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:39,756 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:39,757 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:39,770 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 297130 == 0 2022-02-07 23:10:39,770 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 297130 == 0. Received: {'172.23.120.170': 297130} for bucket 'GleamBookUsers0' 2022-02-07 23:10:39,770 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:39,773 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:39,773 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:40,117 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 720026 == 0 2022-02-07 23:10:40,117 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 720026 == 0. Received: {'172.23.121.123': 720026} for bucket 'GleamBookUsers0' 2022-02-07 23:10:40,117 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:40,118 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:40,118 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:40,190 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:41,352 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 620082 == 0 2022-02-07 23:10:41,354 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 620082 == 0. Received: {'172.23.121.126': 620082} for bucket 'GleamBookUsers0' 2022-02-07 23:10:41,354 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:41,355 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:41,355 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:41,767 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:42,098 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:42,299 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:42,470 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 216181 == 0 2022-02-07 23:10:42,470 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 216181 == 0. Received: {'172.23.121.128': 216181} for bucket 'GleamBookUsers0' 2022-02-07 23:10:42,470 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:42,473 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:42,473 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:42,789 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:43,056 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 182890 == 0 2022-02-07 23:10:43,056 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 182890 == 0. Received: {'172.23.121.129': 182890} for bucket 'GleamBookUsers0' 2022-02-07 23:10:43,056 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:43,059 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:43,059 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:43,098 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 390109 == 0 2022-02-07 23:10:43,098 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 390109 == 0. Received: {'172.23.121.124': 390109} for bucket 'GleamBookUsers0' 2022-02-07 23:10:43,098 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:43,101 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:43,101 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:44,757 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:44,773 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:44,812 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1685869 == 0 2022-02-07 23:10:44,813 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1685869 == 0. Received: {'172.23.121.115': 1685869} for bucket 'GleamBookUsers0' 2022-02-07 23:10:44,813 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:44,815 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:44,815 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:45,119 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:45,618 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 274706 == 0 2022-02-07 23:10:45,618 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 274706 == 0. Received: {'172.23.120.170': 274706} for bucket 'GleamBookUsers0' 2022-02-07 23:10:45,618 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:45,621 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:45,621 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:45,759 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 422460 == 0 2022-02-07 23:10:45,759 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 422460 == 0. Received: {'172.23.121.127': 422460} for bucket 'GleamBookUsers0' 2022-02-07 23:10:45,759 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:45,760 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:45,760 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:46,255 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 691210 == 0 2022-02-07 23:10:46,256 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 691210 == 0. Received: {'172.23.121.123': 691210} for bucket 'GleamBookUsers0' 2022-02-07 23:10:46,256 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:46,257 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:46,257 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:46,355 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:47,444 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 593972 == 0 2022-02-07 23:10:47,444 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 593972 == 0. Received: {'172.23.121.126': 593972} for bucket 'GleamBookUsers0' 2022-02-07 23:10:47,444 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:47,446 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:47,447 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:47,473 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:48,059 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:48,101 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:48,167 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 207053 == 0 2022-02-07 23:10:48,167 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 207053 == 0. Received: {'172.23.121.128': 207053} for bucket 'GleamBookUsers0' 2022-02-07 23:10:48,167 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:48,168 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:48,170 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:48,740 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 168412 == 0 2022-02-07 23:10:48,740 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 168412 == 0. Received: {'172.23.121.129': 168412} for bucket 'GleamBookUsers0' 2022-02-07 23:10:48,740 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:48,742 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:48,742 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:49,058 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 385709 == 0 2022-02-07 23:10:49,059 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 385709 == 0. Received: {'172.23.121.124': 385709} for bucket 'GleamBookUsers0' 2022-02-07 23:10:49,059 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:49,061 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:49,061 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:49,815 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:50,622 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:50,762 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:51,257 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:51,428 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 264194 == 0 2022-02-07 23:10:51,428 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 264194 == 0. Received: {'172.23.120.170': 264194} for bucket 'GleamBookUsers0' 2022-02-07 23:10:51,430 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:51,431 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:51,431 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:51,815 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 387983 == 0 2022-02-07 23:10:51,816 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 387983 == 0. Received: {'172.23.121.127': 387983} for bucket 'GleamBookUsers0' 2022-02-07 23:10:51,816 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:51,819 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:51,819 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:51,858 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1660927 == 0 2022-02-07 23:10:51,858 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1660927 == 0. Received: {'172.23.121.115': 1660927} for bucket 'GleamBookUsers0' 2022-02-07 23:10:51,858 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:51,859 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:51,859 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:52,427 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 680990 == 0 2022-02-07 23:10:52,428 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 680990 == 0. Received: {'172.23.121.123': 680990} for bucket 'GleamBookUsers0' 2022-02-07 23:10:52,428 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:52,430 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:52,430 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:52,447 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:53,170 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:53,507 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 564958 == 0 2022-02-07 23:10:53,509 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 564958 == 0. Received: {'172.23.121.126': 564958} for bucket 'GleamBookUsers0' 2022-02-07 23:10:53,509 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:53,510 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:53,510 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:53,743 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:53,849 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 189482 == 0 2022-02-07 23:10:53,849 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 189482 == 0. Received: {'172.23.121.128': 189482} for bucket 'GleamBookUsers0' 2022-02-07 23:10:53,851 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:53,852 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:53,852 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:54,061 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:54,417 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 156392 == 0 2022-02-07 23:10:54,417 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 156392 == 0. Received: {'172.23.121.129': 156392} for bucket 'GleamBookUsers0' 2022-02-07 23:10:54,417 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:54,418 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:54,418 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:54,966 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 359651 == 0 2022-02-07 23:10:54,966 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 359651 == 0. Received: {'172.23.121.124': 359651} for bucket 'GleamBookUsers0' 2022-02-07 23:10:54,966 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:54,967 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:54,967 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:56,433 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:56,819 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:56,861 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:57,164 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 243928 == 0 2022-02-07 23:10:57,165 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 243928 == 0. Received: {'172.23.120.170': 243928} for bucket 'GleamBookUsers0' 2022-02-07 23:10:57,165 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:57,167 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:57,167 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:57,430 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:57,760 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 365450 == 0 2022-02-07 23:10:57,762 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 365450 == 0. Received: {'172.23.121.127': 365450} for bucket 'GleamBookUsers0' 2022-02-07 23:10:57,762 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:57,763 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:57,763 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:58,512 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:58,753 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 664930 == 0 2022-02-07 23:10:58,753 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 664930 == 0. Received: {'172.23.121.123': 664930} for bucket 'GleamBookUsers0' 2022-02-07 23:10:58,753 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:58,755 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:58,756 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:58,852 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:58,921 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1637121 == 0 2022-02-07 23:10:58,923 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1637121 == 0. Received: {'172.23.121.115': 1637121} for bucket 'GleamBookUsers0' 2022-02-07 23:10:58,923 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:58,924 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:58,924 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:59,420 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:10:59,561 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 178668 == 0 2022-02-07 23:10:59,562 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 178668 == 0. Received: {'172.23.121.128': 178668} for bucket 'GleamBookUsers0' 2022-02-07 23:10:59,562 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:59,563 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:59,563 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:59,657 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 533778 == 0 2022-02-07 23:10:59,657 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 533778 == 0. Received: {'172.23.121.126': 533778} for bucket 'GleamBookUsers0' 2022-02-07 23:10:59,657 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:10:59,660 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:10:59,660 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:10:59,969 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:00,091 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 145681 == 0 2022-02-07 23:11:00,091 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 145681 == 0. Received: {'172.23.121.129': 145681} for bucket 'GleamBookUsers0' 2022-02-07 23:11:00,091 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:00,094 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:00,094 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:00,848 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 342040 == 0 2022-02-07 23:11:00,848 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 342040 == 0. Received: {'172.23.121.124': 342040} for bucket 'GleamBookUsers0' 2022-02-07 23:11:00,848 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:00,851 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:00,851 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:02,167 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:02,763 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:02,911 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 238962 == 0 2022-02-07 23:11:02,911 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 238962 == 0. Received: {'172.23.120.170': 238962} for bucket 'GleamBookUsers0' 2022-02-07 23:11:02,913 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:02,914 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:02,914 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:03,691 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 344497 == 0 2022-02-07 23:11:03,691 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 344497 == 0. Received: {'172.23.121.127': 344497} for bucket 'GleamBookUsers0' 2022-02-07 23:11:03,691 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:03,694 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:03,694 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:03,756 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:03,924 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:04,565 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:04,660 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:04,878 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 647430 == 0 2022-02-07 23:11:04,878 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 647430 == 0. Received: {'172.23.121.123': 647430} for bucket 'GleamBookUsers0' 2022-02-07 23:11:04,878 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:04,881 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:04,881 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:05,094 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:05,214 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 169834 == 0 2022-02-07 23:11:05,214 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 169834 == 0. Received: {'172.23.121.128': 169834} for bucket 'GleamBookUsers0' 2022-02-07 23:11:05,214 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:05,217 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:05,217 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:05,796 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 121690 == 0 2022-02-07 23:11:05,796 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 121690 == 0. Received: {'172.23.121.129': 121690} for bucket 'GleamBookUsers0' 2022-02-07 23:11:05,796 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:05,799 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:05,799 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:05,839 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 525928 == 0 2022-02-07 23:11:05,839 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 525928 == 0. Received: {'172.23.121.126': 525928} for bucket 'GleamBookUsers0' 2022-02-07 23:11:05,839 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:05,842 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:05,842 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:05,851 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:05,966 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1614401 == 0 2022-02-07 23:11:05,966 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1614401 == 0. Received: {'172.23.121.115': 1614401} for bucket 'GleamBookUsers0' 2022-02-07 23:11:05,966 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:05,967 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:05,969 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:06,753 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 324971 == 0 2022-02-07 23:11:06,753 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 324971 == 0. Received: {'172.23.121.124': 324971} for bucket 'GleamBookUsers0' 2022-02-07 23:11:06,753 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:06,756 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:06,756 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:07,914 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:08,632 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 214384 == 0 2022-02-07 23:11:08,634 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 214384 == 0. Received: {'172.23.120.170': 214384} for bucket 'GleamBookUsers0' 2022-02-07 23:11:08,634 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:08,635 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:08,635 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:08,696 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:09,575 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 308498 == 0 2022-02-07 23:11:09,575 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 308498 == 0. Received: {'172.23.121.127': 308498} for bucket 'GleamBookUsers0' 2022-02-07 23:11:09,575 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:09,578 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:09,578 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:09,881 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:10,217 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:10,799 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:10,842 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:10,904 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 148684 == 0 2022-02-07 23:11:10,904 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 148684 == 0. Received: {'172.23.121.128': 148684} for bucket 'GleamBookUsers0' 2022-02-07 23:11:10,904 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:10,907 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:10,907 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:10,941 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 597613 == 0 2022-02-07 23:11:10,941 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 597613 == 0. Received: {'172.23.121.123': 597613} for bucket 'GleamBookUsers0' 2022-02-07 23:11:10,941 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:10,944 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:10,944 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:10,969 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:11,365 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 96434 == 0 2022-02-07 23:11:11,365 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 96434 == 0. Received: {'172.23.121.129': 96434} for bucket 'GleamBookUsers0' 2022-02-07 23:11:11,365 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:11,368 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:11,368 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:11,756 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:11,865 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 495565 == 0 2022-02-07 23:11:11,867 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 495565 == 0. Received: {'172.23.121.126': 495565} for bucket 'GleamBookUsers0' 2022-02-07 23:11:11,867 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:11,868 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:11,868 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:12,664 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 295781 == 0 2022-02-07 23:11:12,664 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 295781 == 0. Received: {'172.23.121.124': 295781} for bucket 'GleamBookUsers0' 2022-02-07 23:11:12,664 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:12,667 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:12,667 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:12,857 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1598673 == 0 2022-02-07 23:11:12,857 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1598673 == 0. Received: {'172.23.121.115': 1598673} for bucket 'GleamBookUsers0' 2022-02-07 23:11:12,857 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:12,858 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:12,858 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:13,635 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:14,336 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 190700 == 0 2022-02-07 23:11:14,336 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 190700 == 0. Received: {'172.23.120.170': 190700} for bucket 'GleamBookUsers0' 2022-02-07 23:11:14,336 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:14,338 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:14,338 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:14,578 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:15,450 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 287675 == 0 2022-02-07 23:11:15,451 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 287675 == 0. Received: {'172.23.121.127': 287675} for bucket 'GleamBookUsers0' 2022-02-07 23:11:15,451 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:15,453 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:15,453 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:15,907 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:15,944 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:16,368 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:16,512 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 139380 == 0 2022-02-07 23:11:16,513 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 139380 == 0. Received: {'172.23.121.128': 139380} for bucket 'GleamBookUsers0' 2022-02-07 23:11:16,513 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:16,515 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:16,515 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:16,875 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:16,943 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 79952 == 0 2022-02-07 23:11:16,944 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 79952 == 0. Received: {'172.23.121.129': 79952} for bucket 'GleamBookUsers0' 2022-02-07 23:11:16,944 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:16,947 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:16,947 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:16,984 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 578121 == 0 2022-02-07 23:11:16,984 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 578121 == 0. Received: {'172.23.121.123': 578121} for bucket 'GleamBookUsers0' 2022-02-07 23:11:16,984 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:16,986 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:16,986 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:17,667 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:17,832 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 463392 == 0 2022-02-07 23:11:17,832 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 463392 == 0. Received: {'172.23.121.126': 463392} for bucket 'GleamBookUsers0' 2022-02-07 23:11:17,834 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:17,835 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:17,835 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:17,859 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:18,490 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 281583 == 0 2022-02-07 23:11:18,490 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 281583 == 0. Received: {'172.23.121.124': 281583} for bucket 'GleamBookUsers0' 2022-02-07 23:11:18,490 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:18,493 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:18,493 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:19,341 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:19,710 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1563221 == 0 2022-02-07 23:11:19,710 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1563221 == 0. Received: {'172.23.121.115': 1563221} for bucket 'GleamBookUsers0' 2022-02-07 23:11:19,710 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:19,711 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:19,711 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:20,028 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 174380 == 0 2022-02-07 23:11:20,029 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 174380 == 0. Received: {'172.23.120.170': 174380} for bucket 'GleamBookUsers0' 2022-02-07 23:11:20,029 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:20,030 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:20,030 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:20,454 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:21,263 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 266903 == 0 2022-02-07 23:11:21,263 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 266903 == 0. Received: {'172.23.121.127': 266903} for bucket 'GleamBookUsers0' 2022-02-07 23:11:21,263 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:21,266 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:21,266 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:21,515 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:21,947 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:21,987 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:22,105 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 129612 == 0 2022-02-07 23:11:22,105 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 129612 == 0. Received: {'172.23.121.128': 129612} for bucket 'GleamBookUsers0' 2022-02-07 23:11:22,105 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:22,107 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:22,107 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:22,463 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 72277 == 0 2022-02-07 23:11:22,463 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 72277 == 0. Received: {'172.23.121.129': 72277} for bucket 'GleamBookUsers0' 2022-02-07 23:11:22,463 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:22,464 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:22,464 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:22,835 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:22,951 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 572701 == 0 2022-02-07 23:11:22,953 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 572701 == 0. Received: {'172.23.121.123': 572701} for bucket 'GleamBookUsers0' 2022-02-07 23:11:22,953 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:22,954 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:22,954 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:23,493 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:23,732 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 436758 == 0 2022-02-07 23:11:23,732 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 436758 == 0. Received: {'172.23.121.126': 436758} for bucket 'GleamBookUsers0' 2022-02-07 23:11:23,733 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:23,733 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:23,734 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:24,266 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 264498 == 0 2022-02-07 23:11:24,266 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 264498 == 0. Received: {'172.23.121.124': 264498} for bucket 'GleamBookUsers0' 2022-02-07 23:11:24,266 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:24,269 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:24,269 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:24,713 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:25,030 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:25,690 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 160894 == 0 2022-02-07 23:11:25,690 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 160894 == 0. Received: {'172.23.120.170': 160894} for bucket 'GleamBookUsers0' 2022-02-07 23:11:25,690 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:25,691 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:25,691 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:26,266 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:26,588 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1533625 == 0 2022-02-07 23:11:26,588 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1533625 == 0. Received: {'172.23.121.115': 1533625} for bucket 'GleamBookUsers0' 2022-02-07 23:11:26,588 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:26,591 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:26,592 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:27,065 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 248981 == 0 2022-02-07 23:11:27,065 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 248981 == 0. Received: {'172.23.121.127': 248981} for bucket 'GleamBookUsers0' 2022-02-07 23:11:27,066 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:27,068 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:27,068 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:27,108 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:27,466 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:27,717 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 123774 == 0 2022-02-07 23:11:27,717 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 123774 == 0. Received: {'172.23.121.128': 123774} for bucket 'GleamBookUsers0' 2022-02-07 23:11:27,719 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:27,720 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:27,721 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:27,954 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:27,979 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 63222 == 0 2022-02-07 23:11:27,979 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 63222 == 0. Received: {'172.23.121.129': 63222} for bucket 'GleamBookUsers0' 2022-02-07 23:11:27,979 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:27,980 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:27,980 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:28,734 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:28,904 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 539381 == 0 2022-02-07 23:11:28,904 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 539381 == 0. Received: {'172.23.121.123': 539381} for bucket 'GleamBookUsers0' 2022-02-07 23:11:28,904 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:28,905 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:28,907 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:29,270 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:29,648 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 432364 == 0 2022-02-07 23:11:29,648 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 432364 == 0. Received: {'172.23.121.126': 432364} for bucket 'GleamBookUsers0' 2022-02-07 23:11:29,648 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:29,651 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:29,651 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:30,020 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 254868 == 0 2022-02-07 23:11:30,020 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 254868 == 0. Received: {'172.23.121.124': 254868} for bucket 'GleamBookUsers0' 2022-02-07 23:11:30,020 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:30,023 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:30,023 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:30,691 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:31,335 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 149723 == 0 2022-02-07 23:11:31,335 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 149723 == 0. Received: {'172.23.120.170': 149723} for bucket 'GleamBookUsers0' 2022-02-07 23:11:31,335 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:31,338 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:31,338 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:31,592 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:32,069 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:32,721 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:32,875 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 237779 == 0 2022-02-07 23:11:32,875 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 237779 == 0. Received: {'172.23.121.127': 237779} for bucket 'GleamBookUsers0' 2022-02-07 23:11:32,875 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:32,878 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:32,878 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:32,982 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:33,355 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 119820 == 0 2022-02-07 23:11:33,357 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 119820 == 0. Received: {'172.23.121.128': 119820} for bucket 'GleamBookUsers0' 2022-02-07 23:11:33,357 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:33,358 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:33,358 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:33,392 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1496895 == 0 2022-02-07 23:11:33,394 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1496895 == 0. Received: {'172.23.121.115': 1496895} for bucket 'GleamBookUsers0' 2022-02-07 23:11:33,394 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:33,395 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:33,395 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:33,463 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 48257 == 0 2022-02-07 23:11:33,464 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 48257 == 0. Received: {'172.23.121.129': 48257} for bucket 'GleamBookUsers0' 2022-02-07 23:11:33,464 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:33,467 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:33,467 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:33,907 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:34,653 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:34,851 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 524303 == 0 2022-02-07 23:11:34,852 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 524303 == 0. Received: {'172.23.121.123': 524303} for bucket 'GleamBookUsers0' 2022-02-07 23:11:34,852 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:34,855 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:34,855 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:35,023 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:35,516 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 414431 == 0 2022-02-07 23:11:35,516 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 414431 == 0. Received: {'172.23.121.126': 414431} for bucket 'GleamBookUsers0' 2022-02-07 23:11:35,516 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:35,519 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:35,519 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:35,763 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 240277 == 0 2022-02-07 23:11:35,765 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 240277 == 0. Received: {'172.23.121.124': 240277} for bucket 'GleamBookUsers0' 2022-02-07 23:11:35,765 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:35,766 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:35,766 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:36,338 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:36,967 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 135637 == 0 2022-02-07 23:11:36,967 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 135637 == 0. Received: {'172.23.120.170': 135637} for bucket 'GleamBookUsers0' 2022-02-07 23:11:36,967 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:36,970 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:36,970 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:37,878 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:38,359 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:38,395 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:38,467 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:38,683 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 222573 == 0 2022-02-07 23:11:38,683 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 222573 == 0. Received: {'172.23.121.127': 222573} for bucket 'GleamBookUsers0' 2022-02-07 23:11:38,683 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:38,684 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:38,686 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:38,921 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 33458 == 0 2022-02-07 23:11:38,921 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 33458 == 0. Received: {'172.23.121.129': 33458} for bucket 'GleamBookUsers0' 2022-02-07 23:11:38,921 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:38,924 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:38,924 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:38,940 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 104024 == 0 2022-02-07 23:11:38,940 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 104024 == 0. Received: {'172.23.121.128': 104024} for bucket 'GleamBookUsers0' 2022-02-07 23:11:38,940 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:38,943 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:38,943 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:39,855 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:40,131 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1475343 == 0 2022-02-07 23:11:40,131 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1475343 == 0. Received: {'172.23.121.115': 1475343} for bucket 'GleamBookUsers0' 2022-02-07 23:11:40,131 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:40,132 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:40,134 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:40,519 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:40,766 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:40,779 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 509581 == 0 2022-02-07 23:11:40,779 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 509581 == 0. Received: {'172.23.121.123': 509581} for bucket 'GleamBookUsers0' 2022-02-07 23:11:40,779 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:40,782 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:40,782 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:41,326 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 379053 == 0 2022-02-07 23:11:41,328 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 379053 == 0. Received: {'172.23.121.126': 379053} for bucket 'GleamBookUsers0' 2022-02-07 23:11:41,328 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:41,329 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:41,329 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:41,473 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 216463 == 0 2022-02-07 23:11:41,473 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 216463 == 0. Received: {'172.23.121.124': 216463} for bucket 'GleamBookUsers0' 2022-02-07 23:11:41,473 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:41,474 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:41,474 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:41,970 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:42,585 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 128446 == 0 2022-02-07 23:11:42,586 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 128446 == 0. Received: {'172.23.120.170': 128446} for bucket 'GleamBookUsers0' 2022-02-07 23:11:42,586 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:42,588 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:42,588 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:43,686 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:43,924 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:43,943 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:44,348 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 24871 == 0 2022-02-07 23:11:44,349 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 24871 == 0. Received: {'172.23.121.129': 24871} for bucket 'GleamBookUsers0' 2022-02-07 23:11:44,349 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:44,351 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:44,352 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:44,476 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 195706 == 0 2022-02-07 23:11:44,476 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 195706 == 0. Received: {'172.23.121.127': 195706} for bucket 'GleamBookUsers0' 2022-02-07 23:11:44,476 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:44,480 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:44,480 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:44,540 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 95846 == 0 2022-02-07 23:11:44,540 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 95846 == 0. Received: {'172.23.121.128': 95846} for bucket 'GleamBookUsers0' 2022-02-07 23:11:44,542 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:44,543 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:44,543 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:45,134 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:45,782 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:46,329 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:46,476 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:46,749 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 461175 == 0 2022-02-07 23:11:46,749 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 461175 == 0. Received: {'172.23.121.123': 461175} for bucket 'GleamBookUsers0' 2022-02-07 23:11:46,749 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:46,752 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:46,752 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:46,959 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1449093 == 0 2022-02-07 23:11:46,960 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1449093 == 0. Received: {'172.23.121.115': 1449093} for bucket 'GleamBookUsers0' 2022-02-07 23:11:46,960 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:46,963 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:46,963 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:47,181 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 362485 == 0 2022-02-07 23:11:47,181 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 362485 == 0. Received: {'172.23.121.126': 362485} for bucket 'GleamBookUsers0' 2022-02-07 23:11:47,181 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:47,184 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:47,184 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:47,209 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 204619 == 0 2022-02-07 23:11:47,210 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 204619 == 0. Received: {'172.23.121.124': 204619} for bucket 'GleamBookUsers0' 2022-02-07 23:11:47,210 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:47,211 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:47,211 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:47,588 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:48,174 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 111781 == 0 2022-02-07 23:11:48,174 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 111781 == 0. Received: {'172.23.120.170': 111781} for bucket 'GleamBookUsers0' 2022-02-07 23:11:48,174 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:48,176 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:48,176 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:49,352 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:49,480 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:49,545 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:49,772 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 19617 == 0 2022-02-07 23:11:49,773 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 19617 == 0. Received: {'172.23.121.129': 19617} for bucket 'GleamBookUsers0' 2022-02-07 23:11:49,773 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:49,775 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:49,775 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:50,082 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 78094 == 0 2022-02-07 23:11:50,082 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 78094 == 0. Received: {'172.23.121.128': 78094} for bucket 'GleamBookUsers0' 2022-02-07 23:11:50,082 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:50,085 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:50,085 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:50,221 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 186546 == 0 2022-02-07 23:11:50,223 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 186546 == 0. Received: {'172.23.121.127': 186546} for bucket 'GleamBookUsers0' 2022-02-07 23:11:50,223 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:50,224 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:50,224 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:51,753 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:51,963 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:52,186 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:52,211 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:52,622 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 452385 == 0 2022-02-07 23:11:52,624 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 452385 == 0. Received: {'172.23.121.123': 452385} for bucket 'GleamBookUsers0' 2022-02-07 23:11:52,624 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:52,625 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:52,625 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:52,921 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 188244 == 0 2022-02-07 23:11:52,921 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 188244 == 0. Received: {'172.23.121.124': 188244} for bucket 'GleamBookUsers0' 2022-02-07 23:11:52,921 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:52,924 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:52,924 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:53,068 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 337335 == 0 2022-02-07 23:11:53,069 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 337335 == 0. Received: {'172.23.121.126': 337335} for bucket 'GleamBookUsers0' 2022-02-07 23:11:53,069 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:53,071 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:53,071 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:53,176 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:53,747 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1432763 == 0 2022-02-07 23:11:53,749 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1432763 == 0. Received: {'172.23.121.115': 1432763} for bucket 'GleamBookUsers0' 2022-02-07 23:11:53,749 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:53,750 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:53,750 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:53,786 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 97528 == 0 2022-02-07 23:11:53,786 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 97528 == 0. Received: {'172.23.120.170': 97528} for bucket 'GleamBookUsers0' 2022-02-07 23:11:53,786 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:53,788 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:53,789 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:54,775 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:55,085 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:55,184 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 17421 == 0 2022-02-07 23:11:55,184 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 17421 == 0. Received: {'172.23.121.129': 17421} for bucket 'GleamBookUsers0' 2022-02-07 23:11:55,184 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:55,187 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:55,187 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:55,224 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:55,586 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 69587 == 0 2022-02-07 23:11:55,588 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 69587 == 0. Received: {'172.23.121.128': 69587} for bucket 'GleamBookUsers0' 2022-02-07 23:11:55,588 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:55,589 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:55,589 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:55,946 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 170140 == 0 2022-02-07 23:11:55,946 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 170140 == 0. Received: {'172.23.121.127': 170140} for bucket 'GleamBookUsers0' 2022-02-07 23:11:55,946 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:55,947 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:55,948 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:57,627 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:57,924 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:58,071 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:58,493 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 430939 == 0 2022-02-07 23:11:58,493 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 430939 == 0. Received: {'172.23.121.123': 430939} for bucket 'GleamBookUsers0' 2022-02-07 23:11:58,493 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:58,496 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:58,496 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:58,608 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 178928 == 0 2022-02-07 23:11:58,608 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 178928 == 0. Received: {'172.23.121.124': 178928} for bucket 'GleamBookUsers0' 2022-02-07 23:11:58,608 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:58,609 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:58,609 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:58,752 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:58,789 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:11:58,881 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 320261 == 0 2022-02-07 23:11:58,881 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 320261 == 0. Received: {'172.23.121.126': 320261} for bucket 'GleamBookUsers0' 2022-02-07 23:11:58,881 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:58,884 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:58,884 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:11:59,331 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 84060 == 0 2022-02-07 23:11:59,332 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 84060 == 0. Received: {'172.23.120.170': 84060} for bucket 'GleamBookUsers0' 2022-02-07 23:11:59,332 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:11:59,334 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:11:59,334 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:00,187 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:00,477 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1415131 == 0 2022-02-07 23:12:00,477 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1415131 == 0. Received: {'172.23.121.115': 1415131} for bucket 'GleamBookUsers0' 2022-02-07 23:12:00,477 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:00,480 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:00,480 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:00,572 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 12971 == 0 2022-02-07 23:12:00,572 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 12971 == 0. Received: {'172.23.121.129': 12971} for bucket 'GleamBookUsers0' 2022-02-07 23:12:00,572 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:00,575 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:00,575 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:00,589 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:00,948 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:01,075 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 59353 == 0 2022-02-07 23:12:01,075 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 59353 == 0. Received: {'172.23.121.128': 59353} for bucket 'GleamBookUsers0' 2022-02-07 23:12:01,075 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:01,078 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:01,078 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:01,627 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 159994 == 0 2022-02-07 23:12:01,628 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 159994 == 0. Received: {'172.23.121.127': 159994} for bucket 'GleamBookUsers0' 2022-02-07 23:12:01,628 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:01,630 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:01,630 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:03,496 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:03,611 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:03,884 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:04,309 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 163930 == 0 2022-02-07 23:12:04,309 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 163930 == 0. Received: {'172.23.121.124': 163930} for bucket 'GleamBookUsers0' 2022-02-07 23:12:04,309 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:04,311 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:04,312 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:04,335 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:04,378 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 404593 == 0 2022-02-07 23:12:04,378 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 404593 == 0. Received: {'172.23.121.123': 404593} for bucket 'GleamBookUsers0' 2022-02-07 23:12:04,378 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:04,380 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:04,380 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:04,624 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 300137 == 0 2022-02-07 23:12:04,625 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 300137 == 0. Received: {'172.23.121.126': 300137} for bucket 'GleamBookUsers0' 2022-02-07 23:12:04,625 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:04,627 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:04,627 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:04,882 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 75548 == 0 2022-02-07 23:12:04,882 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 75548 == 0. Received: {'172.23.120.170': 75548} for bucket 'GleamBookUsers0' 2022-02-07 23:12:04,882 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:04,885 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:04,885 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:05,482 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:05,575 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:05,957 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 9543 == 0 2022-02-07 23:12:05,957 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 9543 == 0. Received: {'172.23.121.129': 9543} for bucket 'GleamBookUsers0' 2022-02-07 23:12:05,957 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:05,959 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:05,959 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:06,078 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:06,542 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 50228 == 0 2022-02-07 23:12:06,542 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 50228 == 0. Received: {'172.23.121.128': 50228} for bucket 'GleamBookUsers0' 2022-02-07 23:12:06,542 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:06,545 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:06,545 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:06,631 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:07,247 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1401845 == 0 2022-02-07 23:12:07,247 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1401845 == 0. Received: {'172.23.121.115': 1401845} for bucket 'GleamBookUsers0' 2022-02-07 23:12:07,247 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:07,250 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:07,250 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:07,269 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 139646 == 0 2022-02-07 23:12:07,269 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 139646 == 0. Received: {'172.23.121.127': 139646} for bucket 'GleamBookUsers0' 2022-02-07 23:12:07,270 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:07,272 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:07,272 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:09,312 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:09,381 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:09,627 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:09,885 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:09,953 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 153998 == 0 2022-02-07 23:12:09,953 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 153998 == 0. Received: {'172.23.121.124': 153998} for bucket 'GleamBookUsers0' 2022-02-07 23:12:09,953 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:09,956 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:09,956 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:10,197 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 385042 == 0 2022-02-07 23:12:10,198 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 385042 == 0. Received: {'172.23.121.123': 385042} for bucket 'GleamBookUsers0' 2022-02-07 23:12:10,198 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:10,200 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:10,200 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:10,358 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 240735 == 0 2022-02-07 23:12:10,358 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 240735 == 0. Received: {'172.23.121.126': 240735} for bucket 'GleamBookUsers0' 2022-02-07 23:12:10,358 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:10,361 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:10,361 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:10,398 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 57461 == 0 2022-02-07 23:12:10,398 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 57461 == 0. Received: {'172.23.120.170': 57461} for bucket 'GleamBookUsers0' 2022-02-07 23:12:10,398 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:10,400 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:10,400 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:10,960 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:11,357 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 7155 == 0 2022-02-07 23:12:11,357 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 7155 == 0. Received: {'172.23.121.129': 7155} for bucket 'GleamBookUsers0' 2022-02-07 23:12:11,357 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:11,358 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:11,358 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:11,545 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:11,966 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 33355 == 0 2022-02-07 23:12:11,966 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 33355 == 0. Received: {'172.23.121.128': 33355} for bucket 'GleamBookUsers0' 2022-02-07 23:12:11,966 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:11,967 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:11,967 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:12,250 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:12,273 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:12,855 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 129042 == 0 2022-02-07 23:12:12,855 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 129042 == 0. Received: {'172.23.121.127': 129042} for bucket 'GleamBookUsers0' 2022-02-07 23:12:12,855 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:12,857 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:12,857 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:13,970 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1379939 == 0 2022-02-07 23:12:13,970 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1379939 == 0. Received: {'172.23.121.115': 1379939} for bucket 'GleamBookUsers0' 2022-02-07 23:12:13,970 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:13,973 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:13,973 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:14,956 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:15,201 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:15,361 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:15,401 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:15,582 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 143252 == 0 2022-02-07 23:12:15,582 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 143252 == 0. Received: {'172.23.121.124': 143252} for bucket 'GleamBookUsers0' 2022-02-07 23:12:15,582 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:15,585 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:15,585 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:15,947 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 40903 == 0 2022-02-07 23:12:15,948 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 40903 == 0. Received: {'172.23.120.170': 40903} for bucket 'GleamBookUsers0' 2022-02-07 23:12:15,948 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:15,950 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:15,951 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:16,073 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 367128 == 0 2022-02-07 23:12:16,073 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 367128 == 0. Received: {'172.23.121.123': 367128} for bucket 'GleamBookUsers0' 2022-02-07 23:12:16,073 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:16,076 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:16,076 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:16,119 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 229077 == 0 2022-02-07 23:12:16,119 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 229077 == 0. Received: {'172.23.121.126': 229077} for bucket 'GleamBookUsers0' 2022-02-07 23:12:16,121 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:16,121 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:16,122 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:16,359 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:16,727 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 5578 == 0 2022-02-07 23:12:16,727 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 5578 == 0. Received: {'172.23.121.129': 5578} for bucket 'GleamBookUsers0' 2022-02-07 23:12:16,727 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:16,730 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:16,730 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:16,969 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:17,378 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 25264 == 0 2022-02-07 23:12:17,378 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 25264 == 0. Received: {'172.23.121.128': 25264} for bucket 'GleamBookUsers0' 2022-02-07 23:12:17,378 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:17,381 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:17,381 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:17,861 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:18,411 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 108467 == 0 2022-02-07 23:12:18,411 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 108467 == 0. Received: {'172.23.121.127': 108467} for bucket 'GleamBookUsers0' 2022-02-07 23:12:18,411 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:18,413 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:18,413 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:18,973 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:20,585 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:20,599 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1319475 == 0 2022-02-07 23:12:20,599 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1319475 == 0. Received: {'172.23.121.115': 1319475} for bucket 'GleamBookUsers0' 2022-02-07 23:12:20,601 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:20,604 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:20,605 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:20,951 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:21,078 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:21,122 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:21,157 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 124048 == 0 2022-02-07 23:12:21,157 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 124048 == 0. Received: {'172.23.121.124': 124048} for bucket 'GleamBookUsers0' 2022-02-07 23:12:21,157 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:21,160 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:21,160 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:21,384 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 25788 == 0 2022-02-07 23:12:21,385 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 25788 == 0. Received: {'172.23.120.170': 25788} for bucket 'GleamBookUsers0' 2022-02-07 23:12:21,385 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:21,388 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:21,388 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:21,730 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:21,822 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 214341 == 0 2022-02-07 23:12:21,822 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 214341 == 0. Received: {'172.23.121.126': 214341} for bucket 'GleamBookUsers0' 2022-02-07 23:12:21,822 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:21,825 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:21,825 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:21,891 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 356518 == 0 2022-02-07 23:12:21,891 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 356518 == 0. Received: {'172.23.121.123': 356518} for bucket 'GleamBookUsers0' 2022-02-07 23:12:21,891 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:21,892 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:21,892 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:22,094 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 4320 == 0 2022-02-07 23:12:22,094 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 4320 == 0. Received: {'172.23.121.129': 4320} for bucket 'GleamBookUsers0' 2022-02-07 23:12:22,094 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:22,095 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:22,095 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:22,381 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:22,793 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 19764 == 0 2022-02-07 23:12:22,793 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 19764 == 0. Received: {'172.23.121.128': 19764} for bucket 'GleamBookUsers0' 2022-02-07 23:12:22,793 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:22,796 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:22,796 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:23,414 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:23,963 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 90608 == 0 2022-02-07 23:12:23,963 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 90608 == 0. Received: {'172.23.121.127': 90608} for bucket 'GleamBookUsers0' 2022-02-07 23:12:23,963 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:23,964 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:23,964 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:25,605 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:26,161 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:26,388 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:26,730 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 115601 == 0 2022-02-07 23:12:26,730 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 115601 == 0. Received: {'172.23.121.124': 115601} for bucket 'GleamBookUsers0' 2022-02-07 23:12:26,730 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:26,733 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:26,733 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:26,825 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:26,861 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 22228 == 0 2022-02-07 23:12:26,862 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 22228 == 0. Received: {'172.23.120.170': 22228} for bucket 'GleamBookUsers0' 2022-02-07 23:12:26,862 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:26,865 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:26,865 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:26,892 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:27,096 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:27,233 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1262425 == 0 2022-02-07 23:12:27,233 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1262425 == 0. Received: {'172.23.121.115': 1262425} for bucket 'GleamBookUsers0' 2022-02-07 23:12:27,233 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:27,236 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:27,236 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:27,483 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 1882 == 0 2022-02-07 23:12:27,483 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1882 == 0. Received: {'172.23.121.129': 1882} for bucket 'GleamBookUsers0' 2022-02-07 23:12:27,484 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:27,486 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:27,486 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:27,519 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 206354 == 0 2022-02-07 23:12:27,519 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 206354 == 0. Received: {'172.23.121.126': 206354} for bucket 'GleamBookUsers0' 2022-02-07 23:12:27,519 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:27,522 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:27,522 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:27,663 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 332682 == 0 2022-02-07 23:12:27,663 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 332682 == 0. Received: {'172.23.121.123': 332682} for bucket 'GleamBookUsers0' 2022-02-07 23:12:27,664 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:27,664 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:27,665 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:27,796 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:28,207 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 12106 == 0 2022-02-07 23:12:28,207 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 12106 == 0. Received: {'172.23.121.128': 12106} for bucket 'GleamBookUsers0' 2022-02-07 23:12:28,207 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:28,210 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:28,210 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:28,966 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:29,467 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 77894 == 0 2022-02-07 23:12:29,467 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 77894 == 0. Received: {'172.23.121.127': 77894} for bucket 'GleamBookUsers0' 2022-02-07 23:12:29,467 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:29,469 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:29,469 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:31,743 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:31,865 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:32,236 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:32,302 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 16932 == 0 2022-02-07 23:12:32,302 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 16932 == 0. Received: {'172.23.120.170': 16932} for bucket 'GleamBookUsers0' 2022-02-07 23:12:32,302 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:32,305 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:32,305 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:32,322 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 106507 == 0 2022-02-07 23:12:32,322 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 106507 == 0. Received: {'172.23.121.124': 106507} for bucket 'GleamBookUsers0' 2022-02-07 23:12:32,322 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:32,323 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:32,325 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:32,487 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:32,523 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:32,665 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:32,845 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 66 == 0 2022-02-07 23:12:32,845 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 66 == 0. Received: {'172.23.121.129': 66} for bucket 'GleamBookUsers0' 2022-02-07 23:12:32,845 | infra | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:32,848 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:32,848 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:33,150 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 192523 == 0 2022-02-07 23:12:33,151 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 192523 == 0. Received: {'172.23.121.126': 192523} for bucket 'GleamBookUsers0' 2022-02-07 23:12:33,151 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:33,153 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:33,153 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:33,210 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:33,368 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 316635 == 0 2022-02-07 23:12:33,368 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 316635 == 0. Received: {'172.23.121.123': 316635} for bucket 'GleamBookUsers0' 2022-02-07 23:12:33,368 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:33,369 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:33,371 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:33,631 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 7242 == 0 2022-02-07 23:12:33,632 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 7242 == 0. Received: {'172.23.121.128': 7242} for bucket 'GleamBookUsers0' 2022-02-07 23:12:33,632 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:33,634 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:33,634 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:33,836 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1220641 == 0 2022-02-07 23:12:33,836 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1220641 == 0. Received: {'172.23.121.115': 1220641} for bucket 'GleamBookUsers0' 2022-02-07 23:12:33,836 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:33,838 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:33,838 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:34,470 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:34,970 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 65365 == 0 2022-02-07 23:12:34,970 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 65365 == 0. Received: {'172.23.121.127': 65365} for bucket 'GleamBookUsers0' 2022-02-07 23:12:34,970 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:34,973 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:34,973 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:37,305 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:37,325 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:37,696 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 10834 == 0 2022-02-07 23:12:37,697 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 10834 == 0. Received: {'172.23.120.170': 10834} for bucket 'GleamBookUsers0' 2022-02-07 23:12:37,697 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:37,697 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:37,698 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:37,852 | infra | DEBUG | pool-3-thread-22 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:37,855 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 88264 == 0 2022-02-07 23:12:37,855 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 88264 == 0. Received: {'172.23.121.124': 88264} for bucket 'GleamBookUsers0' 2022-02-07 23:12:37,857 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:37,858 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:37,858 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:38,154 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:38,213 | test | DEBUG | pool-3-thread-22 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 23:12:38,213 | test | DEBUG | pool-3-thread-22 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.129': 0} for bucket 'GleamBookUsers0' 2022-02-07 23:12:38,213 | infra | INFO | pool-3-thread-22 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304164.11' completed 2022-02-07 23:12:38,371 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:38,635 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:38,776 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 167671 == 0 2022-02-07 23:12:38,776 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 167671 == 0. Received: {'172.23.121.126': 167671} for bucket 'GleamBookUsers0' 2022-02-07 23:12:38,778 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:38,779 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:38,779 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:38,839 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:39,009 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 1282 == 0 2022-02-07 23:12:39,009 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1282 == 0. Received: {'172.23.121.128': 1282} for bucket 'GleamBookUsers0' 2022-02-07 23:12:39,009 | infra | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:39,010 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:39,012 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:39,098 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 316102 == 0 2022-02-07 23:12:39,098 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 316102 == 0. Received: {'172.23.121.123': 316102} for bucket 'GleamBookUsers0' 2022-02-07 23:12:39,098 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:39,101 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:39,101 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:39,976 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:40,380 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1181795 == 0 2022-02-07 23:12:40,380 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1181795 == 0. Received: {'172.23.121.115': 1181795} for bucket 'GleamBookUsers0' 2022-02-07 23:12:40,381 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:40,382 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:40,382 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:40,489 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 53013 == 0 2022-02-07 23:12:40,489 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 53013 == 0. Received: {'172.23.121.127': 53013} for bucket 'GleamBookUsers0' 2022-02-07 23:12:40,490 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:40,492 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:40,492 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:42,698 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:42,858 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:43,082 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 8300 == 0 2022-02-07 23:12:43,082 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 8300 == 0. Received: {'172.23.120.170': 8300} for bucket 'GleamBookUsers0' 2022-02-07 23:12:43,082 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:43,084 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:43,085 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:43,341 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 69742 == 0 2022-02-07 23:12:43,342 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 69742 == 0. Received: {'172.23.121.124': 69742} for bucket 'GleamBookUsers0' 2022-02-07 23:12:43,342 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:43,344 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:43,344 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:43,779 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:44,019 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:44,101 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:44,408 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 142921 == 0 2022-02-07 23:12:44,408 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 142921 == 0. Received: {'172.23.121.126': 142921} for bucket 'GleamBookUsers0' 2022-02-07 23:12:44,408 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:44,411 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:44,411 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:44,415 | test | DEBUG | pool-3-thread-19 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 23:12:44,415 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.128': 0} for bucket 'GleamBookUsers0' 2022-02-07 23:12:44,417 | infra | INFO | pool-3-thread-19 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304163.67' completed 2022-02-07 23:12:44,796 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 309240 == 0 2022-02-07 23:12:44,798 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 309240 == 0. Received: {'172.23.121.123': 309240} for bucket 'GleamBookUsers0' 2022-02-07 23:12:44,798 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:44,799 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:44,799 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:45,382 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:45,492 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:45,957 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 44951 == 0 2022-02-07 23:12:45,957 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 44951 == 0. Received: {'172.23.121.127': 44951} for bucket 'GleamBookUsers0' 2022-02-07 23:12:45,957 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:45,959 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:45,960 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:46,926 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1170619 == 0 2022-02-07 23:12:46,927 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1170619 == 0. Received: {'172.23.121.115': 1170619} for bucket 'GleamBookUsers0' 2022-02-07 23:12:46,927 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:46,928 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:46,928 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:48,085 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:48,345 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:48,441 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 5044 == 0 2022-02-07 23:12:48,441 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 5044 == 0. Received: {'172.23.120.170': 5044} for bucket 'GleamBookUsers0' 2022-02-07 23:12:48,441 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:48,444 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:48,444 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:48,815 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 58185 == 0 2022-02-07 23:12:48,816 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 58185 == 0. Received: {'172.23.121.124': 58185} for bucket 'GleamBookUsers0' 2022-02-07 23:12:48,816 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:48,821 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:48,821 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:49,411 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:49,801 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:49,971 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 134512 == 0 2022-02-07 23:12:49,973 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 134512 == 0. Received: {'172.23.121.126': 134512} for bucket 'GleamBookUsers0' 2022-02-07 23:12:49,973 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:49,974 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:49,974 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:50,447 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 271726 == 0 2022-02-07 23:12:50,447 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 271726 == 0. Received: {'172.23.121.123': 271726} for bucket 'GleamBookUsers0' 2022-02-07 23:12:50,447 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:50,448 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:50,448 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:50,960 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:51,417 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 38463 == 0 2022-02-07 23:12:51,417 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 38463 == 0. Received: {'172.23.121.127': 38463} for bucket 'GleamBookUsers0' 2022-02-07 23:12:51,417 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:51,420 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:51,420 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:51,928 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:53,352 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1130753 == 0 2022-02-07 23:12:53,354 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1130753 == 0. Received: {'172.23.121.115': 1130753} for bucket 'GleamBookUsers0' 2022-02-07 23:12:53,354 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:53,355 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:53,355 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:53,444 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:53,796 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 3782 == 0 2022-02-07 23:12:53,798 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 3782 == 0. Received: {'172.23.120.170': 3782} for bucket 'GleamBookUsers0' 2022-02-07 23:12:53,798 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:53,799 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:53,799 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:53,822 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:54,257 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 47643 == 0 2022-02-07 23:12:54,257 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 47643 == 0. Received: {'172.23.121.124': 47643} for bucket 'GleamBookUsers0' 2022-02-07 23:12:54,257 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:54,259 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:54,259 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:54,976 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:55,448 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:55,509 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 109428 == 0 2022-02-07 23:12:55,510 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 109428 == 0. Received: {'172.23.121.126': 109428} for bucket 'GleamBookUsers0' 2022-02-07 23:12:55,510 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:55,513 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:55,513 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:56,107 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 259388 == 0 2022-02-07 23:12:56,108 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 259388 == 0. Received: {'172.23.121.123': 259388} for bucket 'GleamBookUsers0' 2022-02-07 23:12:56,108 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:56,109 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:56,111 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:56,420 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:56,865 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 34141 == 0 2022-02-07 23:12:56,865 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 34141 == 0. Received: {'172.23.121.127': 34141} for bucket 'GleamBookUsers0' 2022-02-07 23:12:56,865 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:56,868 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:56,868 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:58,357 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:58,799 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:59,151 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 1458 == 0 2022-02-07 23:12:59,151 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1458 == 0. Received: {'172.23.120.170': 1458} for bucket 'GleamBookUsers0' 2022-02-07 23:12:59,151 | infra | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:59,154 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:59,154 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:59,260 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:12:59,700 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 39183 == 0 2022-02-07 23:12:59,700 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 39183 == 0. Received: {'172.23.121.124': 39183} for bucket 'GleamBookUsers0' 2022-02-07 23:12:59,700 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:59,701 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:59,701 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:12:59,808 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1101351 == 0 2022-02-07 23:12:59,808 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1101351 == 0. Received: {'172.23.121.115': 1101351} for bucket 'GleamBookUsers0' 2022-02-07 23:12:59,808 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:12:59,811 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:12:59,811 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:00,513 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:01,016 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 106194 == 0 2022-02-07 23:13:01,016 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 106194 == 0. Received: {'172.23.121.126': 106194} for bucket 'GleamBookUsers0' 2022-02-07 23:13:01,016 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:01,017 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:01,017 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:01,111 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:01,730 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 255816 == 0 2022-02-07 23:13:01,730 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 255816 == 0. Received: {'172.23.121.123': 255816} for bucket 'GleamBookUsers0' 2022-02-07 23:13:01,730 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:01,732 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:01,732 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:01,868 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:02,298 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 27487 == 0 2022-02-07 23:13:02,298 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 27487 == 0. Received: {'172.23.121.127': 27487} for bucket 'GleamBookUsers0' 2022-02-07 23:13:02,298 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:02,299 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:02,299 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:04,154 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:04,497 | test | DEBUG | pool-3-thread-12 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 23:13:04,499 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.120.170': 0} for bucket 'GleamBookUsers0' 2022-02-07 23:13:04,499 | infra | INFO | pool-3-thread-12 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304160.47' completed 2022-02-07 23:13:04,500 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 23:13:04,500 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304160.94 2022-02-07 23:13:04,703 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:04,811 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:05,125 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 33363 == 0 2022-02-07 23:13:05,125 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 33363 == 0. Received: {'172.23.121.124': 33363} for bucket 'GleamBookUsers0' 2022-02-07 23:13:05,125 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:05,128 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:05,128 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:06,019 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:06,237 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1080533 == 0 2022-02-07 23:13:06,239 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1080533 == 0. Received: {'172.23.121.115': 1080533} for bucket 'GleamBookUsers0' 2022-02-07 23:13:06,239 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:06,240 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:06,240 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:06,520 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 88072 == 0 2022-02-07 23:13:06,520 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 88072 == 0. Received: {'172.23.121.126': 88072} for bucket 'GleamBookUsers0' 2022-02-07 23:13:06,520 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:06,523 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:06,523 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:06,733 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:07,301 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:07,325 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 227694 == 0 2022-02-07 23:13:07,325 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 227694 == 0. Received: {'172.23.121.123': 227694} for bucket 'GleamBookUsers0' 2022-02-07 23:13:07,326 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:07,328 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:07,328 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:07,686 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 17810 == 0 2022-02-07 23:13:07,687 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 17810 == 0. Received: {'172.23.121.127': 17810} for bucket 'GleamBookUsers0' 2022-02-07 23:13:07,687 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:07,688 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:07,688 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:10,130 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:10,536 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 23311 == 0 2022-02-07 23:13:10,538 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 23311 == 0. Received: {'172.23.121.124': 23311} for bucket 'GleamBookUsers0' 2022-02-07 23:13:10,538 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:10,539 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:10,539 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:11,242 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:11,523 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:12,000 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 73194 == 0 2022-02-07 23:13:12,000 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 73194 == 0. Received: {'172.23.121.126': 73194} for bucket 'GleamBookUsers0' 2022-02-07 23:13:12,000 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:12,003 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:12,003 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:12,328 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:12,665 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1080533 == 0 2022-02-07 23:13:12,665 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1080533 == 0. Received: {'172.23.121.115': 1080533} for bucket 'GleamBookUsers0' 2022-02-07 23:13:12,667 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:12,668 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:12,668 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:12,690 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:12,888 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 202862 == 0 2022-02-07 23:13:12,888 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 202862 == 0. Received: {'172.23.121.123': 202862} for bucket 'GleamBookUsers0' 2022-02-07 23:13:12,888 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:12,891 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:12,891 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:13,059 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 10056 == 0 2022-02-07 23:13:13,059 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 10056 == 0. Received: {'172.23.121.127': 10056} for bucket 'GleamBookUsers0' 2022-02-07 23:13:13,059 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:13,062 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:13,062 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:15,540 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:15,941 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 19743 == 0 2022-02-07 23:13:15,941 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 19743 == 0. Received: {'172.23.121.124': 19743} for bucket 'GleamBookUsers0' 2022-02-07 23:13:15,941 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:15,944 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:15,944 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:17,003 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:17,437 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 62064 == 0 2022-02-07 23:13:17,437 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 62064 == 0. Received: {'172.23.121.126': 62064} for bucket 'GleamBookUsers0' 2022-02-07 23:13:17,437 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:17,438 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:17,438 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:17,670 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:17,891 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:18,065 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:18,473 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 180854 == 0 2022-02-07 23:13:18,473 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 180854 == 0. Received: {'172.23.121.123': 180854} for bucket 'GleamBookUsers0' 2022-02-07 23:13:18,473 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:18,476 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:18,476 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:18,487 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 7248 == 0 2022-02-07 23:13:18,487 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 7248 == 0. Received: {'172.23.121.127': 7248} for bucket 'GleamBookUsers0' 2022-02-07 23:13:18,487 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:18,490 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:18,490 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:19,055 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 1033164 == 0 2022-02-07 23:13:19,055 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1033164 == 0. Received: {'172.23.121.115': 1033164} for bucket 'GleamBookUsers0' 2022-02-07 23:13:19,055 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:19,058 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:19,058 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:20,944 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:21,329 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 11406 == 0 2022-02-07 23:13:21,331 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 11406 == 0. Received: {'172.23.121.124': 11406} for bucket 'GleamBookUsers0' 2022-02-07 23:13:21,331 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:21,332 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:21,332 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:22,440 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:22,869 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 62064 == 0 2022-02-07 23:13:22,869 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 62064 == 0. Received: {'172.23.121.126': 62064} for bucket 'GleamBookUsers0' 2022-02-07 23:13:22,869 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:22,871 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:22,871 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:23,476 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:23,490 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:23,855 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 3518 == 0 2022-02-07 23:13:23,855 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 3518 == 0. Received: {'172.23.121.127': 3518} for bucket 'GleamBookUsers0' 2022-02-07 23:13:23,855 | infra | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:23,858 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:23,858 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:24,023 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 158797 == 0 2022-02-07 23:13:24,023 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 158797 == 0. Received: {'172.23.121.123': 158797} for bucket 'GleamBookUsers0' 2022-02-07 23:13:24,023 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:24,026 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:24,026 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:24,059 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:25,378 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 978330 == 0 2022-02-07 23:13:25,380 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 978330 == 0. Received: {'172.23.121.115': 978330} for bucket 'GleamBookUsers0' 2022-02-07 23:13:25,380 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:25,381 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:25,381 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:26,334 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:26,704 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 5918 == 0 2022-02-07 23:13:26,706 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 5918 == 0. Received: {'172.23.121.124': 5918} for bucket 'GleamBookUsers0' 2022-02-07 23:13:26,706 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:26,707 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:26,707 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:27,872 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:28,288 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 42816 == 0 2022-02-07 23:13:28,288 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 42816 == 0. Received: {'172.23.121.126': 42816} for bucket 'GleamBookUsers0' 2022-02-07 23:13:28,288 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:28,289 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:28,289 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:28,858 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:29,026 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:29,224 | test | DEBUG | pool-3-thread-18 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 23:13:29,226 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.127': 0} for bucket 'GleamBookUsers0' 2022-02-07 23:13:29,226 | infra | INFO | pool-3-thread-18 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304163.24' completed 2022-02-07 23:13:29,552 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 149999 == 0 2022-02-07 23:13:29,552 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 149999 == 0. Received: {'172.23.121.123': 149999} for bucket 'GleamBookUsers0' 2022-02-07 23:13:29,552 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:29,553 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:29,553 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:30,381 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:31,697 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 928310 == 0 2022-02-07 23:13:31,697 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 928310 == 0. Received: {'172.23.121.115': 928310} for bucket 'GleamBookUsers0' 2022-02-07 23:13:31,697 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:31,700 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:31,700 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:31,709 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:32,072 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 1014 == 0 2022-02-07 23:13:32,072 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1014 == 0. Received: {'172.23.121.124': 1014} for bucket 'GleamBookUsers0' 2022-02-07 23:13:32,072 | infra | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:32,075 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:32,075 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:33,290 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:33,684 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 27832 == 0 2022-02-07 23:13:33,684 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 27832 == 0. Received: {'172.23.121.126': 27832} for bucket 'GleamBookUsers0' 2022-02-07 23:13:33,684 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:33,687 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:33,687 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:34,555 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:35,046 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 109551 == 0 2022-02-07 23:13:35,046 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 109551 == 0. Received: {'172.23.121.123': 109551} for bucket 'GleamBookUsers0' 2022-02-07 23:13:35,046 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:35,049 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:35,049 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:36,700 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:37,075 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:37,437 | test | DEBUG | pool-3-thread-16 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 23:13:37,437 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.124': 0} for bucket 'GleamBookUsers0' 2022-02-07 23:13:37,437 | infra | INFO | pool-3-thread-16 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304162.34' completed 2022-02-07 23:13:37,924 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 869810 == 0 2022-02-07 23:13:37,926 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 869810 == 0. Received: {'172.23.121.115': 869810} for bucket 'GleamBookUsers0' 2022-02-07 23:13:37,926 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:37,927 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:37,927 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:38,687 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:39,066 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 7614 == 0 2022-02-07 23:13:39,066 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 7614 == 0. Received: {'172.23.121.126': 7614} for bucket 'GleamBookUsers0' 2022-02-07 23:13:39,066 | infra | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:39,069 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:39,069 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:40,049 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:40,444 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 17985 == 0 2022-02-07 23:13:40,444 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 17985 == 0. Received: {'172.23.121.123': 17985} for bucket 'GleamBookUsers0' 2022-02-07 23:13:40,444 | infra | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:40,447 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:40,447 | infra | DEBUG | pool-3-thread-15 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:42,927 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:43,948 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 700186 == 0 2022-02-07 23:13:43,950 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 700186 == 0. Received: {'172.23.121.115': 700186} for bucket 'GleamBookUsers0' 2022-02-07 23:13:43,950 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:43,951 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:43,951 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:44,069 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:44,411 | test | DEBUG | pool-3-thread-17 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 23:13:44,411 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.126': 0} for bucket 'GleamBookUsers0' 2022-02-07 23:13:44,413 | infra | INFO | pool-3-thread-17 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304162.79' completed 2022-02-07 23:13:45,447 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:45,799 | test | DEBUG | pool-3-thread-15 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 23:13:45,799 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.123': 0} for bucket 'GleamBookUsers0' 2022-02-07 23:13:45,799 | infra | INFO | pool-3-thread-15 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304161.86' completed 2022-02-07 23:13:48,953 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:49,734 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 572762 == 0 2022-02-07 23:13:49,734 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 572762 == 0. Received: {'172.23.121.115': 572762} for bucket 'GleamBookUsers0' 2022-02-07 23:13:49,736 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:49,736 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:49,737 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:13:54,737 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:13:55,421 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 425232 == 0 2022-02-07 23:13:55,421 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 425232 == 0. Received: {'172.23.121.115': 425232} for bucket 'GleamBookUsers0' 2022-02-07 23:13:55,421 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:13:55,424 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:13:55,424 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:14:00,424 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:14:00,963 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 286162 == 0 2022-02-07 23:14:00,963 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 286162 == 0. Received: {'172.23.121.115': 286162} for bucket 'GleamBookUsers0' 2022-02-07 23:14:00,963 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:14:00,964 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:14:00,964 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:14:05,966 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:14:06,453 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 121768 == 0 2022-02-07 23:14:06,453 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 121768 == 0. Received: {'172.23.121.115': 121768} for bucket 'GleamBookUsers0' 2022-02-07 23:14:06,453 | infra | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-07 23:14:06,456 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-07 23:14:06,457 | infra | DEBUG | pool-3-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-07 23:14:11,459 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-07 23:14:11,811 | test | DEBUG | pool-3-thread-13 | [task:_compare:2755] Comparing 0 == 0 2022-02-07 23:14:11,812 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.115': 0} for bucket 'GleamBookUsers0' 2022-02-07 23:14:11,812 | infra | INFO | pool-3-thread-13 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304160.94' completed 2022-02-07 23:14:11,812 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 23:14:11,813 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304161.39 2022-02-07 23:14:11,815 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 23:14:11,859 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304161.86 2022-02-07 23:14:11,861 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 23:14:11,861 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304162.34 2022-02-07 23:14:11,861 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 23:14:11,862 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304162.79 2022-02-07 23:14:11,862 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 23:14:11,864 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304163.24 2022-02-07 23:14:11,864 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 23:14:11,864 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304163.67 2022-02-07 23:14:11,865 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 23:14:11,865 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644304164.11 2022-02-07 23:14:11,865 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 23:14:11,937 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:651] All nodes in cluster: [ip:172.23.121.128 port:8091 ssh_username:root, ip:172.23.121.115 port:8091 ssh_username:root, ip:172.23.120.170 port:8091 ssh_username:root, ip:172.23.121.129 port:8091 ssh_username:root, ip:172.23.121.124 port:8091 ssh_username:root, ip:172.23.121.126 port:8091 ssh_username:root, ip:172.23.121.116 port:8091 ssh_username:root, ip:172.23.121.127 port:8091 ssh_username:root, ip:172.23.121.123 port:8091 ssh_username:root] 2022-02-07 23:14:11,940 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 23:14:12,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:12,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 23:14:12,275 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:12,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 23:14:12,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 23:14:12,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 23:14:12,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 23:14:12,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 23:14:12,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 23:14:12,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 23:14:12,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 23:14:12,548 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 23:14:12,551 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 23:14:12,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:12,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 23:14:12,921 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:12,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 23:14:12,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 23:14:12,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 23:14:12,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 23:14:12,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 23:14:12,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 23:14:13,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 23:14:13,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 23:14:13,207 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 23:14:13,210 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 23:14:13,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:13,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 23:14:13,552 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:13,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 23:14:13,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 23:14:13,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 23:14:13,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 23:14:13,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 23:14:13,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 23:14:13,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 23:14:13,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 23:14:13,835 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 23:14:13,836 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 23:14:13,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:14,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 23:14:14,197 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:14,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 23:14:14,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 23:14:14,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 23:14:14,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 23:14:14,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 23:14:14,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 23:14:14,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 23:14:14,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 23:14:14,471 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 23:14:14,476 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 23:14:14,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:14,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 23:14:14,826 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:14,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 23:14:14,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 23:14:14,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 23:14:14,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 23:14:14,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 23:14:14,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 23:14:14,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 23:14:14,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 23:14:15,125 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 23:14:15,131 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 23:14:15,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:15,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 23:14:15,499 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:15,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 23:14:15,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 23:14:15,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 23:14:15,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 23:14:15,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 23:14:15,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 23:14:15,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 23:14:15,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 23:14:15,782 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 23:14:15,785 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 23:14:15,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:16,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 23:14:16,115 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:16,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 23:14:16,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 23:14:16,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 23:14:16,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 23:14:16,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 23:14:16,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 23:14:16,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 23:14:16,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 23:14:16,395 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 23:14:16,398 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 23:14:16,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:16,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 23:14:16,739 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:16,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 23:14:16,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 23:14:16,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 23:14:16,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 23:14:16,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 23:14:16,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 23:14:16,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 23:14:16,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 23:14:17,017 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 23:14:17,020 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 23:14:17,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:17,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 23:14:17,348 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:17,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 23:14:17,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 23:14:17,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 23:14:17,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 23:14:17,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 23:14:17,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 23:14:17,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 23:14:17,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-07 23:14:17,622 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 23:14:17,622 | test | DEBUG | MainThread | [bucket_ready_functions:verify_stats_for_bucket:2318] Verifying stats for bucket GleamBookUsers0 2022-02-07 23:14:17,694 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:651] All nodes in cluster: [ip:172.23.121.128 port:8091 ssh_username:root, ip:172.23.121.115 port:8091 ssh_username:root, ip:172.23.120.170 port:8091 ssh_username:root, ip:172.23.121.129 port:8091 ssh_username:root, ip:172.23.121.124 port:8091 ssh_username:root, ip:172.23.121.126 port:8091 ssh_username:root, ip:172.23.121.116 port:8091 ssh_username:root, ip:172.23.121.127 port:8091 ssh_username:root, ip:172.23.121.123 port:8091 ssh_username:root] 2022-02-07 23:14:17,697 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 23:14:17,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:17,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 23:14:18,035 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:18,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 23:14:18,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 23:14:18,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 23:14:18,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 23:14:18,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 23:14:18,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 23:14:18,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 23:14:18,127 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 23:14:18,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:18,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 23:14:18,470 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:18,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 23:14:18,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 23:14:18,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 23:14:18,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 23:14:18,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 23:14:18,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 23:14:18,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 23:14:18,568 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 23:14:18,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:18,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 23:14:18,917 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:18,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 23:14:18,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 23:14:18,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 23:14:18,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 23:14:18,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 23:14:18,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 23:14:18,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 23:14:19,007 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 23:14:19,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:19,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 23:14:19,345 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:19,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 23:14:19,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 23:14:19,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 23:14:19,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 23:14:19,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 23:14:19,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 23:14:19,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 23:14:19,438 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 23:14:19,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:19,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 23:14:19,776 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:19,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 23:14:19,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 23:14:19,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 23:14:19,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 23:14:19,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 23:14:19,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 23:14:19,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 23:14:19,867 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 23:14:19,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:20,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 23:14:20,200 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:20,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 23:14:20,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 23:14:20,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 23:14:20,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 23:14:20,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 23:14:20,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 23:14:20,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 23:14:20,292 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 23:14:20,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:20,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 23:14:20,631 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:20,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 23:14:20,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 23:14:20,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 23:14:20,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 23:14:20,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 23:14:20,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 23:14:20,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 23:14:20,726 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 23:14:20,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:20,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 23:14:21,055 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:21,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 23:14:21,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 23:14:21,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 23:14:21,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 23:14:21,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 23:14:21,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 23:14:21,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 23:14:21,151 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 23:14:21,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:21,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 23:14:21,492 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:21,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 23:14:21,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 23:14:21,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 23:14:21,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 23:14:21,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 23:14:21,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 23:14:21,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 23:14:21,596 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 23:14:21,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:21,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 23:14:21,927 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:21,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 23:14:21,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 23:14:21,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 23:14:21,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 23:14:21,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 23:14:22,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 23:14:22,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 23:14:22,032 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 23:14:22,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:22,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 23:14:22,359 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:22,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 23:14:22,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 23:14:22,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 23:14:22,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 23:14:22,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 23:14:22,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 23:14:22,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 23:14:22,456 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 23:14:22,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:22,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 23:14:22,792 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:22,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 23:14:22,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 23:14:22,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 23:14:22,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 23:14:22,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 23:14:22,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 23:14:22,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 23:14:22,884 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 23:14:22,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:23,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 23:14:23,239 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:23,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 23:14:23,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 23:14:23,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 23:14:23,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 23:14:23,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 23:14:23,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 23:14:23,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 23:14:23,332 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 23:14:23,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:23,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 23:14:23,647 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:23,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 23:14:23,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 23:14:23,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 23:14:23,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 23:14:23,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 23:14:23,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 23:14:23,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 23:14:23,740 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 23:14:23,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:24,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 23:14:24,094 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:24,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 23:14:24,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 23:14:24,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 23:14:24,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 23:14:24,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 23:14:24,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 23:14:24,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 23:14:24,184 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 23:14:24,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:24,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 23:14:24,490 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:24,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 23:14:24,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 23:14:24,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 23:14:24,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 23:14:24,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 23:14:24,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 23:14:24,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 23:14:24,578 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 23:14:24,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:24,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 23:14:24,904 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:24,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 23:14:24,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 23:14:24,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 23:14:24,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 23:14:24,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 23:14:24,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 23:14:24,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 23:14:24,993 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 23:14:25,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:25,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 23:14:25,322 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:25,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 23:14:25,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 23:14:25,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 23:14:25,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 23:14:25,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 23:14:25,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 23:14:25,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 23:14:25,411 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for vb_replica_curr_items on bucket GleamBookUsers0 2022-02-07 23:14:25,413 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_vb_replica_curr_items_1644304465.41 2022-02-07 23:14:25,413 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for curr_items_tot on bucket GleamBookUsers0 2022-02-07 23:14:25,413 | infra | INFO | pool-3-thread-25 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_vb_replica_curr_items_1644304465.41' started 2022-02-07 23:14:25,413 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_curr_items_tot_1644304465.41 2022-02-07 23:14:25,414 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_vb_replica_curr_items_1644304465.41 2022-02-07 23:14:25,414 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:25,414 | infra | INFO | pool-3-thread-21 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_curr_items_tot_1644304465.41' started 2022-02-07 23:14:25,415 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:25,551 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:25,555 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:25,696 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:25,696 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:25,835 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:25,838 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:25,980 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:25,980 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:26,118 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:26,124 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:26,259 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:26,259 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:26,395 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:26,398 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:26,529 | infra | DEBUG | pool-3-thread-25 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:26,532 | infra | DEBUG | pool-3-thread-21 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-07 23:14:26,664 | test | DEBUG | pool-3-thread-25 | [task:_compare:2755] Comparing 40000000 == 40000000 2022-02-07 23:14:26,665 | test | DEBUG | pool-3-thread-25 | [task:_get_all_stats_and_compare:2707] Ready: vb_replica_curr_items 40000000 == 40000000. Received: {'172.23.121.123': '4447200', '172.23.121.124': '4462540', '172.23.121.115': '4450040', '172.23.121.126': '4445220', '172.23.121.116': '4457960', '172.23.121.127': '4477040', '172.23.121.128': '4398720', '172.23.121.129': '4408860', '172.23.120.170': '4452420'} for bucket 'GleamBookUsers0' 2022-02-07 23:14:26,665 | infra | INFO | pool-3-thread-25 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_vb_replica_curr_items_1644304465.41' completed 2022-02-07 23:14:26,665 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_curr_items_tot_1644304465.41 2022-02-07 23:14:26,667 | test | DEBUG | pool-3-thread-21 | [task:_compare:2755] Comparing 80000000 == 80000000 2022-02-07 23:14:26,667 | test | DEBUG | pool-3-thread-21 | [task:_get_all_stats_and_compare:2707] Ready: curr_items_tot 80000000 == 80000000. Received: {'172.23.121.123': '8912760', '172.23.121.124': '8908380', '172.23.121.115': '8893920', '172.23.121.126': '8891820', '172.23.121.116': '8918580', '172.23.121.127': '8932540', '172.23.121.128': '8807560', '172.23.121.129': '8819600', '172.23.120.170': '8914840'} for bucket 'GleamBookUsers0' 2022-02-07 23:14:26,668 | infra | INFO | pool-3-thread-21 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_curr_items_tot_1644304465.41' completed 2022-02-07 23:14:26,668 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 23:14:26,668 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 23:14:26,668 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 23:14:26,670 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 23:14:26,670 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 23:14:26,670 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 23:14:26,670 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 23:14:26,670 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 23:14:26,670 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 23:14:26,671 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 23:14:26,671 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 23:14:26,671 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 23:14:26,671 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 23:14:26,671 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 23:14:26,671 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 23:14:26,671 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 23:14:26,671 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 23:14:26,671 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 23:14:26,734 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3970] Stats: {u'opsPerSec': 0, u'diskFetches': 0, u'dataUsed': 66614273514L, u'itemCount': 40000000, u'quotaPercentUsed': 73.24689626420822, u'storageTotals': {u'hdd': {u'usedByData': 97007713739L, u'used': 192267250230L, u'quotaTotal': 1168799956992L, u'total': 1168799956992L, u'free': 848842070265L}, u'ram': {u'usedByData': 57034578536L, u'used': 99219779584L, u'quotaTotal': 78809923584L, u'total': 110647066624L, u'quotaUsed': 77866205184L, u'quotaUsedPerNode': 8651800576L, u'quotaTotalPerNode': 8756658176L}}, u'diskUsed': 97007713739L, u'vbActiveNumNonResident': 0, u'memUsed': 57034578536L} 2022-02-07 23:14:26,980 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+-----+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 40000000 | 72.52 GiB | 53.12 GiB | 90.35 GiB | 100 | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+-----+ 2022-02-07 23:14:26,980 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+---------------+-----------------+---------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+---------------+-----------------+---------------+---------------+ | 20000000 | 40000000 | 0-0 | 1000000-2000000 | 0-0 | 0-0 | +---------------+---------------+---------------+-----------------+---------------+---------------+ 2022-02-07 23:14:27,249 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 100 2022-02-07 23:14:27,250 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 23:14:27,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:27,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 23:14:27,598 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:27,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 23:14:27,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 23:14:27,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 23:14:27,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 23:14:27,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 23:14:27,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 23:14:27,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 23:14:27,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 23:14:27,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 23:14:27,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 23:14:27,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 23:14:27,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 23:14:28,085 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 23:14:28,086 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 23:14:28,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:28,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 23:14:28,417 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:28,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 23:14:28,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 23:14:28,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 23:14:28,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 23:14:28,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 23:14:28,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 23:14:28,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 23:14:28,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 23:14:28,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 23:14:28,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 23:14:28,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 23:14:28,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 23:14:28,880 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 23:14:28,881 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 23:14:28,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:29,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 23:14:29,223 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:29,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 23:14:29,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 23:14:29,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 23:14:29,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 23:14:29,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 23:14:29,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 23:14:29,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 23:14:29,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 23:14:29,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 23:14:29,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 23:14:29,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 23:14:29,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 23:14:29,778 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 23:14:29,782 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 23:14:29,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:30,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 23:14:30,144 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:30,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 23:14:30,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 23:14:30,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 23:14:30,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 23:14:30,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 23:14:30,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 23:14:30,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 23:14:30,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 23:14:30,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 23:14:30,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 23:14:30,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 23:14:30,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 23:14:30,634 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 23:14:30,637 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 23:14:30,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:30,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 23:14:30,976 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:30,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 23:14:30,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 23:14:31,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 23:14:31,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 23:14:31,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 23:14:31,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 23:14:31,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 23:14:31,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 23:14:31,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 23:14:31,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 23:14:31,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 23:14:31,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 23:14:31,484 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 23:14:31,486 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 23:14:31,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:31,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 23:14:31,828 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:31,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 23:14:31,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 23:14:31,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 23:14:31,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 23:14:31,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 23:14:31,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 23:14:31,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 23:14:31,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 23:14:32,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 23:14:32,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 23:14:32,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 23:14:32,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 23:14:32,289 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 23:14:32,292 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 23:14:32,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:32,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 23:14:32,619 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:32,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 23:14:32,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 23:14:32,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 23:14:32,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 23:14:32,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 23:14:32,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 23:14:32,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 23:14:32,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 23:14:32,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 23:14:32,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 23:14:32,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 23:14:32,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 23:14:33,091 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 23:14:33,094 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 23:14:33,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:33,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 23:14:33,447 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:33,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 23:14:33,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 23:14:33,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 23:14:33,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 23:14:33,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 23:14:33,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 23:14:33,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 23:14:33,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 23:14:33,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 23:14:33,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 23:14:33,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 23:14:33,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 23:14:33,928 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 23:14:33,930 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 23:14:34,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:34,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 23:14:34,259 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:34,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 23:14:34,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 23:14:34,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 23:14:34,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 23:14:34,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 23:14:34,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 23:14:34,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 23:14:34,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 23:14:34,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 23:14:34,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 23:14:34,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 23:14:34,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 23:14:34,750 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 23:14:34,752 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-07 23:14:34,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:35,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-07 23:14:35,096 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:35,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-07 23:14:35,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-07 23:14:35,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-07 23:14:35,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-07 23:14:35,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-07 23:14:35,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-07 23:14:35,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-07 23:14:35,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 23:14:35,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 23:14:35,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 23:14:35,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 23:14:35,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 23:14:35,305 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-07 23:14:35,308 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-07 23:14:35,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:35,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-07 23:14:35,651 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:35,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-07 23:14:35,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-07 23:14:35,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-07 23:14:35,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-07 23:14:35,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-07 23:14:35,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-07 23:14:35,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-07 23:14:35,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 23:14:35,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 23:14:35,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 23:14:35,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 23:14:35,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 23:14:35,845 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-07 23:14:35,846 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-07 23:14:35,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:36,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-07 23:14:36,187 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:36,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-07 23:14:36,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-07 23:14:36,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-07 23:14:36,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-07 23:14:36,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-07 23:14:36,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-07 23:14:36,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-07 23:14:36,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 23:14:36,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 23:14:36,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 23:14:36,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 23:14:36,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 23:14:36,365 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-07 23:14:36,367 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-07 23:14:36,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:36,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-07 23:14:36,721 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:36,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-07 23:14:36,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-07 23:14:36,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-07 23:14:36,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-07 23:14:36,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-07 23:14:36,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-07 23:14:36,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-07 23:14:36,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 23:14:36,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 23:14:36,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 23:14:36,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 23:14:36,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 23:14:36,904 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-07 23:14:36,907 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-07 23:14:36,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:37,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-07 23:14:37,265 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:37,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-07 23:14:37,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-07 23:14:37,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-07 23:14:37,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-07 23:14:37,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-07 23:14:37,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-07 23:14:37,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-07 23:14:37,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-07 23:14:37,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-07 23:14:37,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-07 23:14:37,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-07 23:14:37,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-07 23:14:37,453 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-07 23:14:37,453 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for kvstore is 88042MB 2022-02-07 23:14:37,453 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for wal is 3575MB 2022-02-07 23:14:37,453 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:642] Total Disk usage for keyTree is 7805MB 2022-02-07 23:14:37,453 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:643] Total Disk usage for seqTree is 80013MB 2022-02-07 23:14:37,456 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 23:14:37,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:37,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 23:14:37,793 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:37,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 23:14:37,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 23:14:37,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 23:14:37,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 23:14:37,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 23:14:37,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 23:14:37,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 23:14:37,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 23:14:37,923 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.115 - core(s): 8 2022-02-07 23:14:37,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 23:14:38,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 23:14:38,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 23:14:38,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 23:14:38,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 23:14:38,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 23:14:38,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 23:14:38,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 23:14:38,977 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 23:14:38,980 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 23:14:39,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:39,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 23:14:39,311 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:39,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 23:14:39,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 23:14:39,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 23:14:39,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 23:14:39,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 23:14:39,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 23:14:39,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 23:14:39,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 23:14:39,417 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.116 - core(s): 8 2022-02-07 23:14:39,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 23:14:39,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 23:14:39,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 23:14:39,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 23:14:39,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 23:14:40,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 23:14:40,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 23:14:40,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 23:14:40,486 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 23:14:40,489 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 23:14:40,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:40,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 23:14:40,841 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:40,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 23:14:40,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 23:14:40,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 23:14:40,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 23:14:40,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 23:14:40,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 23:14:40,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 23:14:40,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 23:14:40,944 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.123 - core(s): 8 2022-02-07 23:14:40,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 23:14:41,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 23:14:41,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 23:14:41,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 23:14:41,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 23:14:41,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 23:14:41,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 23:14:41,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 23:14:42,009 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 23:14:42,013 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 23:14:42,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:42,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 23:14:42,357 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:42,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 23:14:42,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 23:14:42,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 23:14:42,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 23:14:42,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 23:14:42,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 23:14:42,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 23:14:42,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 23:14:42,479 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.124 - core(s): 8 2022-02-07 23:14:42,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 23:14:42,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 23:14:42,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 23:14:42,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 23:14:43,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 23:14:43,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 23:14:43,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 23:14:43,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 23:14:43,546 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 23:14:43,548 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 23:14:43,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:43,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 23:14:43,885 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:43,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 23:14:43,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 23:14:43,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 23:14:43,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 23:14:43,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 23:14:43,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 23:14:43,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 23:14:43,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 23:14:43,986 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.126 - core(s): 8 2022-02-07 23:14:43,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 23:14:44,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 23:14:44,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 23:14:44,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 23:14:44,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 23:14:44,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 23:14:44,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 23:14:44,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 23:14:45,075 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 23:14:45,078 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 23:14:45,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:45,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 23:14:45,411 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:45,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 23:14:45,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 23:14:45,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 23:14:45,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 23:14:45,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 23:14:45,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 23:14:45,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 23:14:45,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 23:14:45,513 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.127 - core(s): 8 2022-02-07 23:14:45,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 23:14:45,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 23:14:45,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 23:14:45,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 23:14:46,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 23:14:46,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 23:14:46,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 23:14:46,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 23:14:46,572 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 23:14:46,578 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 23:14:46,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:46,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 23:14:46,892 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:46,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 23:14:46,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 23:14:46,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 23:14:46,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 23:14:46,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 23:14:46,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 23:14:46,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 23:14:46,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 23:14:46,994 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.128 - core(s): 8 2022-02-07 23:14:46,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 23:14:47,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 23:14:47,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 23:14:47,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 23:14:47,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 23:14:47,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 23:14:47,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 23:14:47,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 23:14:48,058 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 23:14:48,061 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 23:14:48,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:48,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 23:14:48,381 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:48,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 23:14:48,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 23:14:48,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 23:14:48,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 23:14:48,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 23:14:48,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 23:14:48,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 23:14:48,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 23:14:48,500 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.129 - core(s): 8 2022-02-07 23:14:48,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 23:14:48,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 23:14:48,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 23:14:48,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 23:14:49,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 23:14:49,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 23:14:49,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 23:14:49,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 23:14:49,573 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 23:14:49,575 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 23:14:49,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:49,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 23:14:49,910 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:49,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 23:14:49,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 23:14:49,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 23:14:49,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 23:14:49,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 23:14:49,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 23:14:49,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 23:14:50,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-07 23:14:50,019 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.120.170 - core(s): 8 2022-02-07 23:14:50,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-07 23:14:50,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-07 23:14:50,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-07 23:14:50,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-07 23:14:50,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-07 23:14:50,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-07 23:14:50,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-07 23:14:50,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-07 23:14:51,089 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 23:14:51,091 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:723] {'172.23.121.115': {'rw_0:magma': {u'NReadBytesGetPerSec': 0.0, u'ReplayWALOffset': u'164:12836864', u'NTablesCreated': 4708, u'RecentReadIOAmpSet': 0.0, u'ReadIOAmp': 308.2113037109375, u'BlockCacheMisses': 84411, u'NFlushes': 2315, u'RecentWriteAmp': 0.0, u'NumSyncFlushes': 0, u'TotalMemUsed': 39867289, u'NBloomFilterHits': 18662884, u'Fragmentation': 0.2201959192752838, u'NumNormalFlushes': 450, u'RecentBlockCacheHitRatio': 0.0, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 2736, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 4693800, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 7963551, u'CheckpointOverhead': 122880, u'NItemsGet': 0, u'NWriteIOs': 8939, u'NReadBytes': 1595768139, u'LevelStats': [{u'TargetSize': 121634816, u'LogicalSize': 3125774, u'SizeRatio': 0.025698020540434737, u'NWriteBytes': 43606016, u'BloomFilterMemUsed': 0, u'NumTables': 2, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 90112}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 93750312, u'SizeRatio': 0.7707522819782126, u'NWriteBytes': 39022592, u'BloomFilterMemUsed': 0, u'NumTables': 58, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 1540096}, {u'TargetSize': 515238514, u'LogicalSize': 236581419, u'SizeRatio': 0.45916873947043485, u'NWriteBytes': 30457856, u'BloomFilterMemUsed': 0, u'NumTables': 111, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 3694592}, {u'TargetSize': 2053458854, u'LogicalSize': 1030477038, u'SizeRatio': 0.5018250236632207, u'NWriteBytes': 2567647232L, u'BloomFilterMemUsed': 0, u'NumTables': 284, u'WastedSpace': 3747611, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 1030438912}], u'NItemsGarbageCollected': 1511590, u'NInternalFragmentationCompacts': 6, u'NTTLCompacts': 14, u'ActiveIndexBlocksSize': 7962617, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 43968, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 604749, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 129156, u'CompactWriter': {u'NDataBlockWrites': 287939, u'NWriteBytes': 846700544, u'NIndexBlockWrites': 3618, u'NBytesIncoming': 1346663445, u'NSyncs': 0, u'NInserts': 3570777, u'NWriteIOs': 2889}, u'NonResidentBloomFilterSize': 0, u'IteratorMemUsed': 0, u'NWriteBytes': 2647238907L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 1511590, u'LogicalDiskSize': 1030477038, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.8289974516538169, u'BufferMemUsed': 344064, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 793895619, u'NTableFiles': 479, u'NFileCountCompacts': 0, u'TableObjectMemUsed': 109920, u'TreeSnapshotMemoryUsed': 170128, u'TotalDiskUsage': 1035886592, u'BytesIncoming': 3193301622L, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 1144, u'NItemsPut': 4270794, u'NSyncs': 1865, u'CompactReader': {u'NReadIOs': 604749, u'NReadBytes': 1595768139, u'NDataBlockReads': 598661, u'NBytesOutgoing': 0, u'NIndexBlockReads': 6088}, u'ActiveDiskUsage': 1035763712, u'NTablesDeleted': 1759, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 690498, u'NWriteBytes': 1800538363, u'NIndexBlockWrites': 6681, u'NBytesIncoming': 3249495672L, u'NSyncs': 1865, u'NInserts': 5267253, u'NWriteIOs': 6050}}, u'NReadBytes': 538553466498L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 0.0, u'NInternalFragmentationCompacts': 6, u'NTTLCompacts': 93, u'ActiveIndexBlocksSize': 8928921, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 28016230, u'NTxns': 57363, u'NWriteBytes': 3416231936L, u'NSegments': 169, u'NSyncs': 57363, u'NLoggedBytes': 3416231936L, u'NActiveSegments': 6, u'NWrites': 72387, u'PreallocateSupported': True, u'NInserts': 3802543, u'DiskUsed': 122826752}, u'NGets': 548580, u'NReadBytesSet': 146359766, u'NReadBytesCompact': 1694759867, u'IndexResidentRatio': 0.5025805830955505, u'CompactQueueSize': 0, u'NSyncsPerSec': 0.6000000238418579, u'BlockCacheHitRatio': 97.45613098144531, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 536858706631L, u'BloomFilterMemUsedRatio': 0.014441903680562973, u'NonResidentBloomFilterSize': 43042, u'NWriteBytes': 6391714413L, u'BytesOutgoing': 0, u'RecentReadIOAmp': 0.0, u'WriteAmp': 2.0138962268829346, u'NBlocksCached': 7599, u'LogicalDataSize': 837831876, u'ActiveBloomFilterMemUsed': 1561856, u'NFileCountCompacts': 64, u'NTableFiles': 663, u'FlushQueueSize': 8, u'TableObjectMemUsed': 154080, u'BlockCacheQuota': 80653765, u'TotalDiskUsage': 1120657408, u'NPurgerRuns': 29, u'ReadIOAmpSet': 0.0, u'NSyncs': 5595, u'NReadIOsPerSec': 0.0, u'NReadIOsSetPerSec': 0.0, u'NTablesDeleted': 3591, u'BloomFilterFPR': 0.015341832302510738, u'NBloomFilterHitsPerSec': 0.0, u'WriteCacheMemUsed': 5234040, u'RecentBloomFilterFPR': 0.0, u'BytesPerRead': 978366.625, u'BytesIncomingPerSec': 0.0, u'NReadIOsGetPerSec': 0.0, u'NumKVStores': 29, u'NGetStats': 17325, u'TotalIndexBlocksSize': 8959719, u'NGetStatsPerSec': 1.2333333492279053, u'NWriteBytesCompact': 935460864, u'NReadIOsSet': 42412, u'ActiveDataSize': 837831876, u'NWriteBytesCompactPerSec': 0.0, u'NBloomFilterMissesPerSec': 0.0, u'NCommitBatches': 57363, u'NGetStatsComputed': 17325, u'NWriteIOs': 87332, u'NReadIOsGet': 169120966, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 20163442, u'LSMTreeObjectMemUsed': 500192, u'NSetsPerSec': 0.0, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.10000000149011612, u'ManagedFileObjectMemUsed': 61632, u'ReadAmpGet': 0.0, u'BlockCacheMemUsed': 25938040, u'NWriterCompacts': 0, u'NReadIOs': 169752911, u'WALBufferMemUsed': 5242880, u'ReadAmp': 0.0, u'keyStats': {u'NItemsTransformed': 1947941, u'NTablesCreated': 986, u'ReadIOAmp': 42.77945979581719, u'MemtablesMemUsed': 450344, u'NBloomFilterHits': 18662884, u'TotalIndexBlocksSize': 990462, u'CheckpointOverhead': 2375680, u'NItemsGet': 3953320, u'NWriteIOs': 3155, u'NReadBytes': 536956787463L, u'LevelStats': [{u'TargetSize': 121634816, u'LogicalSize': 37739601, u'SizeRatio': 0.3102697257337899, u'NWriteBytes': 230305792, u'BloomFilterMemUsed': 749056, u'NumTables': 41, u'WastedSpace': 23989, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 37736448}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 43646162, u'SizeRatio': 0.35882951473367625, u'NWriteBytes': 88154112, u'BloomFilterMemUsed': 811008, u'NumTables': 29, u'WastedSpace': 95022, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 43741184}], u'NItemsGarbageCollected': 1867031, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 37, u'ActiveIndexBlocksSize': 962654, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 6912, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 169146311, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 1000977, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 21780, u'CompactWriter': {u'NDataBlockWrites': 30113, u'NWriteBytes': 88154112, u'NIndexBlockWrites': 454, u'NBytesIncoming': 113611668, u'NSyncs': 0, u'NInserts': 1947941, u'NWriteIOs': 204}, u'NonResidentBloomFilterSize': 42342, u'IteratorMemUsed': 0, u'NWriteBytes': 319437023, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 169120894, u'NReadBytes': 536858669989L, u'NDataBlockReads': 4038648747L, u'NBytesOutgoing': 0, u'NIndexBlockReads': 7590731544L}, u'NBloomFilterMisses': 290784, u'NItemsMerged': 0, u'LogicalDiskSize': 43646162, u'TotalBloomFilterMemUsed': 1560064, u'WriteAmp': 1.8764637129044857, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1560064, u'LogicalDataSize': 43646162, u'NTableFiles': 72, u'NFileCountCompacts': 32, u'TableObjectMemUsed': 17280, u'TreeSnapshotMemoryUsed': 132864, u'TotalDiskUsage': 83853312, u'BytesIncoming': 170233520, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 74, u'NItemsPut': 3404740, u'NSyncs': 1865, u'CompactReader': {u'NReadIOs': 25417, u'NReadBytes': 98117474, u'NDataBlockReads': 59759, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1445}, u'ActiveDiskUsage': 81477632, u'NTablesDeleted': 916, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.015341832514951723, u'Writer': {u'NDataBlockWrites': 53390, u'NWriteBytes': 231282911, u'NIndexBlockWrites': 1364, u'NBytesIncoming': 200609618, u'NSyncs': 1865, u'NInserts': 3400222, u'NWriteIOs': 2951}}, u'NReadBytesCompactPerSec': 0.0, u'TableMetaMemUsed': 183297, u'NDeletes': 0, u'BlockCacheHitsPerSec': 0.0, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 290784, u'LogicalDiskSize': 1074413295, u'NSets': 3404740, u'TotalBloomFilterMemUsed': 1561856, u'BufferMemUsed': 344064, u'RecentReadAmp': 0.0, u'NGetsPerSec': 0.0, u'NInserts': 0, u'NCommitBatchesPerSec': 0.0, u'FragmentationWithOverhead': 0.25237464904785156, u'NCompactsPerSec': 0.0, u'WALMemUsed': 5308496, u'MemoryQuota': 108147512, u'NReadBytesPerSec': 0.0, u'TreeSnapshotMemoryUsed': 397888, u'BytesIncoming': 3173805411L, u'NCompacts': 1292, u'NumPersistentFlushes': 1865, u'NWriteIOsPerSec': 0.8666666746139526, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 0.0, u'ActiveDiskUsage': 1117814784, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 1.2333333492279053, u'NReadBytesSetPerSec': 0.0, u'BlockCacheHits': 3233799, u'NWriteBytesPerSec': 76477.765625, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 986, u'ReadIOAmp': 0.0989010989010989, u'MemtablesMemUsed': 89896, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 5706, u'CheckpointOverhead': 344064, u'NItemsGet': 728, u'NWriteIOs': 2851, u'NReadBytes': 910896, u'LevelStats': [{u'TargetSize': 121634816, u'LogicalSize': 386247, u'SizeRatio': 0.0031754641697324558, u'NWriteBytes': 7471104, u'BloomFilterMemUsed': 864, u'NumTables': 41, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 335872}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 290095, u'SizeRatio': 0.0023849668173954405, u'NWriteBytes': 606208, u'BloomFilterMemUsed': 928, u'NumTables': 29, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 237568}], u'NItemsGarbageCollected': 224482, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 42, u'ActiveIndexBlocksSize': 3650, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 10752, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 1851, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 32361, u'CompactWriter': {u'NDataBlockWrites': 74, u'NWriteBytes': 606208, u'NIndexBlockWrites': 74, u'NBytesIncoming': 163683, u'NSyncs': 0, u'NInserts': 1702, u'NWriteIOs': 74}, u'NonResidentBloomFilterSize': 700, u'IteratorMemUsed': 0, u'NWriteBytes': 8806547, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 72, u'NReadBytes': 36642, u'NDataBlockReads': 36, u'NBytesOutgoing': 0, u'NIndexBlockReads': 36}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 290095, u'TotalBloomFilterMemUsed': 1792, u'WriteAmp': 0.2699790563098228, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1792, u'LogicalDataSize': 290095, u'NTableFiles': 112, u'NFileCountCompacts': 32, u'TableObjectMemUsed': 26880, u'TreeSnapshotMemoryUsed': 94896, u'TotalDiskUsage': 917504, u'BytesIncoming': 32619371, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 74, u'NItemsPut': 225756, u'NSyncs': 1865, u'CompactReader': {u'NReadIOs': 1779, u'NReadBytes': 874254, u'NDataBlockReads': 916, u'NBytesOutgoing': 0, u'NIndexBlockReads': 916}, u'ActiveDiskUsage': 573440, u'NTablesDeleted': 916, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 912, u'NWriteBytes': 8200339, u'NIndexBlockWrites': 912, u'NBytesIncoming': 742106, u'NSyncs': 1865, u'NInserts': 10988, u'NWriteIOs': 2777}}}}} 2022-02-07 23:14:51,094 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:728] magma stats fragmentation result {'172.23.121.123': [0.1782645285129547, 0.23858097195625305, 0.15281358361244202, 0.16190221905708313, 0.18588849902153015, 0.16349110007286072, 0.1886064112186432, 0.2219725400209427], '172.23.121.124': [0.16169682145118713, 0.18611149489879608, 0.18264727294445038, 0.16801151633262634, 0.1722380369901657, 0.15072397887706757, 0.20590806007385254, 0.16497424244880676], '172.23.121.115': [0.2201959192752838, 0.21871459484100342, 0.283067911863327, 0.20011775195598602, 0.2511756718158722, 0.1587427854537964, 0.27344244718551636, 0.24830594658851624], '172.23.121.126': [0.1654670536518097, 0.19553643465042114, 0.15852239727973938, 0.23554667830467224, 0.16526727378368378, 0.22910666465759277, 0.15949684381484985, 0.18074505031108856], '172.23.121.116': [0.37163248658180237, 0.3823547959327698, 0.36743801832199097, 0.39249441027641296, 0.37371087074279785, 0.3627052903175354, 0.3618893027305603, 0.38771262764930725], '172.23.121.127': [0.16963420808315277, 0.1593071073293686, 0.17411069571971893, 0.19940361380577087, 0.1895279884338379, 0.19748088717460632, 0.1842927485704422, 0.1762986183166504], '172.23.121.128': [0.16649942100048065, 0.17814883589744568, 0.1686619222164154, 0.15694406628608704, 0.18694359064102173, 0.1810196042060852, 0.1906876116991043, 0.18903012573719025], '172.23.121.129': [0.20536884665489197, 0.17396175861358643, 0.15772318840026855, 0.15458494424819946, 0.1742992401123047, 0.16447344422340393, 0.1757633239030838, 0.14664499461650848], '172.23.120.170': [0.1743467003107071, 0.17853394150733948, 0.17091242969036102, 0.15023380517959595, 0.14665423333644867, 0.16848410665988922, 0.1675061285495758, 0.14920541644096375]} 2022-02-07 23:14:51,345 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.115 is 25.0 2022-02-07 23:14:51,588 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.116 is 25.0 2022-02-07 23:14:51,826 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.123 is 25.0 2022-02-07 23:14:52,078 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.124 is 25.0 2022-02-07 23:14:52,315 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.126 is 25.0 2022-02-07 23:14:52,561 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.127 is 25.0 2022-02-07 23:14:52,822 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.128 is 25.0 2022-02-07 23:14:53,062 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.129 is 25.0 2022-02-07 23:14:53,334 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.120.170 is 25.0 2022-02-07 23:14:53,334 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:677] KV stats fragmentation values {'172.23.121.123': 25.0, '172.23.121.124': 25.0, '172.23.121.115': 25.0, '172.23.121.126': 25.0, '172.23.121.116': 25.0, '172.23.121.127': 25.0, '172.23.121.128': 25.0, '172.23.121.129': 25.0, '172.23.120.170': 25.0} 2022-02-07 23:14:53,335 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-07 23:14:53,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:53,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-07 23:14:53,667 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:53,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-07 23:14:53,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-07 23:14:53,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-07 23:14:53,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-07 23:14:53,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-07 23:14:53,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-07 23:14:53,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-07 23:14:53,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 23:14:53,779 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-07 23:14:53,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 23:14:53,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:14:53,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 23:14:53,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:14:54,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:14:54,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 23:14:54,104 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-07 23:14:54,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 23:14:54,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:14:54,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 23:14:54,144 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-07 23:14:54,145 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-07 23:14:54,148 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-07 23:14:54,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:54,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-07 23:14:54,496 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:54,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-07 23:14:54,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-07 23:14:54,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-07 23:14:54,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-07 23:14:54,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-07 23:14:54,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-07 23:14:54,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-07 23:14:54,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 23:14:54,604 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-07 23:14:54,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 23:14:54,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:14:54,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 23:14:55,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:14:55,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:14:55,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 23:14:55,233 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-07 23:14:55,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 23:14:55,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:14:55,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 23:14:55,273 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-07 23:14:55,275 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-07 23:14:55,276 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-07 23:14:55,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:55,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-07 23:14:55,611 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:55,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-07 23:14:55,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-07 23:14:55,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-07 23:14:55,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-07 23:14:55,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-07 23:14:55,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-07 23:14:55,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-07 23:14:55,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 23:14:55,714 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-07 23:14:55,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 23:14:55,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:14:55,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 23:14:55,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:14:55,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:14:55,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 23:14:55,897 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-07 23:14:55,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 23:14:55,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:14:55,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 23:14:55,934 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-07 23:14:55,936 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-07 23:14:55,937 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-07 23:14:56,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:56,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-07 23:14:56,305 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:56,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-07 23:14:56,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-07 23:14:56,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-07 23:14:56,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-07 23:14:56,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-07 23:14:56,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-07 23:14:56,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-07 23:14:56,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 23:14:56,413 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-07 23:14:56,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 23:14:56,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:14:56,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 23:14:56,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:14:56,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:14:56,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 23:14:56,819 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-07 23:14:56,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 23:14:56,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:14:56,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 23:14:56,858 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-07 23:14:56,858 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-07 23:14:56,861 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-07 23:14:56,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:57,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-07 23:14:57,207 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:57,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-07 23:14:57,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-07 23:14:57,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-07 23:14:57,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-07 23:14:57,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-07 23:14:57,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-07 23:14:57,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-07 23:14:57,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 23:14:57,328 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-07 23:14:57,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 23:14:57,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:14:57,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 23:14:57,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:14:57,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:14:57,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 23:14:57,697 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-07 23:14:57,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 23:14:57,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:14:57,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 23:14:57,746 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-07 23:14:57,746 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-07 23:14:57,750 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-07 23:14:57,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:58,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-07 23:14:58,091 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:58,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-07 23:14:58,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-07 23:14:58,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-07 23:14:58,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-07 23:14:58,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-07 23:14:58,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-07 23:14:58,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-07 23:14:58,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 23:14:58,194 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-07 23:14:58,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 23:14:58,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:14:58,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 23:14:58,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:14:58,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:14:58,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 23:14:58,602 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-07 23:14:58,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 23:14:58,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:14:58,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 23:14:58,642 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-07 23:14:58,642 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-07 23:14:58,644 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-07 23:14:58,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:58,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-07 23:14:58,963 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:58,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-07 23:14:58,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-07 23:14:58,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-07 23:14:59,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-07 23:14:59,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-07 23:14:59,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-07 23:14:59,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-07 23:14:59,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 23:14:59,063 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-07 23:14:59,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 23:14:59,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:14:59,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 23:14:59,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:14:59,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:14:59,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 23:14:59,466 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-07 23:14:59,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 23:14:59,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:14:59,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 23:14:59,506 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-07 23:14:59,506 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-07 23:14:59,509 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-07 23:14:59,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:14:59,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-07 23:14:59,855 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:14:59,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-07 23:14:59,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-07 23:14:59,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-07 23:14:59,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-07 23:14:59,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-07 23:14:59,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-07 23:14:59,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-07 23:14:59,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 23:14:59,951 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-07 23:14:59,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 23:14:59,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:15:00,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 23:15:00,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:15:00,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:15:00,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 23:15:00,269 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-07 23:15:00,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 23:15:00,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:15:00,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 23:15:00,303 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-07 23:15:00,303 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-07 23:15:00,306 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-07 23:15:00,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:15:00,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-07 23:15:00,631 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:15:00,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-07 23:15:00,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-07 23:15:00,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-07 23:15:00,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-07 23:15:00,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-07 23:15:00,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-07 23:15:00,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-07 23:15:00,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 23:15:00,737 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-07 23:15:00,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 23:15:00,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:15:00,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 23:15:00,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:15:01,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:15:01,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 23:15:01,079 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-07 23:15:01,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 23:15:01,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:15:01,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 23:15:01,119 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-07 23:15:01,121 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-07 23:15:01,122 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-07 23:15:01,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:15:01,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-07 23:15:01,450 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:15:01,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-07 23:15:01,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-07 23:15:01,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-07 23:15:01,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-07 23:15:01,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-07 23:15:01,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-07 23:15:01,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-07 23:15:01,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 23:15:01,555 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-07 23:15:01,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 23:15:01,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:15:01,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 23:15:01,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:15:01,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:15:01,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 23:15:01,661 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-07 23:15:01,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 23:15:01,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:15:01,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 23:15:01,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:01,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-07 23:15:01,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:01,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:01,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:01,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:01,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:01,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:01,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:01,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:01,822 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-07 23:15:01,825 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-07 23:15:01,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:15:02,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-07 23:15:02,171 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:15:02,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-07 23:15:02,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-07 23:15:02,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-07 23:15:02,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-07 23:15:02,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-07 23:15:02,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-07 23:15:02,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-07 23:15:02,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 23:15:02,280 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-07 23:15:02,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 23:15:02,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:15:02,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 23:15:02,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:15:02,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:15:02,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 23:15:02,391 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-07 23:15:02,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 23:15:02,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:15:02,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 23:15:02,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:02,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-07 23:15:02,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:02,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:02,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:02,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:02,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:02,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:02,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:02,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:02,559 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-07 23:15:02,563 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-07 23:15:02,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:15:02,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-07 23:15:02,894 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:15:02,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-07 23:15:02,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-07 23:15:02,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-07 23:15:02,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-07 23:15:02,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-07 23:15:02,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-07 23:15:02,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-07 23:15:02,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 23:15:02,993 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-07 23:15:03,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 23:15:03,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:15:03,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 23:15:03,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:15:03,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:15:03,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 23:15:03,099 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-07 23:15:03,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 23:15:03,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:15:03,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 23:15:03,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-07 23:15:03,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,273 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-07 23:15:03,275 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-07 23:15:03,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:15:03,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-07 23:15:03,607 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:15:03,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-07 23:15:03,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-07 23:15:03,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-07 23:15:03,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-07 23:15:03,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-07 23:15:03,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-07 23:15:03,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-07 23:15:03,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 23:15:03,707 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-07 23:15:03,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 23:15:03,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:15:03,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 23:15:03,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:15:03,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:15:03,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 23:15:03,812 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-07 23:15:03,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 23:15:03,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:15:03,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 23:15:03,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-07 23:15:03,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:03,967 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-07 23:15:03,969 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-07 23:15:04,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-07 23:15:04,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-07 23:15:04,306 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-07 23:15:04,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-07 23:15:04,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-07 23:15:04,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-07 23:15:04,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-07 23:15:04,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-07 23:15:04,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-07 23:15:04,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-07 23:15:04,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-07 23:15:04,407 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-07 23:15:04,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-07 23:15:04,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:15:04,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-07 23:15:04,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:15:04,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-07 23:15:04,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-07 23:15:04,512 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-07 23:15:04,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-07 23:15:04,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-07 23:15:04,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-07 23:15:04,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:04,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-07 23:15:04,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:04,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:04,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:04,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:04,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:04,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:04,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:04,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-07 23:15:04,667 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 ############################################################ # # Step 3.1: Update 1000000 RandonKey keys to create 50 percent fragmentation # ############################################################ ############################################################ # # RAM FootPrint: 3577.265625 # ############################################################ Read Start: 0 Read End: 0 Update Start: 1000000 Update End: 2000000 Expiry Start: 0 Expiry End: 0 Delete Start: 0 Delete End: 0 Create Start: 0 Create End: 0 Final Start: 1000000 Final End: 2000000 2022-02-07 23:15:04,714 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:04,714 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection9_10_1644304505.71 2022-02-07 23:15:05,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:05,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection8_9_1644304506.72 2022-02-07 23:15:06,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:06,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection19_8_1644304507.73 2022-02-07 23:15:07,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:07,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection13_7_1644304508.88 2022-02-07 23:15:08,903 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:08,903 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection14_6_1644304509.9 2022-02-07 23:15:09,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:09,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection11_5_1644304510.92 2022-02-07 23:15:10,927 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:10,927 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection12_4_1644304511.93 2022-02-07 23:15:11,937 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:11,937 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection17_3_1644304512.94 2022-02-07 23:15:12,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:12,948 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection18_2_1644304513.95 2022-02-07 23:15:13,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:13,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection15_1_1644304514.96 2022-02-07 23:15:14,971 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:14,971 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 23:15:15,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:15,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 23:15:16,993 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:16,993 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 23:15:18,107 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:18,107 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 23:15:19,124 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:19,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 23:15:20,137 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:20,137 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 23:15:21,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:21,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 23:15:22,160 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:22,160 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 23:15:23,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:23,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-07 23:15:24,181 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-07 23:15:24,181 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Loader_GleamBookUsers0__default_VolumeCollection13_7_1644304508.88 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection16_0_1644304515.97 Loader_GleamBookUsers0__default_VolumeCollection8_9_1644304506.72 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection7_-1_1644304516.98 Failed mutations count so far: update == 256 Failed mutations count so far: update == 256 Failed mutations count so far: update == 256 Failed mutations count so far: update == 256 Failed mutations count so far: update == 59 Failed mutations count so far: update == 47 Failed mutations count so far: update == 256 Failed mutations count so far: update == 22 Loader_GleamBookUsers0__default_VolumeCollection12_4_1644304511.93 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection6_-2_1644304518.0 Loader_GleamBookUsers0__default_VolumeCollection19_8_1644304507.73 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection5_-3_1644304519.11 Loader_GleamBookUsers0__default_VolumeCollection18_2_1644304513.95 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection4_-4_1644304520.13 Loader_GleamBookUsers0__default_VolumeCollection15_1_1644304514.96 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection3_-5_1644304521.14 Loader_GleamBookUsers0__default_VolumeCollection9_10_1644304505.71 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection2_-6_1644304522.15 Loader_GleamBookUsers0__default_VolumeCollection11_5_1644304510.92 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection10_-7_1644304523.16 Loader_GleamBookUsers0__default_VolumeCollection14_6_1644304509.9 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection1_-8_1644304524.17 Loader_GleamBookUsers0__default_VolumeCollection17_3_1644304512.94 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection0_-9_1644304525.18 Failed mutations count so far: update == 256 Failed mutations count so far: update == 256 Failed mutations count so far: update == 256 Failed mutations count so far: update == 256 Failed mutations count so far: update == 256 Failed mutations count so far: update == 256 Failed mutations count so far: update == 256 Failed mutations count so far: update == 256 Loader_GleamBookUsers0__default_VolumeCollection16_0_1644304515.97 is completed! Loader_GleamBookUsers0__default_VolumeCollection4_-4_1644304520.13 is completed! Failed mutations count so far: update == 256 Loader_GleamBookUsers0__default_VolumeCollection5_-3_1644304519.11 is completed! Loader_GleamBookUsers0__default_VolumeCollection2_-6_1644304522.15 is completed! Loader_GleamBookUsers0__default_VolumeCollection6_-2_1644304518.0 is completed! Loader_GleamBookUsers0__default_VolumeCollection10_-7_1644304523.16 is completed! Loader_GleamBookUsers0__default_VolumeCollection7_-1_1644304516.98 is completed! Loader_GleamBookUsers0__default_VolumeCollection0_-9_1644304525.18 is completed! Loader_GleamBookUsers0__default_VolumeCollection3_-5_1644304521.14 is completed! Loader_GleamBookUsers0__default_VolumeCollection1_-8_1644304524.17 is completed! Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:PCTJhb9uC8-1966243 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:nq5fg0tdVo-1966264 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:rsyI2cVYf2-1966286 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:mTcTTRw9c4-1966009 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:GP75n7tqpW-1966130 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:VwKFnP1giN-1966257 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:I5hbg2QT4F-1966004 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:cUEBEaxkCE-1966094 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:53y6u6hhZK-1966299 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:E6AuOmijom-1966308 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Aejnq5fg0t-1966508 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:9L9VwKFnP1-1966501 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:E1UZMXm5ze-1966457 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:2pjugyTsLF-1966432 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:HRCxfpwT8b-1966579 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:dWn3PCTJhb-1966551 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:R1H5IFousc-1966548 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:DLHGQaDgNx-1966684 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:jRZhzAUGC6-1966618 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:fy93Ynsvpe-1966565 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:CTukIXB7hQ-1966626 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:ocHEjdgwNg-1966553 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Twt9AUZshG-1966725 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:98zWjbYzKo-1966048 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:v2tueyz23H-1966557 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Lcx98zWjbY-1966254 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:iWuNIqagrp-1966803 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:AtsfqU3npe-1966455 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:R5HftJtKv8-1966174 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:JCy8EtQsDg-1966708 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Bif1vl1s66-1966632 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:H0ybNcJAkq-1966896 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:PjogxvDE8s-1966871 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Qgujc32weh-1966750 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:TOOWRdPEok-1966964 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:VQAvAa5u6d-1967078 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:wm18VmS9Pa-1967037 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:EUWqb73Pg9-1967044 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:E4im53bcJP-1967089 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:t0YmWm85hP-1967049 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:NfFsTDQlt3-1967082 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:dgwNgise9e-1967150 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:eEuRjWbWw5-1966465 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:qeA0e7FJRI-1966418 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Km0FbdGkLa-1966440 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:HqtZb3BeDY-1966628 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:0oifSkIaqL-1966380 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:MVnMl4FbA5-1966395 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:E84ovOs28p-1966402 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:JlBV01BYwr-1966761 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:YCgFQgKLqY-1966181 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:IYM3gq1lfw-1966883 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:7bp3VnLQ7x-1966406 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:p8DahqKSgS-1966514 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:P7QIH24Ow9-1967119 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:fpwT8bKnld-1966303 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:2dpfufxpXM-1966809 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:NxsVXGw5sd-1967156 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:NHjgDyEShV-1966266 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:y7pP3cb3pn-1966731 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:3mcxHiZvuJ-1966183 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:pSRtlqEQE5-1966568 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:cerzxxKWld-1966318 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:nldRu3sP1i-1966775 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:frK1dpNNbJ-1966168 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:0576kmpySn-1966281 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:NaV1FqkJkT-1966248 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:VtBmRwYuH5-1966569 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:6lIxpEKApM-1966705 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:KUYZjSU5MP-1967073 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Gd7JCy8EtQ-1966952 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:sH9WAQjicU-1967035 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:nWZgE7fWvE-1966265 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:tM7VSgqKqi-1966833 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:FdbhecBF38-1966694 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:ZfcefrCy3m-1966821 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:2fYaQ1HtpP-1966673 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Y9K8NaV1Fq-1966556 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:xw5IQ4xUUP-1966807 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Ch9QLBh5ey-1966986 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:783kMDmZv9-1966672 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:4M2xT14Sq2-1967153 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:KApMVF97id-1966244 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:qfflnjKDZE-1966532 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:g89IpkikSP-1966007 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:tgJR5HftJt-1966450 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:9rrf4urJJv-1966157 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:1dgjdITysv-1966019 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Zs3qhUYkGv-1966792 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:D6htVVbshj-1966301 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:jyXAsqmvAw-1966327 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:NTx2kdfTMp-1966488 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:DyEShVV5SY-1966982 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:6fY9K8NaV1-1966709 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:DgQAS22XcU-1967148 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:ZGFStAdahD-1966518 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:qU3npe6VJD-1966147 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:iDZF46LNFI-1966481 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:R57oCUDNGZ-1966711 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:sqmvAwVfr3-1967043 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:NpmSdUhIMg-1966140 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:2Nw5ke8iIc-1966487 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:oChzMZwxef-1966207 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:YnsvpeyJlB-1966321 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Z5m6M3mUiS-1966653 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:pZG8DEvNkc-1966766 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:dpNNbJzxhP-1966884 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:gQQ3EQwz1L-1966163 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:6c6S6bpiWe-1966538 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:2wDnpjUpw5-1966216 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:TehqwqZbs3-1966341 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:g55QZlTWXR-1966478 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:SYpi7VpCtf-1966718 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Dg5A8X2wlD-1966528 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:DNm1UH0GVK-1966928 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:sqM64WzYhP-1966730 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:RIoG1dh7y3-1966858 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:sBYCgFQgKL-1967106 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:uF6oChzMZw-1967124 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:9UJDUslYqA-1966500 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:6thCuO5h4V-1966516 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:THtPy7pP3c-1967039 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:WueYJBwFFS-1966683 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:n7tqpWVsFe-1966878 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:UZMXm5zeCZ-1966003 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:eWygu7ngCW-1966206 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:NSd8sZIuQx-1966352 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:A8X2wlDesP-1966252 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:xSDzHarVlI-1966285 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:L8JrUe1qWK-1966799 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:mg9AW2rENP-1967133 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:I24ez4KDKs-1966378 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:SylWeF1V4I-1966313 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:SC9fadb8CF-1966001 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:pRpsmDNm1U-1966002 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:9CLSPXdDSj-1966012 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:xsVXGw5sdR-1966021 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:gDyEShVV5S-1966028 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:zvh3y41Gnu-1966030 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:LeVdxGDdiZ-1966031 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:6QkkhTSFzt-1966033 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:BYCgFQgKLq-1966035 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:NxsH9WAQji-1966049 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:z23HQfZaUZ-1966051 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:2wlDesPc6S-1966054 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:piWe5mmLkL-1966056 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:KDZEokk8ON-1966058 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:kLcQzf8Dli-1966082 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:zeYOoL9lgT-1966084 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:1FZvMgpK7F-1966098 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:6WFedxhMNg-1966111 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:xHdJTKDuWK-1966137 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:e7FJRIoG1d-1966142 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:jwO0qcPrKO-1966144 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:TjxtqaSdvT-1966151 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:mtm8B5zFQ6-1966160 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:g6m0LjT32S-1966161 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:aXYSwYguT9-1966164 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Uh9sKhdDg7-1966169 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:4JaHPPp1e4-1966172 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:XXhzqCfzAJ-1966189 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:YVTL6jkhw8-1966214 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:2Cy1dgjdIT-1966217 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:mUiSKhOgBu-1966224 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:BeDY3mcxHi-1966235 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:it9QdShham-1966274 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:ZkEmOs7STZ-1966278 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:njKDZEokk8-1966288 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Z74ZZ6ltsq-1966290 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:VwbvwJDbfS-1966295 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:1cgk3p4UHh-1966315 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:LxBtNjO6xR-1966340 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:3qrdpKfUzL-1966343 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:qO9zV00ct0-1966353 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:iSqhd5bvKg-1966383 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:aA251eYggx-1966388 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:vb2rCEZKih-1966403 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:a5NSXEs4yJ-1966417 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:IDVqxP1E1D-1966489 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:tb9AwZ9CLS-1966494 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Z0EJzWTHSz-1966497 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:kMDmZv9rqv-1966523 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:lKiHumXzSO-1966527 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:O7GlpmyS6y-1966544 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:S5pfq2YmgA-1966545 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:OoSxSDzHar-1966561 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:WYP7rsyI2c-1966562 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:T2mu8A2YsL-1966564 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:sP1iVfeGMp-1966570 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:8eNOdDfiwY-1966578 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:m7dgM5OxAp-1966591 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:b4DtD9d2pj-1966605 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:gq1lfwZs4I-1966607 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:FPOkw6Z5ey-1966627 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:JJvvaRDUmg-1966629 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:D53j9e3EsB-1966634 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:rKOFdeSLLd-1966648 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:WRdPEokRew-1966656 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:4MpzxJa1s0-1966667 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:TI5uXP2Cy1-1966682 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:kmZ0EJzWTH-1966686 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:ObWXerIIbl-1966693 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:3jBNhefYtW-1966695 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:WPHrBBMonS-1966704 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:EDWocHEjdg-1966710 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:1wbKTDLcx9-1966715 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:kTckKDqQEa-1966720 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:JfJ9eRfGy2-1966722 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:WAQjicUo9c-1966727 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:kk8ON1m7dg-1966728 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:9J3XGOtuXw-1966733 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:fSBSWCWYP7-1966735 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:vAwVfr3idl-1966767 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:S88CgJl3i8-1966772 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:NP8IblIWiW-1966779 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:EIjmsdkU7g-1966780 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:UHhoLGtVD8-1966787 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:qnBnZZ3aHl-1966791 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:jYjLmaqfzk-1966827 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:0sNZXPeswG-1966829 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:1VMFE8wLyh-1966835 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:hGUYRfHzRj-1966837 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:TDQlt3h6Hv-1966838 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:aiVjk5RhWT-1966875 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:rtq8DXiDZF-1966882 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:KhdDg7NTx2-1966885 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:gCOPgOFdGL-1966894 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:T1T1tYZGFS-1966915 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:SR9Wh6fods-1966921 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:7jO6TI5uXP-1966926 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:db8CFPOkw6-1966935 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:4urJJvvaRD-1966937 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:iNx96fY9K8-1966953 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:QgKLqY9UJD-1966961 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:hXPVD2iO7G-1966973 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:eVdxGDdiZV-1966977 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:f6pkW80X7J-1966978 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:SO70QHGNvu-1966999 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:OYH8S88CgJ-1967016 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:8zWjbYzKow-1967034 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:48PNRjj3DD-1967057 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Yy8LVbM4sy-1967071 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Jy6Z1VMFE8-1967079 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:9IcTvZsajL-1967080 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:0KmLB6EKSy-1967091 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:Hp3CzCYxo0-1967092 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:KehRqdQ45V-1967097 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:7yxXqnBnZZ-1967099 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:O5h4V2Ihyr-1967109 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:mXzSO70QHG-1967116 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:iSKhOgBu1I-1967125 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:zOnmKuIQ5v-1967128 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:lTWXR7ELA3-1967135 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:o41RGxQguj-1967147 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:FScsI7jZbw-1967155 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection9:jgDyEShVV5-1967167 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:c2VtaL0Rdn-1972038 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:3kZzxw9Ude-1972104 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:MwRpgCTukI-1972123 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:D1xw5IQ4xU-1972341 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:H8S88CgJl3-1972314 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:FJRIoG1dh7-1972392 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:hVG3CiOMOL-1972434 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:mTcTTRw9c4-1972497 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:taGd7JCy8E-1972238 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:egTKddNGqw-1972066 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:gBu1IjGlKi-1972462 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:piWe5mmLkL-1972544 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:krmsRPq5JJ-1972207 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:mSdUhIMg2m-1972398 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:wT8bKnldRu-1972553 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:tfP5CTwt9A-1972575 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:IuQxUp9ofW-1972590 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:4ovOs28p3s-1972684 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:A0e7FJRIoG-1972700 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:OoSxSDzHar-1972744 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:bhecBF38MP-1972752 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:Mhwu62JnWs-1972793 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:PW6pFpOsvw-1972850 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:IBHdne8dxw-1972840 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:UtmdwGwSpv-1972861 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:Oi8Y7bp3Vn-1972893 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:AJvvHRCxfp-1973006 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:S5pfq2YmgA-1972984 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:uEDWocHEjd-1973012 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:5IQ4xUUPcK-1973057 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:TFDo3MZZ74-1973021 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:iKkV4ud6th-1972106 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:zxxKWld6Oz-1972376 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:qO9zV00ct0-1972792 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:yAejnq5fg0-1972983 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:qQEacp5fUA-1972766 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:WiWaKm9tWt-1972785 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:VTJJt0a8JS-1972858 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:HvLyDxcBfF-1972695 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:fqU3npe6VJ-1972705 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:xpXMqE8vqb-1972823 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:YaQ1HtpPQE-1972931 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:GQaDgNxsVX-1972942 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:kjrWdNaW9v-1972890 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:nBSimvwPf6-1972941 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:9lgTSkJVTJ-1973122 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:eAvcKHqu1c-1973082 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:BnZZ3aHlLO-1973105 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:gMiq8xuD1x-1972637 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:9lgTSkJVTJ-1972001 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:fFKZWrNdBc-1972050 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:Ss7fPtmqO9-1972288 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:F6oChzMZwx-1972509 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:BmRwYuH5JI-1972635 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:tqaSdvT3Mx-1972116 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:2rENPDLHGQ-1972246 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:TWXR7ELA3k-1972484 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:ZgE7fWvEio-1972523 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:eHfY5H5frK-1972875 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:3CiOMOLpqs-1973182 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:ijomrU3AHl-1972322 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:fm2Ur6lIxp-1972142 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:XccNF9CgX7-1972593 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:gJjdflNulT-1972928 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:g2FaZaJMVn-1972330 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:Ccupt2a5DO-1972283 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:a4xZAASMdg-1973115 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:7JCy8EtQsD-1973018 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:cJ1iIEOjjs-1972673 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:O40sNZXPes-1972054 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:3CiOMOLpqs-1972061 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:IW48PNRjj3-1972599 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:8eNOdDfiwY-1972757 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:zb4DtD9d2p-1972086 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:4LgASshbbZ-1973138 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:k0npOCjlNv-1972136 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:2rENPDLHGQ-1972149 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:WwESaYUO40-1972699 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:9UJDUslYqA-1972943 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:xApP5W2sXO-1972597 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:va9INIC7g5-1972473 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:SpvUnltsoV-1972432 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:TOjoMQAx8Q-1972601 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:sdR57oCUDN-1973013 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:UCvgMlbl8c-1972046 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:ZZ6ltsqM64-1972572 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:53ei8DX93Q-1972790 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:Mxdt2LqFDW-1973066 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:O8FmJQwUHU-1972041 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:JITFNqsdrL-1972339 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:fiwYWblG44-1972552 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:8JSA6WE1z2-1972393 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:VD8kLcQzf8-1972825 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:a5NSXEs4yJ-1972856 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:1tXFl0uH79-1972407 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:smDNm1UH0G-1972470 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:gDyEShVV5S-1972516 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:RNfFsTDQlt-1972360 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:aA251eYggx-1972831 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:1ZJfJ9eRfG-1972256 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:AUvcqIW3eg-1972711 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:pi7VpCtfP5-1972968 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:9p1pVKsAOn-1973132 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:bs30ocMa4x-1972355 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:zXgE1X9yrT-1972452 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:ogCOPgOFdG-1972428 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:VxP7OkQ8sx-1972616 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:g2QT4FTVWe-1972472 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:zV00ct0YmW-1972611 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:WHnHFOXZTy-1973142 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:bNcJAkqYOn-1973154 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:O8FmJQwUHU-1973162 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:ZQV3DWzvb2-1973170 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:IvFQHXUeDr-1972743 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:0oifSkIaqL-1972855 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:12II4hmBeI-1972913 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:VYf2ZfASsU-1972203 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:IblIWiWaKm-1973029 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:s28p3sGTcP-1972440 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:qu1cgk3p4U-1972617 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:bY6EJWjG1Q-1972084 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:CTwt9AUZsh-1972267 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:PXdDSjnKiS-1972448 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:WQdQWDThzx-1973075 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:Jj9435C2Lx-1972343 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:Wst9vh2dcJ-1972352 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:UeDrVeHdWF-1972602 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:W7jO6TI5uX-1972460 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:krmsRPq5JJ-1972296 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:gwNgise9eo-1972567 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:ZsUOXMkB96-1972917 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:3sGTcPaCNX-1972043 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:DkD3eEuRjW-1972067 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:hpLAOWuB6z-1972069 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:CZ5m6M3mUi-1972070 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:NIC7g55QZl-1972072 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:9c40p8Dahq-1972076 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:4D53j9e3Es-1972079 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:PXdDSjnKiS-1972103 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:qIW3egTKdd-1972118 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:x12ADkD3eE-1972119 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:csuhpLAOWu-1972121 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:hw8qYsmHPO-1972131 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:14Sq2WPHrB-1972137 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:Gw5sdR57oC-1972140 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:yGQ5uEDWoc-1972143 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:cxHiZvuJgm-1972148 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:9WAQjicUo9-1972168 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:QfZaUZxh4q-1972170 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:xGDdiZVuDg-1972182 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:QLBh5eyBRk-1972187 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:F9CgX7s8ze-1972192 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:9EIjmsdkU7-1972193 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:VeHdWFO7ZR-1972197 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:VlIFWzOrA0-1972206 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:hZKcmr80uU-1972212 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:MShc3E6ake-1972213 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:H8S88CgJl3-1972217 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:Gw5sdR57oC-1972233 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:Afm2Ur6lIx-1972235 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:T8VZHKEEo3-1972243 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:okk8ON1m7d-1972270 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:CQ2ra8wA6c-1972277 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:fWvEioo0qf-1972279 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:Oh5b5U88Cw-1972286 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:pySnWQdQWD-1972295 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:ocGddIqiBs-1972319 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:Wld6OzyEn1-1972324 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:7Zs3qhUYkG-1972334 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:xuh6ADuvol-1972338 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:gk3p4UHhoL-1972373 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:ZvMgpK7FxG-1972380 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:mN1DHDZd0V-1972400 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:bA5TwesBDH-1972401 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:5n7tqpWVsF-1972412 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:S1YntjYjLm-1972413 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:bjqoVZfcef-1972415 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:m8B5zFQ6uf-1972418 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:1dpNNbJzxh-1972426 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:9sKhdDg7NT-1972427 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:KihPKJC12I-1972441 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:3EQwz1LZsU-1972445 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:iKkV4ud6th-1972455 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:O0qcPrKOFd-1972466 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:hQPiXSR9Wh-1972483 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:ZeWygu7ngC-1972508 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:FnP1giNx96-1972515 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:zvh3y41Gnu-1972518 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:LeVdxGDdiZ-1972519 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:CF56CQ2ra8-1972521 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:Jhb9uC8dr3-1972525 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:HSzAfm2Ur6-1972543 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:32dl7js27l-1972555 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:98zWjbYzKo-1972568 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:X93QIEiWuN-1972581 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:aQJOJj9435-1972587 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:MlgjSs7fPt-1972596 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:HlLOmN1DHD-1972644 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:Y6W2B65f4J-1972647 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:vWDwhPjogx-1972649 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:5I6UQKehRq-1972659 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:BtNjO6xR7Z-1972662 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:SaaJUtxSNy-1972687 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:4ez4KDKs9p-1972692 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:B96kMwRpgC-1972712 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:SC9fadb8CF-1972713 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:Ng6m0LjT32-1972727 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:a8XQckTMRW-1972733 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:o9cZkEmOs7-1972765 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:JJKlNHzE4i-1972768 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:afCVxP7OkQ-1972769 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:rLsVANqtc1-1972779 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:WDThzxiPIB-1972799 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:xjdswCV1tX-1972800 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:MVnMl4FbA5-1972802 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:bZlJ2jkaiV-1972810 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:fzkJc6NyoN-1972833 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:dNaW9vLx8n-1972838 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:vb2rCEZKih-1972842 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:Zw9ftMaa6Z-1972849 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:H8kRKWXgQQ-1972866 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:efrCy3mCen-1972887 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:pVKsAOnSaa-1972888 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:NZXPeswGK3-1972895 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:ukIXB7hQPi-1972908 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:kw6Z5eyHSy-1972909 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:0GVKXuF6oC-1972910 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:qRPmDUhL9r-1972924 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:OWuB6zOnmK-1972946 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:3j9e3EsBYC-1972948 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:5gTQXp99vv-1972985 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:dWn3PCTJhb-1972990 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:a8wA6c6S6b-1972993 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:OCjlNvbaDo-1973009 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:C1wbKTDLcx-1973017 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:Gnucg7zmRK-1973022 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:X7s8zeNTOK-1973033 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:dIqiBs3uhv-1973035 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:c6SkEW9tqR-1973046 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:S9PaIvFQHX-1973051 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:qMMebLAPHX-1973054 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:fWVwj6xQ9Z-1973062 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:NqsdrLsVAN-1973087 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:tHciSqhd5b-1973098 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:H79oWErfNp-1973103 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:U5MPAEtRa2-1973119 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:gxvDE8sA1v-1973121 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:xkCESKeDJp-1973124 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:bWSYGhfbE8-1973133 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:3sGTcPaCNX-1973156 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:zFQ6ufIR9N-1973166 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:O40sNZXPes-1973171 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:oG1dh7y3RT-1973172 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:szo6eHfY5H-1973183 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:CFPOkw6Z5e-1973185 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection19:SPUPHz7bgr-1973191 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:T0wVit9QdS-1942150 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:HZCF56CQ2r-1942095 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:pi7VpCtfP5-1942161 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:DUhL9rrf4u-1942065 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:PW6pFpOsvw-1942267 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:t2a5DOwlpk-1942412 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:u7NIAtsfqU-1942331 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:cJPUE1J10Y-1942516 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Bu1IjGlKiH-1942552 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Qgujc32weh-1942574 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:kk8ON1m7dg-1942616 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:HrBBMonSSa-1942379 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:8ON1m7dgM5-1942435 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:X7s8zeNTOK-1942486 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:WYcJ1iIEOj-1942247 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:4rhkXNzOo5-1942296 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:2XcUuhT0wV-1942611 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:GVmcsuhpLA-1942808 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:ZGFStAdahD-1942342 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:35C2LxBtNj-1942472 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:zAqiB3p4D5-1942786 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:gCOPgOFdGL-1942782 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:dmXYu7Y9pd-1942232 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Qlt3h6HvLy-1942505 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:ueyz23HQfZ-1942137 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Cy3mCenoH0-1942248 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:DNlHpXcaef-1942354 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:wZ9CLSPXdD-1942074 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:h4q1GQLjni-1942148 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:VSgqKqid2L-1942504 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:rENPDLHGQa-1942528 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:VoZm27n7wF-1942560 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:8zWjbYzKow-1942858 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:1RGxQgujc3-1942850 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:omrU3AHlB7-1942636 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:EIjmsdkU7g-1942604 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:dITysvnR1H-1942577 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:aqLthFpH8k-1942740 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Uqrd2LocGd-1942929 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:GQLjnigN2x-1942928 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:bftdTOOWRd-1942840 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:PHXTOjoMQA-1942943 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:cerzxxKWld-1942974 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:LvkUIYM3gq-1943015 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:QEacp5fUAq-1943048 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:c6S6bpiWe5-1943071 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:iWaKm9tWtT-1943075 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Tepa2jbjqo-1942257 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:hyrrucUjOV-1942536 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:amiA23VtBm-1942602 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:KkV4ud6thC-1942545 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:UHbnpSRtlq-1942444 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Q7x3dQ5gtv-1942734 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:qRXccNF9Cg-1942935 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:AsqmvAwVfr-1942675 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:sKhdDg7NTx-1942773 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:ebLAPHXTOj-1942195 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:QIH24Ow9Za-1942822 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:2QT4FTVWeM-1942762 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:3EsBYCgFQg-1942057 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:a4xZAASMdg-1942496 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:WvEioo0qff-1942561 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Icva9INIC7-1942815 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:SR9Wh6fods-1942553 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:OYH8S88CgJ-1942904 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:DThzxiPIBe-1943081 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:TDLcx98zWj-1943063 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:zMZwxefhTj-1942555 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:RPq5JJKlNH-1942493 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:FBBHw1BpwI-1942533 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:m2Ur6lIxpE-1942581 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:st9vh2dcJh-1942706 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:l3i8jyXAsq-1942203 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:53bcJPUE1J-1942637 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:T2mu8A2YsL-1942196 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:CiIX53ei8D-1942483 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:8VZHKEEo33-1942557 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:WeF1V4IU3L-1942661 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:JSA6WE1z2X-1942747 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:VANqtc1g2F-1942972 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:S6bpiWe5mm-1942118 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:wJDbfSBSWC-1942899 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:QsDgQAS22X-1942139 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:lKiHumXzSO-1942351 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:V01BYwrGuH-1942464 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:mZIXBtjcK2-1942638 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:WgqQIObDgV-1942902 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:CNXAUvcqIW-1942021 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:fflnjKDZEo-1943065 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:7oCUDNGZ8Z-1942382 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:ugyTsLFlHq-1942812 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:kMDmZv9rqv-1942347 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:IBeNlYMxdt-1942684 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:XBtjcK2OSU-1942513 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Zv9rqvJYcP-1942103 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:PjogxvDE8s-1942695 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:3pndlMlgjS-1943091 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:HiZvuJgm3x-1942531 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:QPiXSR9Wh6-1942797 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:hwu62JnWsH-1943083 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Hz7bgrZWQ5-1942360 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:0576kmpySn-1942937 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:vb2rCEZKih-1942035 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:m7dgM5OxAp-1942159 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:CHdqYy8LVb-1942915 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:IblIWiWaKm-1942482 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:ke8iIcva9I-1942067 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:WYP7rsyI2c-1942194 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:6jkhw8qYsm-1942818 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:kbT8VZHKEE-1942833 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:i7VpCtfP5C-1943066 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:7wF1rOdTHt-1943064 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:c6SkEW9tqR-1942463 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:S4LnI24ez4-1942510 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:nldRu3sP1i-1942599 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:PCTJhb9uC8-1942131 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:qbWHnHFOXZ-1942708 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:dpNNbJzxhP-1942772 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Km0FbdGkLa-1942072 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:pvUnltsoVx-1942722 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:CdoUtM7VSg-1942709 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:zRjdeL6xz7-1942264 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:J9eRfGy2D6-1942453 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:ufIR9NSC9f-1942307 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:8tUKtgJR5H-1942326 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:hb9uC8dr3j-1942879 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:t0a8JSA6WE-1942991 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:CenoH0ybNc-1943028 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:YRfHzRjdeL-1942508 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:uscOh5b5U8-1942876 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:gZ4mTcTTRw-1942071 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:kRewpbLoRz-1942087 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:d2LocGddIq-1942909 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:3E6akeoCHd-1942443 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:pbLoRzCC1w-1942867 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:IEX3hMCcGV-1942304 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:JlBV01BYwr-1942649 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:I1cvGw4rhk-1942693 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:JRIoG1dh7y-1942746 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:mgCoEc4N4k-1942694 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:nBSimvwPf6-1942394 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:ra5NSXEs4y-1942241 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:zAJyAejnq5-1942416 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:U3L2aA251e-1942520 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:htVVbshjUH-1942004 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:IBHdne8dxw-1942033 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:bgrZWQ5D8o-1942052 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Z5eyHSyW7j-1942054 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:6zOnmKuIQ5-1942055 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Umg9AW2rEN-1942056 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:32GsbftdTO-1942060 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:q2YmgACcup-1942093 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:a1s0Ch9QLB-1942094 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Z74ZZ6ltsq-1942114 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:2D6htVVbsh-1942125 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:guBRAz5nR3-1942129 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:NaV1FqkJkT-1942136 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:DNGZ8Z1ZJf-1942138 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:RKNxsH9WAQ-1942143 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:IvFQHXUeDr-1942156 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:BYwrGuHTHp-1942188 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:tuXwVIf9kt-1942192 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:TtUMTEaQJO-1942205 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:tVD8kLcQzf-1942214 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:AASMdg89D0-1942220 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:cBeOwIfmgC-1942222 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:xUUPcKWst9-1942234 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:1qWKcUEBEa-1942258 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:YtHciSqhd5-1942291 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Y1iC8KD9QF-1942303 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:DbDYD6IvPK-1942322 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:12II4hmBeI-1942334 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:6thCuO5h4V-1942340 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:KddNGqwdIk-1942352 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:OWuB6zOnmK-1942363 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:g55QZlTWXR-1942366 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:d93OHZCF56-1942371 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:WocHEjdgwN-1942377 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:HGQaDgNxsV-1942391 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:BpwI1F5zvh-1942392 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:53siMiLeVd-1942393 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:V3NHonR9NH-1942418 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:UX60d93OHZ-1942423 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:ysvnR1H5IF-1942424 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:8CgJl3i8jy-1942447 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:zKowdXhFUq-1942457 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:3Pg93qrdpK-1942475 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:85hPdmXYu7-1942476 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:WQdQWDThzx-1942488 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:FWzOrA0lE6-1942492 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:FxG0xjdswC-1942525 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:aZaJMVnMl4-1942527 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:qCfzAJyAej-1942537 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:EN6783kMDm-1942548 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:8oADhYUY20-1942556 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:7jO6TI5uXP-1942558 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:iNx96fY9K8-1942585 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:fSBSWCWYP7-1942623 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:bHCP5M3Gz5-1942626 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:EiWuNIqagr-1942627 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Yf2ZfASsU7-1942646 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Hp3CzCYxo0-1942660 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:kQ8sxIorJa-1942662 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:KehRqdQ45V-1942665 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:3QIEiWuNIq-1942679 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:t0YmWm85hP-1942681 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:aiVjk5RhWT-1942699 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:n7tqpWVsFe-1942702 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:1YntjYjLma-1942703 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:VQAvAa5u6d-1942710 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:edxhMNgS4L-1942715 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:8nv70WoaGT-1942727 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:xwHUCvgMlb-1942729 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:B3p4D53j9e-1942766 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:pe6VJDDNlH-1942783 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:1DAWke3rqd-1942785 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:GiXiKkV4ud-1942789 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:KhOgBu1IjG-1942796 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:eWygu7ngCW-1942798 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:6oChzMZwxe-1942799 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:eIx12ADkD3-1942806 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:rqvJYcPs4M-1942819 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:4V2Ihyrruc-1942844 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:hMufzKV3NH-1942847 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:FqkJkTckKD-1942852 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:xsH9WAQjic-1942859 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Os7STZo0QP-1942882 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:w9fW0n8qMM-1942896 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:iWe5mmLkL3-1942898 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:8A2YsLQ1Xr-1942912 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:RwYuH5JITF-1942917 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:TOKn9C6IyF-1942926 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:ZkEmOs7STZ-1942934 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:HXUeDrVeHd-1942936 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:p1FZvMgpK7-1942946 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:FDWsJy6Z1V-1942955 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:10YI9IcTvZ-1942956 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:E6AuOmijom-1942964 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:cyEv0KmLB6-1942967 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:GuHTHp3CzC-1942968 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:I6UQKehRqd-1942973 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:h3S1YntjYj-1942979 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:e7FJRIoG1d-1942990 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:51eYggxc0K-1942992 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:wCV1tXFl0u-1942997 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:wIfmgCoEc4-1943002 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:WDwhPjogxv-1943003 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:mtm8B5zFQ6-1943008 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:g6m0LjT32S-1943009 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Uh9sKhdDg7-1943017 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:J1iIEOjjsS-1943027 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:9vLx8nv70W-1943035 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:Gy2D6htVVb-1943050 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:9K8NaV1Fqk-1943057 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:7zmRKNxsH9-1943060 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection14:MZZ74ZZ6lt-1943067 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:E6AuOmijom-1948530 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:ra5NSXEs4y-1948607 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:qHiFamp2FB-1948651 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:AvcKHqu1cg-1948674 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:zSO70QHGNv-1948413 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:cyEv0KmLB6-1948493 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:SgSW27pnRk-1948400 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:pKfUzLzveY-1948537 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:J2jkaiVjk5-1948309 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:VKsAOnSaaJ-1948736 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:qeA0e7FJRI-1948600 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:UHU7Tjxtqa-1948609 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:HzRjdeL6xz-1948598 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:Y6EJWjG1QG-1948825 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:4FTVWeMu6P-1948812 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:WVwj6xQ9ZK-1948686 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:WtTHrDcyEv-1948702 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:AkqYOna8XQ-1948790 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:gq1lfwZs4I-1948821 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:icUo9cZkEm-1948921 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:F9J3XGOtuX-1948955 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:nldRu3sP1i-1948957 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:QV3DWzvb2r-1949050 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:jYjLmaqfzk-1949041 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:pvUnltsoVx-1949080 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:7idAJvvHRC-1949162 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:XqnBnZZ3aH-1948973 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:uscOh5b5U8-1948186 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:2xT14Sq2WP-1948390 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:l4FbA5Twes-1948525 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:KUYZjSU5MP-1948239 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:g6m0LjT32S-1948351 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:guBRAz5nR3-1948431 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:xjdswCV1tX-1948551 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:K7FxG0xjds-1949008 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:TrPbQUUpyg-1949032 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:Jc6NyoN7Tr-1948336 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:GK34mklO8F-1948346 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:L8JrUe1qWK-1949013 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:HciSqhd5bv-1948754 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:HfY5H5frK1-1948787 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:ESaYUO40sN-1949188 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:s7fPtmqO9z-1948936 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:dBccR7Tddi-1948586 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:nI24ez4KDK-1948592 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:pOsvwbWSYG-1949190 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:pmSdUhIMg2-1949199 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:Maa6ZsmxEb-1949189 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:tm8B5zFQ6u-1949219 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:MXm5zeCZ5m-1949268 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:wO0qcPrKOF-1949267 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:gjdITysvnR-1949284 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:TChXPVD2iO-1949280 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:jnKiSNqHiF-1948852 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:mDNm1UH0GV-1949118 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:5zeCZ5m6M3-1949119 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:eL6xz7ZQV3-1949191 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:GQ5uEDWocH-1949168 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:QXp99vvoaB-1948424 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:1Apoctk0np-1948437 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:ulTChXPVD2-1948415 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:aKm9tWtTbH-1948480 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:8WTxOYH8S8-1948458 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:8oJV0UG3KU-1948567 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:NlYMxdt2Lq-1948966 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:8DX93QIEiW-1948481 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:QHXUeDrVeH-1948502 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:IObDgVrOoS-1948952 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:h6HvLyDxcB-1948595 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:Mlbl8cpRps-1948799 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:w5sdR57oCU-1949169 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:j73szo6eHf-1949083 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:XNzOo5ohVG-1948578 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:2Cy1dgjdIT-1948439 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:gZ4mTcTTRw-1948621 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:hGUYRfHzRj-1948995 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:4Ow9ZaS5pf-1949128 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:uWKyrYzYhD-1949055 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:kLaMi7RTaF-1949102 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:V38kxqxHdJ-1948752 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:kLcQzf8Dli-1948520 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:NaV1FqkJkT-1948470 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:7y3RTNg6m0-1948744 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:JkTckKDqQE-1949166 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:fzXgE1X9yr-1949253 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:q2YmgACcup-1948427 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:p1pVKsAOnS-1949044 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:cK2OSUECdo-1948571 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:FdbhecBF38-1948876 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:lkeqdRF8tU-1949081 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:tmqO9zV00c-1948692 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:TRw9c40p8D-1949093 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:45VGKDfzeY-1948735 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:SgqKqid2Lz-1948715 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:8DXiDZF46L-1948820 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:E6akeoCHdq-1948906 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:dGkLaMi7RT-1949255 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:f4urJJvvaR-1949095 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:GDdiZVuDg5-1949179 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:3YnsvpeyJl-1948511 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:UE1J10YI9I-1948574 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:l4mKUh9sKh-1948635 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:NIqagrp1FZ-1948677 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:wj6xQ9ZKgt-1948561 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:5ohVG3CiOM-1949235 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:h6fodsp2wD-1948835 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:jZbwAQNH5h-1949132 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:eoB03W48WT-1948919 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:3dQ5gtvLE1-1948143 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:HSyW7jO6TI-1948360 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:AOnSaaJUtx-1948587 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:lG8oJV0UG3-1948684 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:9QFClkeqdR-1948301 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:k5RhWTUtmd-1949005 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:R9NSC9fadb-1948613 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:JPUE1J10YI-1948727 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:lpmyS6yZgr-1948426 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:scE07yxXqn-1948533 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:JhOS0aMNZj-1948704 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:0e7FJRIoG1-1948292 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:VG3CiOMOLp-1949082 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:M5OxApP5W2-1948497 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:maYMR7LvkU-1948750 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:4WnWYcJ1iI-1948762 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:0Ng1UfmC2d-1948495 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:PHb0i4cohG-1948523 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:pXcaefzXgE-1948660 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:wJj73szo6e-1949232 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:JVQAvAa5u6-1948236 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:DKs9p1pVKs-1948264 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:svwbWSYGhf-1948265 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:xcBfFKZWrN-1948267 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:zf8DliKZIe-1948276 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:9IcTvZsajL-1948278 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:i4cohGUYRf-1948279 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:wESaYUO40s-1948291 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:fSkIaqLthF-1948294 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:WDwhPjogxv-1948305 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:fzeYOoL9lg-1948306 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:UvcqIW3egT-1948335 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:aaJUtxSNyr-1948343 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:ne8dxwHUCv-1948347 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:rKC3kZzxw9-1948353 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:KhOgBu1IjG-1948362 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:QPiXSR9Wh6-1948363 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:sbftdTOOWR-1948406 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:QIH24Ow9Za-1948412 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:NHjgDyEShV-1948416 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:VpzLz8DpDc-1948417 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:1F5zvh3y41-1948418 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:a1s0Ch9QLB-1948420 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:p2wDnpjUpw-1948438 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:VwKFnP1giN-1948447 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:GQLjnigN2x-1948462 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:Uqrd2LocGd-1948463 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:EjdgwNgise-1948467 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:mRKNxsH9WA-1948469 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:tlqEQE5xuh-1948482 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:8sZIuQxUp9-1948490 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:V00ct0YmWm-1948491 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:ndlMlgjSs7-1948496 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:VIf9ktRZmW-1948506 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:8jyXAsqmvA-1948509 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:OSUECdoUtM-1948519 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:u1cgk3p4UH-1948529 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:lNHzE4im53-1948531 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:cBeOwIfmgC-1948548 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:Tylnh3S1Yn-1948557 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:KIrYeEiJVQ-1948564 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:UtmdwGwSpv-1948576 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:Cy3mCenoH0-1948582 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:RKWXgQQ3EQ-1948589 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:XMkB96kMwR-1948612 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:ke8iIcva9I-1948617 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:YhDLmtm8B5-1948626 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:T3DX25togC-1948636 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:Upyg86daVU-1948639 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:DUmg9AW2rE-1948646 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:32GsbftdTO-1948650 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:YVTL6jkhw8-1948652 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:Km0FbdGkLa-1948662 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:IqiBs3uhvg-1948691 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:FO7ZR0deQy-1948698 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:Pq5JJKlNHz-1948700 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:hUYkGvWDwh-1948706 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:4LnI24ez4K-1948713 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:OXZTylnh3S-1948714 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:ajLPW6pFpO-1948718 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:LGtVD8kLcQ-1948725 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:nZZ3aHlLOm-1948729 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:aJMVnMl4Fb-1948734 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:Wzvb2rCEZK-1948742 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:z2XeQdRkwN-1948745 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:8kRKWXgQQ3-1948746 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:E8sA1vktSC-1948757 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:sFeAP1OYf6-1948764 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:NlHpXcaefz-1948781 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:tUKtgJR5Hf-1948785 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:2II4hmBeIx-1948793 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:EUCqRPmDUh-1948824 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:v8IDVqxP1E-1948828 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:RWttb9AwZ9-1948829 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:Ijx53siMiL-1948844 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:Bif1vl1s66-1948854 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:m27n7wF1rO-1948874 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:oo0qfflnjK-1948875 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:r6lIxpEKAp-1948895 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:L3GzaN0576-1948896 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:SWCWYP7rsy-1948897 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:XGOtuXwVIf-1948903 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:JvvHRCxfpw-1948918 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:KowdXhFUqr-1948920 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:6SkEW9tqRX-1948926 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:shjUHbnpSR-1948931 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:TZo0QPTtUM-1948932 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:cGddIqiBs3-1948935 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:5W2sXO3Mhw-1948937 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:QAx8QO3LNA-1948941 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:ySnWQdQWDT-1948943 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:0n8qMMebLA-1948954 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:blG44Hcfy9-1948956 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:Shc3E6akeo-1948958 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:bHCP5M3Gz5-1948984 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:EiWuNIqagr-1948985 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:kCESKeDJpL-1949004 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:UjRZhzAUGC-1949024 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:sSDbDYD6Iv-1949025 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:CiOMOLpqsz-1949030 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:zo6eHfY5H5-1949031 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:3hnOi8Y7bp-1949049 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:40sNZXPesw-1949051 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:hFpH8kRKWX-1949054 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:TKDuWKyrYz-1949075 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:EOjjsSDbDY-1949077 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:yoN7TrPbQU-1949084 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:Wke3rqdjvT-1949099 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:B91UIjx53s-1949120 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:IhyrrucUjO-1949126 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:xHiZvuJgm3-1949145 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:TSFzt9nBSi-1949149 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:FQgKLqY9UJ-1949151 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:WAQjicUo9c-1949165 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:ShVV5SYpi7-1949176 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:y41Gnucg7z-1949178 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:LBh5eyBRkJ-1949180 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:Q2ra8wA6c6-1949181 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:5bvKgR3hnO-1949194 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:DwhPjogxvD-1949210 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:g89D0jHayI-1949211 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:rK1dpNNbJz-1949227 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:1iIEOjjsSD-1949234 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:FClkeqdRF8-1949238 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:vLx8nv70Wo-1949242 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:7TddipfgTu-1949243 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:w5B4jtF2fY-1949256 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection11:tk0npOCjlN-1949290 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:ChzMZwxefh-1977898 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:R7ELA3kmZ0-1977901 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:JDbfSBSWCW-1977966 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:AQjicUo9cZ-1977801 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:QA66NEUCqR-1977857 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:5nR3ybObWX-1977976 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:CgX7s8zeNT-1977825 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:GddIqiBs3u-1977827 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:YVmCTHSd8e-1977982 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:I7jZbwAQNH-1977921 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:sDgQAS22Xc-1977325 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:sI7jZbwAQN-1977052 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:gOFdGLIaeQ-1977724 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:yW7jO6TI5u-1977897 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:bT8VZHKEEo-1977900 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:wYWblG44Hc-1977068 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:SaYVmCTHSd-1977077 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:pmyS6yZgrb-1977318 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:f2ZfASsU70-1977832 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:HQfZaUZxh4-1977952 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:fSBSWCWYP7-1977841 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:F97idAJvvH-1977983 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:wRpgCTukIX-1977752 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:qiB3p4D53j-1977913 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:eHdWFO7ZR0-1977834 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:p5fUAqNP8I-1977998 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:V2IhyrrucU-1977947 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:cqIW3egTKd-1977868 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:C3kZzxw9Ud-1977894 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:BKT1T1tYZG-1977890 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:Gd7JCy8EtQ-1977810 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:8oADhYUY20-1977782 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:cva9INIC7g-1977914 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:VZHKEEo33U-1977783 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:fadb8CFPOk-1977910 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:1Sffgzb4Dt-1977856 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:6kMwRpgCTu-1977905 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:yTsLFlHqtZ-1977915 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:XGw5sdR57o-1977930 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:W2rENPDLHG-1977939 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:khTSFzt9nB-1977942 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:jkhw8qYsmH-1977945 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:fW0n8qMMeb-1977967 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:b9uC8dr3jB-1977978 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:cOh5b5U88C-1977979 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:k6w9fW0n8q-1977987 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection12:qrd2LocGdd-1977996 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:9L9VwKFnP1-1920793 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:HRCxfpwT8b-1920807 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:0wVit9QdSh-1920810 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:qfflnjKDZE-1920824 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:sU70Ng1Ufm-1920841 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:hwu62JnWsH-1920842 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:IvFQHXUeDr-1920848 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:tBmRwYuH5J-1920861 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:P1iVfeGMpe-1920862 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:cK2OSUECdo-1920865 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:8mdUWDRNfF-1920870 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:GKDfzeYOoL-1920876 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:j6xQ9ZKgtR-1920895 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:xG0xjdswCV-1920897 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:4xZAASMdg8-1920900 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:OXZTylnh3S-1920911 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:ebVKIrYeEi-1920918 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:xo0NRcIHoU-1920927 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:1iC8KD9QFC-1920931 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:LmaqfzkJc6-1920934 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:rhkXNzOo5o-1920956 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection17:KeDJpL85wJ-1920957 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:PCTJhb9uC8-1966243 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:nq5fg0tdVo-1966264 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:rsyI2cVYf2-1966286 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:9L9VwKFnP1-1966501 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:E1UZMXm5ze-1966457 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:mTcTTRw9c4-1966009 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:GP75n7tqpW-1966130 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:VwKFnP1giN-1966257 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:I5hbg2QT4F-1966004 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:2pjugyTsLF-1966432 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:cUEBEaxkCE-1966094 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:53y6u6hhZK-1966299 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:E6AuOmijom-1966308 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:HRCxfpwT8b-1966579 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Aejnq5fg0t-1966508 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:dWn3PCTJhb-1966551 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:R1H5IFousc-1966548 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:jRZhzAUGC6-1966618 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Bif1vl1s66-1966632 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:CTukIXB7hQ-1966626 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:fy93Ynsvpe-1966565 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:DLHGQaDgNx-1966684 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Qgujc32weh-1966750 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:TOOWRdPEok-1966964 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:t0YmWm85hP-1967049 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:VQAvAa5u6d-1967078 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:ocHEjdgwNg-1966553 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Twt9AUZshG-1966725 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:EUWqb73Pg9-1967044 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:98zWjbYzKo-1966048 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:v2tueyz23H-1966557 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Lcx98zWjbY-1966254 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:iWuNIqagrp-1966803 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:AtsfqU3npe-1966455 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:R5HftJtKv8-1966174 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:JCy8EtQsDg-1966708 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:H0ybNcJAkq-1966896 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:wm18VmS9Pa-1967037 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:PjogxvDE8s-1966871 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:NfFsTDQlt3-1967082 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:E4im53bcJP-1967089 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:dgwNgise9e-1967150 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:eEuRjWbWw5-1966465 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:qeA0e7FJRI-1966418 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Km0FbdGkLa-1966440 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:HqtZb3BeDY-1966628 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:0oifSkIaqL-1966380 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:MVnMl4FbA5-1966395 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:E84ovOs28p-1966402 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:JlBV01BYwr-1966761 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:YCgFQgKLqY-1966181 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:IYM3gq1lfw-1966883 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:7bp3VnLQ7x-1966406 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:p8DahqKSgS-1966514 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:P7QIH24Ow9-1967119 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:fpwT8bKnld-1966303 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:2dpfufxpXM-1966809 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:NxsVXGw5sd-1967156 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:NHjgDyEShV-1966266 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:y7pP3cb3pn-1966731 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:3mcxHiZvuJ-1966183 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:pSRtlqEQE5-1966568 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:cerzxxKWld-1966318 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:nldRu3sP1i-1966775 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:frK1dpNNbJ-1966168 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:0576kmpySn-1966281 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:NaV1FqkJkT-1966248 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:VtBmRwYuH5-1966569 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:6lIxpEKApM-1966705 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:KUYZjSU5MP-1967073 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Gd7JCy8EtQ-1966952 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:sH9WAQjicU-1967035 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:nWZgE7fWvE-1966265 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:tM7VSgqKqi-1966833 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:FdbhecBF38-1966694 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:ZfcefrCy3m-1966821 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:2fYaQ1HtpP-1966673 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Y9K8NaV1Fq-1966556 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:xw5IQ4xUUP-1966807 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Ch9QLBh5ey-1966986 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:783kMDmZv9-1966672 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:4M2xT14Sq2-1967153 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:KApMVF97id-1966244 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:qfflnjKDZE-1966532 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:g89IpkikSP-1966007 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:tgJR5HftJt-1966450 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:9rrf4urJJv-1966157 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:1dgjdITysv-1966019 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Zs3qhUYkGv-1966792 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:D6htVVbshj-1966301 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:jyXAsqmvAw-1966327 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:NTx2kdfTMp-1966488 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:DyEShVV5SY-1966982 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:6fY9K8NaV1-1966709 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:DgQAS22XcU-1967148 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:ZGFStAdahD-1966518 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:qU3npe6VJD-1966147 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:iDZF46LNFI-1966481 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:R57oCUDNGZ-1966711 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:sqmvAwVfr3-1967043 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:NpmSdUhIMg-1966140 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:2Nw5ke8iIc-1966487 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:oChzMZwxef-1966207 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:YnsvpeyJlB-1966321 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Z5m6M3mUiS-1966653 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:pZG8DEvNkc-1966766 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:dpNNbJzxhP-1966884 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:gQQ3EQwz1L-1966163 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:6c6S6bpiWe-1966538 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:2wDnpjUpw5-1966216 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:TehqwqZbs3-1966341 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:g55QZlTWXR-1966478 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:SYpi7VpCtf-1966718 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Dg5A8X2wlD-1966528 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:DNm1UH0GVK-1966928 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:sqM64WzYhP-1966730 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:RIoG1dh7y3-1966858 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:sBYCgFQgKL-1967106 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:uF6oChzMZw-1967124 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:9UJDUslYqA-1966500 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:6thCuO5h4V-1966516 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:THtPy7pP3c-1967039 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:WueYJBwFFS-1966683 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:n7tqpWVsFe-1966878 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:UZMXm5zeCZ-1966003 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:eWygu7ngCW-1966206 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:NSd8sZIuQx-1966352 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:A8X2wlDesP-1966252 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:xSDzHarVlI-1966285 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:L8JrUe1qWK-1966799 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:mg9AW2rENP-1967133 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:I24ez4KDKs-1966378 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:SylWeF1V4I-1966313 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:SC9fadb8CF-1966001 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:pRpsmDNm1U-1966002 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:9CLSPXdDSj-1966012 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:xsVXGw5sdR-1966021 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:gDyEShVV5S-1966028 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:zvh3y41Gnu-1966030 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:LeVdxGDdiZ-1966031 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:6QkkhTSFzt-1966033 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:BYCgFQgKLq-1966035 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:NxsH9WAQji-1966049 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:z23HQfZaUZ-1966051 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:2wlDesPc6S-1966054 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:piWe5mmLkL-1966056 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:KDZEokk8ON-1966058 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:kLcQzf8Dli-1966082 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:zeYOoL9lgT-1966084 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:1FZvMgpK7F-1966098 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:6WFedxhMNg-1966111 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:xHdJTKDuWK-1966137 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:e7FJRIoG1d-1966142 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:jwO0qcPrKO-1966144 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:TjxtqaSdvT-1966151 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:mtm8B5zFQ6-1966160 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:g6m0LjT32S-1966161 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:aXYSwYguT9-1966164 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Uh9sKhdDg7-1966169 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:4JaHPPp1e4-1966172 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:XXhzqCfzAJ-1966189 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:YVTL6jkhw8-1966214 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:2Cy1dgjdIT-1966217 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:mUiSKhOgBu-1966224 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:BeDY3mcxHi-1966235 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:it9QdShham-1966274 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:ZkEmOs7STZ-1966278 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:njKDZEokk8-1966288 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Z74ZZ6ltsq-1966290 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:VwbvwJDbfS-1966295 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:1cgk3p4UHh-1966315 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:LxBtNjO6xR-1966340 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:3qrdpKfUzL-1966343 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:qO9zV00ct0-1966353 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:iSqhd5bvKg-1966383 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:aA251eYggx-1966388 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:vb2rCEZKih-1966403 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:a5NSXEs4yJ-1966417 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:IDVqxP1E1D-1966489 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:tb9AwZ9CLS-1966494 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Z0EJzWTHSz-1966497 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:kMDmZv9rqv-1966523 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:lKiHumXzSO-1966527 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:O7GlpmyS6y-1966544 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:S5pfq2YmgA-1966545 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:OoSxSDzHar-1966561 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:WYP7rsyI2c-1966562 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:T2mu8A2YsL-1966564 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:sP1iVfeGMp-1966570 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:8eNOdDfiwY-1966578 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:m7dgM5OxAp-1966591 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:b4DtD9d2pj-1966605 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:gq1lfwZs4I-1966607 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:FPOkw6Z5ey-1966627 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:JJvvaRDUmg-1966629 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:D53j9e3EsB-1966634 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:rKOFdeSLLd-1966648 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:WRdPEokRew-1966656 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:4MpzxJa1s0-1966667 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:TI5uXP2Cy1-1966682 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:kmZ0EJzWTH-1966686 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:ObWXerIIbl-1966693 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:3jBNhefYtW-1966695 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:WPHrBBMonS-1966704 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:EDWocHEjdg-1966710 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:1wbKTDLcx9-1966715 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:kTckKDqQEa-1966720 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:JfJ9eRfGy2-1966722 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:WAQjicUo9c-1966727 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:kk8ON1m7dg-1966728 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:9J3XGOtuXw-1966733 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:fSBSWCWYP7-1966735 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:vAwVfr3idl-1966767 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:S88CgJl3i8-1966772 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:NP8IblIWiW-1966779 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:EIjmsdkU7g-1966780 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:UHhoLGtVD8-1966787 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:qnBnZZ3aHl-1966791 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:jYjLmaqfzk-1966827 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:0sNZXPeswG-1966829 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:1VMFE8wLyh-1966835 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:hGUYRfHzRj-1966837 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:TDQlt3h6Hv-1966838 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:aiVjk5RhWT-1966875 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:rtq8DXiDZF-1966882 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:KhdDg7NTx2-1966885 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:gCOPgOFdGL-1966894 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:T1T1tYZGFS-1966915 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:SR9Wh6fods-1966921 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:7jO6TI5uXP-1966926 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:db8CFPOkw6-1966935 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:4urJJvvaRD-1966937 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:iNx96fY9K8-1966953 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:QgKLqY9UJD-1966961 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:hXPVD2iO7G-1966973 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:eVdxGDdiZV-1966977 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:f6pkW80X7J-1966978 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:SO70QHGNvu-1966999 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:OYH8S88CgJ-1967016 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:8zWjbYzKow-1967034 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:48PNRjj3DD-1967057 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Yy8LVbM4sy-1967071 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Jy6Z1VMFE8-1967079 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:9IcTvZsajL-1967080 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:0KmLB6EKSy-1967091 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:Hp3CzCYxo0-1967092 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:KehRqdQ45V-1967097 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:7yxXqnBnZZ-1967099 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:O5h4V2Ihyr-1967109 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:mXzSO70QHG-1967116 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:iSKhOgBu1I-1967125 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:zOnmKuIQ5v-1967128 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:lTWXR7ELA3-1967135 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:o41RGxQguj-1967147 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:FScsI7jZbw-1967155 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection18:jgDyEShVV5-1967167 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:S0aMNZjI1c-1959877 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:jcK2OSUECd-1959966 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:t2LqFDWsJy-1959960 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:3aHlLOmN1D-1959874 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:xpXMqE8vqb-1959892 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:cJhOS0aMNZ-1959161 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:24Ow9ZaS5p-1959297 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:WMaEcBeOwI-1959162 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:tjYjLmaqfz-1959182 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:IvFQHXUeDr-1959812 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:E84ovOs28p-1959914 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:7Zs3qhUYkG-1959407 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:sqmvAwVfr3-1959531 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:bY6EJWjG1Q-1959254 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:Km9tWtTbHC-1959554 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:VwbvwJDbfS-1959807 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:74ZZ6ltsqM-1959802 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:ZR0deQyuFv-1959871 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:53ei8DX93Q-1959863 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:E1J10YI9Ic-1959896 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:qrdpKfUzLz-1959855 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:ciSqhd5bvK-1959992 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:WDThzxiPIB-1959868 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:xUUPcKWst9-1959890 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:qeA0e7FJRI-1959930 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:bZlJ2jkaiV-1959883 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:QP7QIH24Ow-1959762 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:TJJt0a8JSA-1959931 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:RTNg6m0LjT-1959917 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:HZCF56CQ2r-1959751 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:IFouscOh5b-1959752 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:onSSaYVmCT-1959759 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:x4M2xT14Sq-1959764 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:wKFnP1giNx-1959769 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:D6htVVbshj-1959781 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:GQLjnigN2x-1959784 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:Uqrd2LocGd-1959785 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:cp5fUAqNP8-1959787 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:sDgQAS22Xc-1959795 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:njKDZEokk8-1959800 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:1rOdTHtPy7-1959803 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:BF38MPAF9J-1959805 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:T2mu8A2YsL-1959820 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:dXhFUqrd2L-1959837 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:vNkcSylWeF-1959845 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:fCVxP7OkQ8-1959846 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:KHqu1cgk3p-1959847 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:agrp1FZvMg-1959854 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:GKDfzeYOoL-1959872 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:j6xQ9ZKgtR-1959891 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:HlB76WFedx-1959899 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:aA251eYggx-1959900 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:VD8kLcQzf8-1959902 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:NRcIHoUVkp-1959903 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:7bp3VnLQ7x-1959918 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:2aA251eYgg-1959937 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:nWYcJ1iIEO-1959968 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:sA1vktSCYL-1959971 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection15:RZw9ftMaa6-1959995 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:DNlHpXcaef-1018356 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:c4N4kra5NS-1018452 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:B5zFQ6ufIR-1018518 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:Bu1IjGlKiH-1018530 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:2XcUuhT0wV-1018613 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:y8LVbM4sya-1018673 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:zf8DliKZIe-1018688 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:kMDmZv9rqv-1018349 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:ke8iIcva9I-1018069 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:EX3hMCcGVm-1018314 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:ZlTWXR7ELA-1018060 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:KHqu1cgk3p-1018225 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:arVlIFWzOr-1018675 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:H0ybNcJAkq-1018730 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:rLsVANqtc1-1018234 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:fmC2dpfufx-1018663 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:bQUUpyg86d-1018335 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:zeCZ5m6M3m-1018507 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:bJzxhPgZ4m-1018474 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:7x3dQ5gtvL-1018512 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:5togCOPgOF-1018804 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:jxtqaSdvT3-1018809 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:IkmoEN6783-1018834 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:0ZeWygu7ng-1018953 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:JJt0a8JSA6-1018894 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:cryGQ5uEDW-1018864 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:BWwESaYUO4-1018884 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:JBwFFScsI7-1019033 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:g5A8X2wlDe-1019071 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:erIIbla6MS-1019051 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:L9VwKFnP1g-1019008 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:wQ53y6u6hh-1019050 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:c6S6bpiWe5-1019041 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:1iVfeGMpef-1019073 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:ufxpXMqE8v-1019167 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:3AHlB76WFe-1019138 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:M3mUiSKhOg-1019235 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:yJsoyhl76K-1018723 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:VdxGDdiZVu-1018875 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:Z9CLSPXdDS-1018945 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:3xFhnocryG-1019037 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:iOMOLpqsz8-1018482 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:HPOUnWZgE7-1018119 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:tpPQE1CMbU-1018110 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:7VpCtfP5CT-1019068 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:NACm0ZLiXm-1019121 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:e8dxwHUCvg-1018759 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:HvLyDxcBfF-1018880 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:z2XeQdRkwN-1018320 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:8Y7bp3VnLQ-1019217 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:Lpqsz8ripJ-1019230 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:VqxP1E1DAW-1019252 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:tYZGFStAda-1019265 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:OVpzLz8DpD-1019018 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:0EJzWTHSzA-1018139 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:kJ4Y0Qwm18-1019040 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:qAtaGd7JCy-1018870 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:SaYVmCTHSd-1018853 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:9zV00ct0Ym-1019084 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:ufIR9NSC9f-1018309 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:a6MShc3E6a-1018654 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:8tUKtgJR5H-1018328 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:2xCiIX53ei-1018386 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:6LNFIgbA21-1018968 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:z5I6UQKehR-1019132 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:MCcGVmcsuh-1018070 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:VtaL0Rdnfx-1018487 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:DNm1UH0GVK-1018506 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:z7ZQV3DWzv-1018746 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:cHEjdgwNgi-1019060 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:V38kxqxHdJ-1019196 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:sU70Ng1Ufm-1018191 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:4cohGUYRfH-1018691 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:KSyX8mdUWD-1018264 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:WsHZEdHlG8-1018656 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:eMu6PPkbT8-1019236 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:tHciSqhd5b-1018293 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:EokRewpbLo-1019014 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:uO5h4V2Ihy-1018106 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:yFuLqDTehq-1018376 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:Mp5BbTfBif-1018794 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:ITysvnR1H5-1018587 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:SYGhfbE84o-1018433 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:maqfzkJc6N-1019177 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:tSCYLAgUjR-1018772 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:suhpLAOWuB-1018510 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:hc3E6akeoC-1018602 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:66QkkhTSFz-1019002 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:hWTUtmdwGw-1018272 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:TrPbQUUpyg-1018484 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:ruocBKT1T1-1018950 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:SNqHiFamp2-1019244 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:WbWw5B4jtF-1018952 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:LqFDWsJy6Z-1019114 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:bDYD6IvPKQ-1018324 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:H5frK1dpNN-1018923 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:D9d2pjugyT-1019251 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:ud6thCuO5h-1019271 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:1UZMXm5zeC-1018964 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:g93qrdpKfU-1019158 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:b4DtD9d2pj-1018471 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:zkJc6NyoN7-1018933 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:nMl4FbA5Tw-1018902 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:pd4LgASshb-1018714 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:GLIaeQGiXi-1019256 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:2ADkD3eEuR-1018500 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:PmDUhL9rrf-1019248 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:0GVKXuF6oC-1018361 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:D9QFClkeqd-1018934 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:ei8DX93QIE-1019082 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:mBeIx12ADk-1018961 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:XYu7Y9pd4L-1019111 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:FeAP1OYf6D-1019176 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:gm3xFhnocr-1018136 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:AQjicUo9cZ-1018617 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:A0e7FJRIoG-1018889 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:jnKiSNqHiF-1019264 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:5u6dTepa2j-1018255 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:OWuB6zOnmK-1018365 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:3W48WTxOYH-1018154 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:7NIAtsfqU3-1018333 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:mKuIQ5vWue-1018837 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:p3VnLQ7x3d-1018941 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:AkqYOna8XQ-1018329 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:9fW0n8qMMe-1018650 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:XEs4yJsoyh-1018775 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:vvoaB8SFdb-1018848 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:2Nw5ke8iIc-1018313 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:iDZF46LNFI-1018315 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:tD9d2pjugy-1018318 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:MOLpqsz8ri-1018325 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:ddNGqwdIkm-1018362 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:eEuRjWbWw5-1018363 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:iWuNIqagrp-1018373 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:9ofWVwj6xQ-1018382 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:f2ZfASsU70-1018392 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:msRPq5JJKl-1018396 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:liKZIeekBW-1018412 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:2dpfufxpXM-1018419 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:8lkjrWdNaW-1018434 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:KZWrNdBccR-1018435 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:QV3DWzvb2r-1018438 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:0KYtHciSqh-1018450 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:CESKeDJpL8-1018456 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:Gw4rhkXNzO-1018459 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:YjLmaqfzkJ-1018461 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:dDQXsvKm0F-1018479 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:RF8tUKtgJR-1018481 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:JC12II4hmB-1018489 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:yrYrR70hL0-1018491 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:1LZsUOXMkB-1018493 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:pe6VJDDNlH-1018497 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:qcPrKOFdeS-1018502 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:ELA3kmZ0EJ-1018532 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:8oADhYUY20-1018534 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:VZHKEEo33U-1018535 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:7pnRkm9L9V-1018536 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:91UIjx53si-1018540 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:FBBHw1BpwI-1018543 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:T1T1tYZGFS-1018549 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:bTfBif1vl1-1018550 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:RzCC1wbKTD-1018561 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:hVV5SYpi7V-1018564 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:41Gnucg7zm-1018566 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:DdiZVuDg5A-1018567 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:LBh5eyBRkJ-1018568 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:C8dr3jBNhe-1018573 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:MbUX60d93O-1018582 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:XPVD2iO7Gl-1018583 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:kk8ON1m7dg-1018594 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:9J3XGOtuXw-1018599 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:js27lOT2mu-1018603 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:S88CgJl3i8-1018606 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:R3ybObWXer-1018611 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:wt9AUZshGz-1018623 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:2LocGddIqi-1018631 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:wlDesPc6Sk-1018642 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:wm18VmS9Pa-1018647 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:MPAF9J3XGO-1018651 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:OmijomrU3A-1018666 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:ktRZmWZWtT-1018674 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:xuD1xw5IQ4-1018685 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:gxc0KYtHci-1018694 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:gCoEc4N4kr-1018696 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:1cvGw4rhkX-1018703 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:dUhIMg2maY-1018722 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:keqdRF8tUK-1018725 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:ipfgTu7NIA-1018734 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:n7tqpWVsFe-1018736 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:DKs9p1pVKs-1018740 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:6ZsmxEbIjI-1018744 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:PKQA66NEUC-1018764 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:V1Sffgzb4D-1018767 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:wGwSpvUnlt-1018776 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:enoH0ybNcJ-1018782 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:gyTsLFlHqt-1018790 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:5HftJtKv8I-1018800 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:86daVUJedD-1018807 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:jwO0qcPrKO-1018810 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:bftdTOOWRd-1018818 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:3HQfZaUZxh-1018863 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:sVXGw5sdR5-1018865 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:vh3y41Gnuc-1018874 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:RjdeL6xz7Z-1018887 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:aHlLOmN1DH-1018897 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:A251eYggxc-1018899 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:0aMNZjI1cv-1018904 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:tRa2dQ1yAJ-1018911 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:WXgQQ3EQwz-1018914 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:GC6V1Sffgz-1018916 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:yg86daVUJe-1018924 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:NSC9fadb8C-1018970 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:pjugyTsLFl-1018975 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:pPQE1CMbUX-1018979 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:XzSO70QHGN-1018982 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:mg9AW2rENP-1018999 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:iFamp2FBBH-1019000 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:wI1F5zvh3y-1019015 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:OUnWZgE7fW-1019016 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:ejnq5fg0td-1019019 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:hCuO5h4V2I-1019027 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:201Apoctk0-1019038 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:33Ux4M2xT1-1019039 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:fflnjKDZEo-1019043 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:axVwbvwJDb-1019046 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:Owlpk6w9fW-1019047 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:5hguBRAz5n-1019056 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:pEKApMVF97-1019063 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:i8jyXAsqmv-1019078 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:dgM5OxApP5-1019098 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:rdpKfUzLzv-1019106 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:K2OSUECdoU-1019112 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:B76WFedxhM-1019118 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:1J10YI9IcT-1019119 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:R0deQyuFv2-1019122 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:ThzxiPIBeN-1019123 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:JPUE1J10YI-1019139 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:5VGKDfzeYO-1019147 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:WVwj6xQ9ZK-1019162 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:SyX8mdUWDR-1019165 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:KsAOnSaaJU-1019180 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:fbE84ovOs2-1019181 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:t3h6HvLyDx-1019188 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:SgqKqid2Lz-1019191 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:HciSqhd5bv-1019198 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:W20oifSkIa-1019199 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:GT3DX25tog-1019201 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:L0Rdnfx4Ja-1019203 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:PaCNXAUvcq-1019204 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:IR9NSC9fad-1019214 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:Wzvb2rCEZK-1019218 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:2XeQdRkwNQ-1019221 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:VKXuF6oChz-1019234 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:uB6zOnmKuI-1019238 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:FStAdahDFB-1019245 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection16:83kMDmZv9r-1019274 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:IEOjjsSDbD-1002046 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:va9INIC7g5-1002058 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:CMbUX60d93-1002095 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:PPkbT8VZHK-1002128 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:h4V2Ihyrru-1002143 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:0Ch9QLBh5e-1002165 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:KcUEBEaxkC-1002297 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:cR7Tddipfg-1002331 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:gk3p4UHhoL-1002214 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:vWDwhPjogx-1002298 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:pndlMlgjSs-1002485 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:3PCTJhb9uC-1002450 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Sq2WPHrBBM-1001982 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:FedxhMNgS4-1002010 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:1UfmC2dpfu-1002236 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:MlgjSs7fPt-1002177 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:32dl7js27l-1002196 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:sVANqtc1g2-1002271 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:DcyEv0KmLB-1002262 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:MOLpqsz8ri-1002519 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:KddNGqwdIk-1002548 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:DgVrOoSxSD-1002633 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Vfr3idlcer-1002666 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:a8wA6c6S6b-1002642 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:BpwI1F5zvh-1002588 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:4rhkXNzOo5-1002748 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:n7wF1rOdTH-1002619 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:S9PaIvFQHX-1002628 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:fTMp5BbTfB-1002339 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:jG1QGzAqiB-1002385 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:EKApMVF97i-1002455 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:GsbftdTOOW-1002139 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:9sKhdDg7NT-1002068 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:YrR70hL0jw-1002524 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:TQXp99vvoa-1002413 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:k8lkjrWdNa-1002779 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:ZZ74ZZ6lts-1002461 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:jHayI4WnWY-1002771 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:27lOT2mu8A-1002636 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:BMonSSaYVm-1002454 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Y1iC8KD9QF-1002723 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:CFPOkw6Z5e-1002834 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:3dQ5gtvLE1-1002805 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:zb4DtD9d2p-1002812 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:4D53j9e3Es-1002837 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:uEDWocHEjd-1002849 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:1ZJfJ9eRfG-1002893 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:sdR57oCUDN-1002854 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:ioo0qfflnj-1002860 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:9EIjmsdkU7-1002927 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Ss7fPtmqO9-1002925 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:esPc6SkEW9-1002935 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:RZmWZWtTHr-1002967 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:0Qwm18VmS9-1002156 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:fzKV3NHonR-1002090 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:0scE07yxXq-1002270 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:7TrPbQUUpy-1002765 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:9AwZ9CLSPX-1002389 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:pNUl9EIjms-1002179 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:THSd8eNOdD-1002650 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:SPUPHz7bgr-1002832 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:8NaV1FqkJk-1002459 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:wVIf9ktRZm-1002495 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:krmsRPq5JJ-1002965 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:gEUWqb73Pg-1002231 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:okRewpbLoR-1002406 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:xGDdiZVuDg-1002908 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:2s8KebVKIr-1002954 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:mDUhL9rrf4-1002384 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:4syafCVxP7-1002662 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:0FbdGkLaMi-1002343 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:zxhPgZ4mTc-1002511 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:ZvMgpK7FxG-1002217 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:ZGFStAdahD-1002538 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:ZeWygu7ngC-1002345 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:CxfpwT8bKn-1002446 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:6ADuvol0sc-1002667 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:H5IFouscOh-1002451 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:2Nw5ke8iIc-1002507 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:6thCuO5h4V-1002536 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:OdTHtPy7pP-1002158 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:44Hcfy93Yn-1002637 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:khw8qYsmHP-1002089 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:48WTxOYH8S-1002447 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:giNx96fY9K-1002904 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:1FqkJkTckK-1002151 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:yI2cVYf2Zf-1002181 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:oCHdqYy8LV-1002466 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:8CgJl3i8jy-1002675 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Kv8IDVqxP1-1002785 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Jhb9uC8dr3-1002174 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:9L9VwKFnP1-1002585 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:xApP5W2sXO-1002182 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:cZkEmOs7ST-1002481 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:iDZF46LNFI-1002501 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:hZKcmr80uU-1002914 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:VYf2ZfASsU-1002961 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:J10YI9IcTv-1002255 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:22XcUuhT0w-1002930 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:P5W2sXO3Mh-1002962 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:XgQQ3EQwz1-1002306 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:ZfASsU70Ng-1002685 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:flnjKDZEok-1002435 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:vWueYJBwFF-1002826 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:z23HQfZaUZ-1002148 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:gBu1IjGlKi-1002107 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Pg93qrdpKf-1002671 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:kJkTckKDqQ-1002931 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:lE6AuOmijo-1002263 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Oo5ohVG3Ci-1002323 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:5uXP2Cy1dg-1002577 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:kcSylWeF1V-1002264 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:m53bcJPUE1-1002956 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:d93OHZCF56-1002567 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:3BeDY3mcxH-1002378 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:ESKeDJpL85-1001848 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:D8oADhYUY2-1002111 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:3j9e3EsBYC-1002529 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:eNlYMxdt2L-1002955 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:UJedDQXsvK-1001891 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Ru3sP1iVfe-1002674 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:CiIX53ei8D-1002679 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:3sGTcPaCNX-1002801 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:kV4ud6thCu-1001947 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:HRCxfpwT8b-1002599 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:NQrtq8DXiD-1002029 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Oi8Y7bp3Vn-1002734 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:S5pfq2YmgA-1002565 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:F2fYaQ1Htp-1002816 -> DurabilityAmbiguousException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:SU5MPAEtRa-1001799 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:M7VSgqKqid-1001803 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:vDE8sA1vkt-1001845 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:WVsFeAP1OY-1001852 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:YjLmaqfzkJ-1001853 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:AY1iC8KD9Q-1001854 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:ipfgTu7NIA-1001870 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:BfFKZWrNdB-1001879 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Nm1UH0GVKX-1001898 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:QT4FTVWeMu-1001900 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:uhpLAOWuB6-1001902 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:x3dQ5gtvLE-1001904 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:YM3gq1lfwZ-1001909 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:YguT92Nw5k-1001914 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:JtKv8IDVqx-1001916 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:u1IjGlKiHu-1001922 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:SFzt9nBSim-1001929 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:urJJvvaRDU-1001939 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Zzxw9UdewY-1001945 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:zCC1wbKTDL-1001953 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:yrrucUjOVp-1001970 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:PVD2iO7Glp-1001975 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:5sdR57oCUD-1001981 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:YOoL9lgTSk-1001987 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:jI1cvGw4rh-1001988 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:3KUYZjSU5M-1002000 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:ECdoUtM7VS-1002004 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:oH0ybNcJAk-1002019 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:LQ7x3dQ5gt-1002025 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:JTKDuWKyrY-1002036 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Jj73szo6eH-1002044 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:SpvUnltsoV-1002045 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:tqaSdvT3Mx-1002050 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:W27pnRkm9L-1002081 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:M2xT14Sq2W-1002123 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:AhMufzKV3N-1002142 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:gwNgise9eo-1002144 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:GZ8Z1ZJfJ9-1002145 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:99vvoaB8SF-1002169 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:CF56CQ2ra8-1002170 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:zHarVlIFWz-1002184 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:GMpefpZG8D-1002189 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:v0KmLB6EKS-1002210 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:zxxKWld6Oz-1002213 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:yuFv2s8Keb-1002238 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Yu7Y9pd4Lg-1002247 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:ACm0ZLiXmZ-1002257 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:rp1FZvMgpK-1002269 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:HvLyDxcBfF-1002272 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:lnh3S1Yntj-1002274 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:HlLOmN1DHD-1002289 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:IHoUVkpzwN-1002290 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:swCV1tXFl0-1002292 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Ra2dQ1yAJG-1002303 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Lmtm8B5zFQ-1002307 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:4ovOs28p3s-1002329 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:drKC3kZzxw-1002346 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:dIkmoEN678-1002347 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:mBeIx12ADk-1002353 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:L0jwO0qcPr-1002355 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:3mUiSKhOgB-1002371 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:B6zOnmKuIQ-1002374 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:l1s66QkkhT-1002382 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:LIaeQGiXiK-1002392 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:eSLLdrKC3k-1002398 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:3Ux4M2xT14-1002431 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:J4Y0Qwm18V-1002432 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:cBF38MPAF9-1002436 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:rIIbla6MSh-1002443 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:93YnsvpeyJ-1002464 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:u8A2YsLQ1X-1002467 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:i8DX93QIEi-1002474 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:d8sZIuQxUp-1002479 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:PgjIW48PNR-1002484 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:SxSDzHarVl-1002492 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:gMlbl8cpRp-1002496 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:gtvLE1UZMX-1002497 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:IEX3hMCcGV-1002500 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:ZsUOXMkB96-1002502 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:QXsvKm0Fbd-1002544 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:0GVKXuF6oC-1002555 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Hz7bgrZWQ5-1002556 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:onR9NHjgDy-1002562 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:NH5hguBRAz-1002569 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:HrBBMonSSa-1002575 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:nBSimvwPf6-1002590 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:WXerIIbla6-1002592 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:dXhFUqrd2L-1002601 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:T0wVit9QdS-1002602 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:qQEacp5fUA-1002603 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:qfflnjKDZE-1002616 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:DOwlpk6w9f-1002620 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:c6SkEW9tqR-1002627 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:qMMebLAPHX-1002635 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:G8DEvNkcSy-1002661 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:dIqiBs3uhv-1002676 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:HlG8oJV0UG-1002705 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:KSyX8mdUWD-1002714 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:rU3AHlB76W-1002715 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:bIjIBHdne8-1002729 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:KZIeekBWwE-1002737 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:SKeDJpL85w-1002749 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:I4WnWYcJ1i-1002751 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:oG1dh7y3RT-1002753 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:c2VtaL0Rdn-1002764 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:NbJzxhPgZ4-1002787 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:p1e4fwpijr-1002791 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:m1UH0GVKXu-1002799 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:SNyrYrR70h-1002800 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:fgTu7NIAts-1002803 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Mi7RTaFlOo-1002847 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:iZVuDg5A8X-1002856 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:V5SYpi7VpC-1002857 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:Zm27n7wF1r-1002863 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:BHw1BpwI1F-1002864 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:qYsmHPOUnW-1002869 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:VD2iO7Glpm-1002872 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:GNvu5gTQXp-1002878 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:CQ2ra8wA6c-1002886 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:xQgujc32we-1002889 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:k0npOCjlNv-1002902 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:pkW80X7Jv2-1002905 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:MShc3E6ake-1002915 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:tsqM64WzYh-1002933 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:5mmLkL3Gza-1002937 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:IEiWuNIqag-1002946 -> AmbiguousTimeoutException Test Retrying update: GleamBookUsers0:_default:VolumeCollection7:C2dpfufxpX-1002952 -> AmbiguousTimeoutException 2022-02-08 00:05:04,434 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:651] All nodes in cluster: [ip:172.23.121.128 port:8091 ssh_username:root, ip:172.23.121.115 port:8091 ssh_username:root, ip:172.23.120.170 port:8091 ssh_username:root, ip:172.23.121.129 port:8091 ssh_username:root, ip:172.23.121.124 port:8091 ssh_username:root, ip:172.23.121.126 port:8091 ssh_username:root, ip:172.23.121.116 port:8091 ssh_username:root, ip:172.23.121.127 port:8091 ssh_username:root, ip:172.23.121.123 port:8091 ssh_username:root] 2022-02-08 00:05:04,438 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:05:04,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:05:05,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:05:05,237 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:05:05,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:05:05,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:05:05,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:05:05,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:05:05,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:05:05,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:05:05,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:05:05,374 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-08 00:05:05,375 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307505.37 2022-02-08 00:05:05,375 | infra | INFO | pool-3-thread-26 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307505.37' started 2022-02-08 00:05:05,380 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:05:05,381 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:05,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:05:05,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:05:05,826 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:05:05,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:05:05,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:05:05,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:05:05,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:05:05,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:05:05,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:05:05,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:05:05,927 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-08 00:05:05,928 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307505.93 2022-02-08 00:05:05,928 | infra | INFO | pool-3-thread-27 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307505.93' started 2022-02-08 00:05:05,930 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:05,933 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:05:06,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:05:06,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:05:06,292 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:05:06,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:05:06,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:05:06,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:05:06,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:05:06,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:05:06,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:05:06,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:05:06,401 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-08 00:05:06,403 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307506.4 2022-02-08 00:05:06,404 | infra | INFO | pool-3-thread-20 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307506.4' started 2022-02-08 00:05:06,404 | infra | DEBUG | pool-3-thread-20 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:06,407 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:05:06,767 | test | DEBUG | pool-3-thread-20 | [task:_compare:2755] Comparing 0 == 0 2022-02-08 00:05:06,769 | test | DEBUG | pool-3-thread-20 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.116': 0} for bucket 'GleamBookUsers0' 2022-02-08 00:05:06,769 | infra | INFO | pool-3-thread-20 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307506.4' completed 2022-02-08 00:05:07,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:05:07,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:05:07,588 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:05:07,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:05:07,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:05:07,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:05:07,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:05:07,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:05:07,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:05:07,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:05:07,726 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-08 00:05:07,726 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307507.73 2022-02-08 00:05:07,726 | infra | INFO | pool-3-thread-24 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307507.73' started 2022-02-08 00:05:07,729 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:07,732 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:05:08,000 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 2140401 == 0 2022-02-08 00:05:08,000 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2140401 == 0. Received: {'172.23.121.115': 2140401} for bucket 'GleamBookUsers0' 2022-02-08 00:05:08,000 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:08,003 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:08,003 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:08,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:05:08,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:05:09,003 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:05:09,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:05:09,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:05:09,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:05:09,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:05:09,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:05:09,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:05:09,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:05:09,155 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-08 00:05:09,155 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307509.16 2022-02-08 00:05:09,157 | infra | INFO | pool-3-thread-23 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307509.16' started 2022-02-08 00:05:09,157 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:09,160 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:05:09,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:05:10,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:05:10,447 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:05:10,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:05:10,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:05:10,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:05:10,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:05:10,500 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 2059496 == 0 2022-02-08 00:05:10,500 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2059496 == 0. Received: {'172.23.121.123': 2059496} for bucket 'GleamBookUsers0' 2022-02-08 00:05:10,500 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:10,503 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:10,503 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:10,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:05:10,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:05:10,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:05:10,575 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-08 00:05:10,575 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307510.58 2022-02-08 00:05:10,575 | infra | INFO | pool-3-thread-28 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307510.58' started 2022-02-08 00:05:10,576 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:10,578 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:05:11,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:05:11,394 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1860902 == 0 2022-02-08 00:05:11,394 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1860902 == 0. Received: {'172.23.121.124': 1860902} for bucket 'GleamBookUsers0' 2022-02-08 00:05:11,394 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:11,395 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:11,397 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:11,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:05:11,884 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:05:11,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:05:11,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:05:11,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:05:11,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:05:11,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:05:11,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:05:11,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:05:12,007 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-08 00:05:12,007 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307512.01 2022-02-08 00:05:12,007 | infra | INFO | pool-3-thread-29 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307512.01' started 2022-02-08 00:05:12,009 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:12,012 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:05:12,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:05:13,003 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:13,042 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1969425 == 0 2022-02-08 00:05:13,042 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1969425 == 0. Received: {'172.23.121.126': 1969425} for bucket 'GleamBookUsers0' 2022-02-08 00:05:13,042 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:13,045 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:13,045 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:13,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:05:13,305 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:05:13,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:05:13,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:05:13,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:05:13,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:05:13,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:05:13,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:05:13,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:05:13,437 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-08 00:05:13,437 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307513.44 2022-02-08 00:05:13,438 | infra | INFO | pool-3-thread-30 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307513.44' started 2022-02-08 00:05:13,440 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:13,443 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:05:14,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:05:14,213 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1806656 == 0 2022-02-08 00:05:14,213 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1806656 == 0. Received: {'172.23.121.127': 1806656} for bucket 'GleamBookUsers0' 2022-02-08 00:05:14,213 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:14,216 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:14,216 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:14,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:05:14,752 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:05:14,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:05:14,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:05:14,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:05:14,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:05:14,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:05:14,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:05:14,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:05:14,871 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2022-02-08 00:05:14,871 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307514.87 2022-02-08 00:05:14,871 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307505.37 2022-02-08 00:05:14,871 | infra | INFO | pool-3-thread-3 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307514.87' started 2022-02-08 00:05:14,875 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:15,036 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 2122206 == 0 2022-02-08 00:05:15,036 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2122206 == 0. Received: {'172.23.121.115': 2122206} for bucket 'GleamBookUsers0' 2022-02-08 00:05:15,036 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:15,039 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:15,039 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:15,503 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:16,013 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1848912 == 0 2022-02-08 00:05:16,013 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1848912 == 0. Received: {'172.23.121.128': 1848912} for bucket 'GleamBookUsers0' 2022-02-08 00:05:16,013 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:16,016 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:16,016 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:16,397 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:16,821 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1944915 == 0 2022-02-08 00:05:16,821 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1944915 == 0. Received: {'172.23.121.129': 1944915} for bucket 'GleamBookUsers0' 2022-02-08 00:05:16,821 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:16,822 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:16,822 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:17,769 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 2141910 == 0 2022-02-08 00:05:17,770 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2141910 == 0. Received: {'172.23.120.170': 2141910} for bucket 'GleamBookUsers0' 2022-02-08 00:05:17,770 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:17,773 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:17,773 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:17,951 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 2043092 == 0 2022-02-08 00:05:17,953 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2043092 == 0. Received: {'172.23.121.123': 2043092} for bucket 'GleamBookUsers0' 2022-02-08 00:05:17,953 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:17,956 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:17,956 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:18,045 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:18,335 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1808964 == 0 2022-02-08 00:05:18,335 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1808964 == 0. Received: {'172.23.121.124': 1808964} for bucket 'GleamBookUsers0' 2022-02-08 00:05:18,335 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:18,338 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:18,338 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:19,217 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:20,039 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:20,127 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1905379 == 0 2022-02-08 00:05:20,127 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1905379 == 0. Received: {'172.23.121.126': 1905379} for bucket 'GleamBookUsers0' 2022-02-08 00:05:20,127 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:20,128 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:20,128 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:21,016 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:21,046 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1766952 == 0 2022-02-08 00:05:21,046 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1766952 == 0. Received: {'172.23.121.127': 1766952} for bucket 'GleamBookUsers0' 2022-02-08 00:05:21,048 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:21,049 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:21,049 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:21,823 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:22,038 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 2066810 == 0 2022-02-08 00:05:22,038 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2066810 == 0. Received: {'172.23.121.115': 2066810} for bucket 'GleamBookUsers0' 2022-02-08 00:05:22,038 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:22,039 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:22,039 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:22,776 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:22,956 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:22,996 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1775600 == 0 2022-02-08 00:05:22,997 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1775600 == 0. Received: {'172.23.121.128': 1775600} for bucket 'GleamBookUsers0' 2022-02-08 00:05:22,997 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:22,999 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:23,000 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:23,338 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:23,572 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1908103 == 0 2022-02-08 00:05:23,572 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1908103 == 0. Received: {'172.23.121.129': 1908103} for bucket 'GleamBookUsers0' 2022-02-08 00:05:23,573 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:23,575 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:23,575 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:24,937 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 2094113 == 0 2022-02-08 00:05:24,937 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2094113 == 0. Received: {'172.23.120.170': 2094113} for bucket 'GleamBookUsers0' 2022-02-08 00:05:24,937 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:24,940 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:24,940 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:25,130 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:25,142 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1979600 == 0 2022-02-08 00:05:25,142 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1979600 == 0. Received: {'172.23.121.123': 1979600} for bucket 'GleamBookUsers0' 2022-02-08 00:05:25,142 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:25,145 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:25,145 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:25,331 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1763820 == 0 2022-02-08 00:05:25,331 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1763820 == 0. Received: {'172.23.121.124': 1763820} for bucket 'GleamBookUsers0' 2022-02-08 00:05:25,332 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:25,334 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:25,334 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:26,049 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:27,040 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:27,161 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1892153 == 0 2022-02-08 00:05:27,161 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1892153 == 0. Received: {'172.23.121.126': 1892153} for bucket 'GleamBookUsers0' 2022-02-08 00:05:27,161 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:27,164 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:27,164 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:27,871 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1753207 == 0 2022-02-08 00:05:27,872 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1753207 == 0. Received: {'172.23.121.127': 1753207} for bucket 'GleamBookUsers0' 2022-02-08 00:05:27,872 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:27,875 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:27,875 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:28,000 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:28,576 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:29,036 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 2052574 == 0 2022-02-08 00:05:29,036 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2052574 == 0. Received: {'172.23.121.115': 2052574} for bucket 'GleamBookUsers0' 2022-02-08 00:05:29,036 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:29,039 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:29,039 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:29,940 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:29,963 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1751352 == 0 2022-02-08 00:05:29,963 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1751352 == 0. Received: {'172.23.121.128': 1751352} for bucket 'GleamBookUsers0' 2022-02-08 00:05:29,963 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:29,966 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:29,966 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:30,145 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:30,335 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:30,378 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1897813 == 0 2022-02-08 00:05:30,378 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1897813 == 0. Received: {'172.23.121.129': 1897813} for bucket 'GleamBookUsers0' 2022-02-08 00:05:30,378 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:30,381 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:30,381 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:32,009 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 2094113 == 0 2022-02-08 00:05:32,010 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2094113 == 0. Received: {'172.23.120.170': 2094113} for bucket 'GleamBookUsers0' 2022-02-08 00:05:32,010 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:32,013 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:32,013 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:32,164 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:32,302 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1952726 == 0 2022-02-08 00:05:32,302 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1952726 == 0. Received: {'172.23.121.123': 1952726} for bucket 'GleamBookUsers0' 2022-02-08 00:05:32,302 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:32,305 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:32,305 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:32,423 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1763820 == 0 2022-02-08 00:05:32,423 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1763820 == 0. Received: {'172.23.121.124': 1763820} for bucket 'GleamBookUsers0' 2022-02-08 00:05:32,423 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:32,424 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:32,424 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:32,878 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:34,039 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:34,160 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1873627 == 0 2022-02-08 00:05:34,161 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1873627 == 0. Received: {'172.23.121.126': 1873627} for bucket 'GleamBookUsers0' 2022-02-08 00:05:34,161 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:34,236 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:34,236 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:34,697 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1734833 == 0 2022-02-08 00:05:34,698 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1734833 == 0. Received: {'172.23.121.127': 1734833} for bucket 'GleamBookUsers0' 2022-02-08 00:05:34,698 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:34,730 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:34,730 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:35,045 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:35,382 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:36,022 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 2036970 == 0 2022-02-08 00:05:36,022 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2036970 == 0. Received: {'172.23.121.115': 2036970} for bucket 'GleamBookUsers0' 2022-02-08 00:05:36,022 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:36,177 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:36,177 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:37,032 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:37,232 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1724884 == 0 2022-02-08 00:05:37,232 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1724884 == 0. Received: {'172.23.121.128': 1724884} for bucket 'GleamBookUsers0' 2022-02-08 00:05:37,232 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:37,233 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:37,234 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:37,309 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:37,378 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1884435 == 0 2022-02-08 00:05:37,380 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1884435 == 0. Received: {'172.23.121.129': 1884435} for bucket 'GleamBookUsers0' 2022-02-08 00:05:37,380 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:37,381 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:37,381 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:37,424 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:38,999 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 2066773 == 0 2022-02-08 00:05:38,999 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2066773 == 0. Received: {'172.23.120.170': 2066773} for bucket 'GleamBookUsers0' 2022-02-08 00:05:38,999 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:39,002 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:39,002 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:39,237 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:39,338 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1921420 == 0 2022-02-08 00:05:39,338 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1921420 == 0. Received: {'172.23.121.123': 1921420} for bucket 'GleamBookUsers0' 2022-02-08 00:05:39,338 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:39,341 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:39,341 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:39,453 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1747009 == 0 2022-02-08 00:05:39,453 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1747009 == 0. Received: {'172.23.121.124': 1747009} for bucket 'GleamBookUsers0' 2022-02-08 00:05:39,453 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:39,456 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:39,457 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:39,730 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:41,177 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:41,200 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1873627 == 0 2022-02-08 00:05:41,200 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1873627 == 0. Received: {'172.23.121.126': 1873627} for bucket 'GleamBookUsers0' 2022-02-08 00:05:41,200 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:41,203 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:41,203 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:41,529 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1715523 == 0 2022-02-08 00:05:41,530 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1715523 == 0. Received: {'172.23.121.127': 1715523} for bucket 'GleamBookUsers0' 2022-02-08 00:05:41,530 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:41,532 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:41,532 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:42,234 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:42,382 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:43,130 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1982893 == 0 2022-02-08 00:05:43,131 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1982893 == 0. Received: {'172.23.121.115': 1982893} for bucket 'GleamBookUsers0' 2022-02-08 00:05:43,131 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:43,132 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:43,132 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:44,002 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:44,246 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1724884 == 0 2022-02-08 00:05:44,247 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1724884 == 0. Received: {'172.23.121.128': 1724884} for bucket 'GleamBookUsers0' 2022-02-08 00:05:44,247 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:44,250 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:44,250 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:44,256 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1849905 == 0 2022-02-08 00:05:44,257 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1849905 == 0. Received: {'172.23.121.129': 1849905} for bucket 'GleamBookUsers0' 2022-02-08 00:05:44,257 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:44,259 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:44,259 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:44,342 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:44,457 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:45,926 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 2045965 == 0 2022-02-08 00:05:45,926 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2045965 == 0. Received: {'172.23.120.170': 2045965} for bucket 'GleamBookUsers0' 2022-02-08 00:05:45,926 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:45,928 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:45,928 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:46,203 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:46,299 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1899812 == 0 2022-02-08 00:05:46,299 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1899812 == 0. Received: {'172.23.121.123': 1899812} for bucket 'GleamBookUsers0' 2022-02-08 00:05:46,299 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:46,311 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:46,311 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:46,440 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1732613 == 0 2022-02-08 00:05:46,440 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1732613 == 0. Received: {'172.23.121.124': 1732613} for bucket 'GleamBookUsers0' 2022-02-08 00:05:46,440 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:46,443 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:46,443 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:46,532 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:48,132 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:48,263 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1810307 == 0 2022-02-08 00:05:48,263 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1810307 == 0. Received: {'172.23.121.126': 1810307} for bucket 'GleamBookUsers0' 2022-02-08 00:05:48,263 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:48,266 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:48,266 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:48,509 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1685281 == 0 2022-02-08 00:05:48,510 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1685281 == 0. Received: {'172.23.121.127': 1685281} for bucket 'GleamBookUsers0' 2022-02-08 00:05:48,510 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:48,513 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:48,513 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:49,250 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:49,259 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:50,112 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1961819 == 0 2022-02-08 00:05:50,112 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1961819 == 0. Received: {'172.23.121.115': 1961819} for bucket 'GleamBookUsers0' 2022-02-08 00:05:50,112 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:50,115 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:50,115 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:50,930 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:51,219 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1813245 == 0 2022-02-08 00:05:51,219 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1813245 == 0. Received: {'172.23.121.129': 1813245} for bucket 'GleamBookUsers0' 2022-02-08 00:05:51,219 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:51,220 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:51,221 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:51,293 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1693010 == 0 2022-02-08 00:05:51,293 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1693010 == 0. Received: {'172.23.121.128': 1693010} for bucket 'GleamBookUsers0' 2022-02-08 00:05:51,293 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:51,296 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:51,296 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:51,311 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:51,444 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:53,013 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 2031869 == 0 2022-02-08 00:05:53,013 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2031869 == 0. Received: {'172.23.120.170': 2031869} for bucket 'GleamBookUsers0' 2022-02-08 00:05:53,013 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:53,016 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:53,016 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:53,266 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:53,306 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1671173 == 0 2022-02-08 00:05:53,308 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1671173 == 0. Received: {'172.23.121.124': 1671173} for bucket 'GleamBookUsers0' 2022-02-08 00:05:53,308 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:53,309 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:53,309 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:53,319 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1881556 == 0 2022-02-08 00:05:53,319 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1881556 == 0. Received: {'172.23.121.123': 1881556} for bucket 'GleamBookUsers0' 2022-02-08 00:05:53,319 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:53,322 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:53,322 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:53,513 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:55,115 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:55,322 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1798661 == 0 2022-02-08 00:05:55,322 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1798661 == 0. Received: {'172.23.121.126': 1798661} for bucket 'GleamBookUsers0' 2022-02-08 00:05:55,322 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:55,325 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:55,325 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:55,404 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1660693 == 0 2022-02-08 00:05:55,405 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1660693 == 0. Received: {'172.23.121.127': 1660693} for bucket 'GleamBookUsers0' 2022-02-08 00:05:55,405 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:55,407 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:55,407 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:56,221 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:56,298 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:57,052 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1961819 == 0 2022-02-08 00:05:57,052 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1961819 == 0. Received: {'172.23.121.115': 1961819} for bucket 'GleamBookUsers0' 2022-02-08 00:05:57,052 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:57,053 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:57,053 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:58,016 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:58,055 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1803229 == 0 2022-02-08 00:05:58,055 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1803229 == 0. Received: {'172.23.121.129': 1803229} for bucket 'GleamBookUsers0' 2022-02-08 00:05:58,055 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:58,058 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:58,058 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:58,263 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1693010 == 0 2022-02-08 00:05:58,263 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1693010 == 0. Received: {'172.23.121.128': 1693010} for bucket 'GleamBookUsers0' 2022-02-08 00:05:58,263 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:05:58,265 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:05:58,265 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:05:58,309 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:05:58,322 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:00,118 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 2010335 == 0 2022-02-08 00:06:00,119 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 2010335 == 0. Received: {'172.23.120.170': 2010335} for bucket 'GleamBookUsers0' 2022-02-08 00:06:00,119 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:00,121 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:00,122 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:00,216 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1652640 == 0 2022-02-08 00:06:00,216 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1652640 == 0. Received: {'172.23.121.124': 1652640} for bucket 'GleamBookUsers0' 2022-02-08 00:06:00,216 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:00,217 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:00,219 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:00,319 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1855338 == 0 2022-02-08 00:06:00,319 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1855338 == 0. Received: {'172.23.121.123': 1855338} for bucket 'GleamBookUsers0' 2022-02-08 00:06:00,319 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:00,322 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:00,322 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:00,326 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:00,408 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:02,055 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:02,265 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1628503 == 0 2022-02-08 00:06:02,266 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1628503 == 0. Received: {'172.23.121.127': 1628503} for bucket 'GleamBookUsers0' 2022-02-08 00:06:02,266 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:02,269 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:02,269 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:02,299 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1798661 == 0 2022-02-08 00:06:02,299 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1798661 == 0. Received: {'172.23.121.126': 1798661} for bucket 'GleamBookUsers0' 2022-02-08 00:06:02,299 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:02,301 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:02,302 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:03,059 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:03,266 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:03,999 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1922637 == 0 2022-02-08 00:06:03,999 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1922637 == 0. Received: {'172.23.121.115': 1922637} for bucket 'GleamBookUsers0' 2022-02-08 00:06:03,999 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:04,000 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:04,000 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:05,032 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1791061 == 0 2022-02-08 00:06:05,033 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1791061 == 0. Received: {'172.23.121.129': 1791061} for bucket 'GleamBookUsers0' 2022-02-08 00:06:05,033 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:05,036 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:05,036 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:05,122 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:05,203 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1684022 == 0 2022-02-08 00:06:05,203 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1684022 == 0. Received: {'172.23.121.128': 1684022} for bucket 'GleamBookUsers0' 2022-02-08 00:06:05,203 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:05,206 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:05,206 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:05,219 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:05,322 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:07,124 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1635132 == 0 2022-02-08 00:06:07,124 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1635132 == 0. Received: {'172.23.121.124': 1635132} for bucket 'GleamBookUsers0' 2022-02-08 00:06:07,124 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:07,127 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:07,127 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:07,244 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1989013 == 0 2022-02-08 00:06:07,244 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1989013 == 0. Received: {'172.23.120.170': 1989013} for bucket 'GleamBookUsers0' 2022-02-08 00:06:07,244 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:07,246 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:07,247 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:07,269 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:07,302 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:07,338 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1846190 == 0 2022-02-08 00:06:07,339 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1846190 == 0. Received: {'172.23.121.123': 1846190} for bucket 'GleamBookUsers0' 2022-02-08 00:06:07,339 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:07,342 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:07,342 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:09,002 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:09,158 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1628503 == 0 2022-02-08 00:06:09,160 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1628503 == 0. Received: {'172.23.121.127': 1628503} for bucket 'GleamBookUsers0' 2022-02-08 00:06:09,160 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:09,161 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:09,161 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:09,378 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1779635 == 0 2022-02-08 00:06:09,378 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1779635 == 0. Received: {'172.23.121.126': 1779635} for bucket 'GleamBookUsers0' 2022-02-08 00:06:09,378 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:09,381 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:09,381 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:10,036 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:10,207 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:10,888 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1922637 == 0 2022-02-08 00:06:10,888 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1922637 == 0. Received: {'172.23.121.115': 1922637} for bucket 'GleamBookUsers0' 2022-02-08 00:06:10,888 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:10,891 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:10,891 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:11,835 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1791061 == 0 2022-02-08 00:06:11,835 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1791061 == 0. Received: {'172.23.121.129': 1791061} for bucket 'GleamBookUsers0' 2022-02-08 00:06:11,835 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:11,838 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:11,838 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:12,128 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:12,148 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1643402 == 0 2022-02-08 00:06:12,150 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1643402 == 0. Received: {'172.23.121.128': 1643402} for bucket 'GleamBookUsers0' 2022-02-08 00:06:12,150 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:12,151 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:12,151 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:12,247 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:12,342 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:14,007 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1579782 == 0 2022-02-08 00:06:14,007 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1579782 == 0. Received: {'172.23.121.124': 1579782} for bucket 'GleamBookUsers0' 2022-02-08 00:06:14,007 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:14,010 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:14,010 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:14,163 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:14,285 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1942721 == 0 2022-02-08 00:06:14,286 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1942721 == 0. Received: {'172.23.120.170': 1942721} for bucket 'GleamBookUsers0' 2022-02-08 00:06:14,286 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:14,288 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:14,289 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:14,355 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1846190 == 0 2022-02-08 00:06:14,355 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1846190 == 0. Received: {'172.23.121.123': 1846190} for bucket 'GleamBookUsers0' 2022-02-08 00:06:14,355 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:14,358 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:14,358 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:14,381 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:15,891 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:15,973 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1617295 == 0 2022-02-08 00:06:15,973 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1617295 == 0. Received: {'172.23.121.127': 1617295} for bucket 'GleamBookUsers0' 2022-02-08 00:06:15,974 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:15,976 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:15,976 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:16,348 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1779635 == 0 2022-02-08 00:06:16,348 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1779635 == 0. Received: {'172.23.121.126': 1779635} for bucket 'GleamBookUsers0' 2022-02-08 00:06:16,348 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:16,349 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:16,349 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:16,838 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:17,153 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:17,769 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1922637 == 0 2022-02-08 00:06:17,769 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1922637 == 0. Received: {'172.23.121.115': 1922637} for bucket 'GleamBookUsers0' 2022-02-08 00:06:17,769 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:17,770 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:17,772 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:18,714 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1763269 == 0 2022-02-08 00:06:18,714 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1763269 == 0. Received: {'172.23.121.129': 1763269} for bucket 'GleamBookUsers0' 2022-02-08 00:06:18,714 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:18,717 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:18,717 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:19,012 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:19,092 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1643402 == 0 2022-02-08 00:06:19,092 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1643402 == 0. Received: {'172.23.121.128': 1643402} for bucket 'GleamBookUsers0' 2022-02-08 00:06:19,092 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:19,095 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:19,095 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:19,289 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:19,358 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:20,792 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1547489 == 0 2022-02-08 00:06:20,793 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1547489 == 0. Received: {'172.23.121.124': 1547489} for bucket 'GleamBookUsers0' 2022-02-08 00:06:20,793 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:20,796 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:20,796 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:20,977 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:21,348 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1942721 == 0 2022-02-08 00:06:21,349 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1942721 == 0. Received: {'172.23.120.170': 1942721} for bucket 'GleamBookUsers0' 2022-02-08 00:06:21,349 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:21,351 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:21,351 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:21,351 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:21,391 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1823962 == 0 2022-02-08 00:06:21,391 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1823962 == 0. Received: {'172.23.121.123': 1823962} for bucket 'GleamBookUsers0' 2022-02-08 00:06:21,391 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:21,394 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:21,394 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:22,730 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1575537 == 0 2022-02-08 00:06:22,730 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1575537 == 0. Received: {'172.23.121.127': 1575537} for bucket 'GleamBookUsers0' 2022-02-08 00:06:22,730 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:22,733 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:22,733 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:22,772 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:23,181 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1703949 == 0 2022-02-08 00:06:23,181 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1703949 == 0. Received: {'172.23.121.126': 1703949} for bucket 'GleamBookUsers0' 2022-02-08 00:06:23,181 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:23,184 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:23,184 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:23,719 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:24,095 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:24,598 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1922637 == 0 2022-02-08 00:06:24,598 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1922637 == 0. Received: {'172.23.121.115': 1922637} for bucket 'GleamBookUsers0' 2022-02-08 00:06:24,599 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:24,601 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:24,601 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:25,469 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1692522 == 0 2022-02-08 00:06:25,469 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1692522 == 0. Received: {'172.23.121.129': 1692522} for bucket 'GleamBookUsers0' 2022-02-08 00:06:25,470 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:25,476 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:25,476 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:25,796 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:25,996 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1632282 == 0 2022-02-08 00:06:25,996 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1632282 == 0. Received: {'172.23.121.128': 1632282} for bucket 'GleamBookUsers0' 2022-02-08 00:06:25,996 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:25,999 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:25,999 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:26,352 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:26,394 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:27,444 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1547489 == 0 2022-02-08 00:06:27,444 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1547489 == 0. Received: {'172.23.121.124': 1547489} for bucket 'GleamBookUsers0' 2022-02-08 00:06:27,444 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:27,446 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:27,446 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:27,733 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:28,184 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:28,325 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1929894 == 0 2022-02-08 00:06:28,325 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1929894 == 0. Received: {'172.23.120.170': 1929894} for bucket 'GleamBookUsers0' 2022-02-08 00:06:28,325 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:28,328 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:28,328 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:28,365 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1798746 == 0 2022-02-08 00:06:28,365 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1798746 == 0. Received: {'172.23.121.123': 1798746} for bucket 'GleamBookUsers0' 2022-02-08 00:06:28,365 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:28,367 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:28,368 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:29,571 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1539353 == 0 2022-02-08 00:06:29,572 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1539353 == 0. Received: {'172.23.121.127': 1539353} for bucket 'GleamBookUsers0' 2022-02-08 00:06:29,572 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:29,576 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:29,576 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:29,601 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:30,020 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1689785 == 0 2022-02-08 00:06:30,020 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1689785 == 0. Received: {'172.23.121.126': 1689785} for bucket 'GleamBookUsers0' 2022-02-08 00:06:30,020 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:30,023 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:30,023 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:30,477 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:31,000 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:31,447 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1922637 == 0 2022-02-08 00:06:31,447 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1922637 == 0. Received: {'172.23.121.115': 1922637} for bucket 'GleamBookUsers0' 2022-02-08 00:06:31,447 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:31,448 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:31,450 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:32,167 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1692522 == 0 2022-02-08 00:06:32,167 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1692522 == 0. Received: {'172.23.121.129': 1692522} for bucket 'GleamBookUsers0' 2022-02-08 00:06:32,167 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:32,170 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:32,170 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:32,447 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:32,861 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1623178 == 0 2022-02-08 00:06:32,861 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1623178 == 0. Received: {'172.23.121.128': 1623178} for bucket 'GleamBookUsers0' 2022-02-08 00:06:32,861 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:32,865 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:32,865 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:33,329 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:33,368 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:34,026 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1515953 == 0 2022-02-08 00:06:34,026 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1515953 == 0. Received: {'172.23.121.124': 1515953} for bucket 'GleamBookUsers0' 2022-02-08 00:06:34,026 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:34,028 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:34,028 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:34,578 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:35,023 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:35,237 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1781390 == 0 2022-02-08 00:06:35,237 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1781390 == 0. Received: {'172.23.121.123': 1781390} for bucket 'GleamBookUsers0' 2022-02-08 00:06:35,237 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:35,240 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:35,240 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:35,273 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1907370 == 0 2022-02-08 00:06:35,275 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1907370 == 0. Received: {'172.23.120.170': 1907370} for bucket 'GleamBookUsers0' 2022-02-08 00:06:35,275 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:35,276 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:35,276 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:36,426 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1526461 == 0 2022-02-08 00:06:36,426 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1526461 == 0. Received: {'172.23.121.127': 1526461} for bucket 'GleamBookUsers0' 2022-02-08 00:06:36,426 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:36,428 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:36,428 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:36,450 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:36,868 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1689785 == 0 2022-02-08 00:06:36,868 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1689785 == 0. Received: {'172.23.121.126': 1689785} for bucket 'GleamBookUsers0' 2022-02-08 00:06:36,868 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:36,871 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:36,871 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:37,171 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:37,865 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:38,230 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1908779 == 0 2022-02-08 00:06:38,230 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1908779 == 0. Received: {'172.23.121.115': 1908779} for bucket 'GleamBookUsers0' 2022-02-08 00:06:38,230 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:38,233 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:38,233 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:38,911 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1681114 == 0 2022-02-08 00:06:38,911 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1681114 == 0. Received: {'172.23.121.129': 1681114} for bucket 'GleamBookUsers0' 2022-02-08 00:06:38,911 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:38,913 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:38,914 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:39,029 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:39,693 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1587938 == 0 2022-02-08 00:06:39,693 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1587938 == 0. Received: {'172.23.121.128': 1587938} for bucket 'GleamBookUsers0' 2022-02-08 00:06:39,693 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:39,697 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:39,697 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:40,240 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:40,276 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:40,617 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1498869 == 0 2022-02-08 00:06:40,618 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1498869 == 0. Received: {'172.23.121.124': 1498869} for bucket 'GleamBookUsers0' 2022-02-08 00:06:40,618 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:40,619 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:40,619 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:41,430 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:41,871 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:42,203 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1907370 == 0 2022-02-08 00:06:42,203 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1907370 == 0. Received: {'172.23.120.170': 1907370} for bucket 'GleamBookUsers0' 2022-02-08 00:06:42,203 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:42,206 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:42,206 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:42,299 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1729978 == 0 2022-02-08 00:06:42,299 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1729978 == 0. Received: {'172.23.121.123': 1729978} for bucket 'GleamBookUsers0' 2022-02-08 00:06:42,299 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:42,302 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:42,302 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:43,168 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1526461 == 0 2022-02-08 00:06:43,168 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1526461 == 0. Received: {'172.23.121.127': 1526461} for bucket 'GleamBookUsers0' 2022-02-08 00:06:43,168 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:43,171 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:43,171 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:43,234 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:43,696 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1668233 == 0 2022-02-08 00:06:43,696 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1668233 == 0. Received: {'172.23.121.126': 1668233} for bucket 'GleamBookUsers0' 2022-02-08 00:06:43,697 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:43,697 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:43,698 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:43,914 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:44,706 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:45,049 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1865057 == 0 2022-02-08 00:06:45,049 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1865057 == 0. Received: {'172.23.121.115': 1865057} for bucket 'GleamBookUsers0' 2022-02-08 00:06:45,049 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:45,051 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:45,051 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:45,622 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:45,644 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1647184 == 0 2022-02-08 00:06:45,644 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1647184 == 0. Received: {'172.23.121.129': 1647184} for bucket 'GleamBookUsers0' 2022-02-08 00:06:45,644 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:45,647 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:45,647 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:46,677 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1579744 == 0 2022-02-08 00:06:46,677 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1579744 == 0. Received: {'172.23.121.128': 1579744} for bucket 'GleamBookUsers0' 2022-02-08 00:06:46,677 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:46,680 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:46,680 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:47,203 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1474367 == 0 2022-02-08 00:06:47,203 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1474367 == 0. Received: {'172.23.121.124': 1474367} for bucket 'GleamBookUsers0' 2022-02-08 00:06:47,203 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:47,206 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:47,206 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:47,207 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:47,302 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:48,171 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:48,704 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:49,174 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1729978 == 0 2022-02-08 00:06:49,174 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1729978 == 0. Received: {'172.23.121.123': 1729978} for bucket 'GleamBookUsers0' 2022-02-08 00:06:49,176 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:49,177 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:49,177 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:49,193 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1898146 == 0 2022-02-08 00:06:49,194 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1898146 == 0. Received: {'172.23.120.170': 1898146} for bucket 'GleamBookUsers0' 2022-02-08 00:06:49,194 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:49,196 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:49,196 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:49,960 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1516157 == 0 2022-02-08 00:06:49,961 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1516157 == 0. Received: {'172.23.121.127': 1516157} for bucket 'GleamBookUsers0' 2022-02-08 00:06:49,961 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:49,963 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:49,963 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:50,052 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:50,467 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1645036 == 0 2022-02-08 00:06:50,467 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1645036 == 0. Received: {'172.23.121.126': 1645036} for bucket 'GleamBookUsers0' 2022-02-08 00:06:50,467 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:50,469 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:50,470 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:50,648 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:51,680 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:51,871 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1844511 == 0 2022-02-08 00:06:51,871 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1844511 == 0. Received: {'172.23.121.115': 1844511} for bucket 'GleamBookUsers0' 2022-02-08 00:06:51,871 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:51,875 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:51,875 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:52,207 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:52,342 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1624104 == 0 2022-02-08 00:06:52,344 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1624104 == 0. Received: {'172.23.121.129': 1624104} for bucket 'GleamBookUsers0' 2022-02-08 00:06:52,344 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:52,345 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:52,345 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:53,802 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1532798 == 0 2022-02-08 00:06:53,803 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1532798 == 0. Received: {'172.23.121.128': 1532798} for bucket 'GleamBookUsers0' 2022-02-08 00:06:53,803 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:53,805 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:53,805 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:53,846 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1461373 == 0 2022-02-08 00:06:53,846 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1461373 == 0. Received: {'172.23.121.124': 1461373} for bucket 'GleamBookUsers0' 2022-02-08 00:06:53,846 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:53,848 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:53,848 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:54,178 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:54,196 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:54,967 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:55,470 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:56,157 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1708184 == 0 2022-02-08 00:06:56,157 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1708184 == 0. Received: {'172.23.121.123': 1708184} for bucket 'GleamBookUsers0' 2022-02-08 00:06:56,157 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:56,160 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:56,160 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:56,210 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1882182 == 0 2022-02-08 00:06:56,211 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1882182 == 0. Received: {'172.23.120.170': 1882182} for bucket 'GleamBookUsers0' 2022-02-08 00:06:56,211 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:56,213 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:56,213 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:56,694 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1494017 == 0 2022-02-08 00:06:56,694 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1494017 == 0. Received: {'172.23.121.127': 1494017} for bucket 'GleamBookUsers0' 2022-02-08 00:06:56,694 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:56,696 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:56,697 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:56,875 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:57,270 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1613446 == 0 2022-02-08 00:06:57,270 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1613446 == 0. Received: {'172.23.121.126': 1613446} for bucket 'GleamBookUsers0' 2022-02-08 00:06:57,272 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:57,273 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:57,273 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:57,345 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:58,664 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1825007 == 0 2022-02-08 00:06:58,664 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1825007 == 0. Received: {'172.23.121.115': 1825007} for bucket 'GleamBookUsers0' 2022-02-08 00:06:58,664 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:58,667 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:58,667 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:06:58,806 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:58,849 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:06:59,042 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1624104 == 0 2022-02-08 00:06:59,042 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1624104 == 0. Received: {'172.23.121.129': 1624104} for bucket 'GleamBookUsers0' 2022-02-08 00:06:59,042 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:06:59,043 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:06:59,043 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:00,420 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1429206 == 0 2022-02-08 00:07:00,420 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1429206 == 0. Received: {'172.23.121.124': 1429206} for bucket 'GleamBookUsers0' 2022-02-08 00:07:00,421 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:00,423 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:00,423 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:00,746 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1524334 == 0 2022-02-08 00:07:00,746 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1524334 == 0. Received: {'172.23.121.128': 1524334} for bucket 'GleamBookUsers0' 2022-02-08 00:07:00,746 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:00,749 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:00,750 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:01,161 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:01,214 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:01,697 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:02,273 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:03,098 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1685302 == 0 2022-02-08 00:07:03,098 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1685302 == 0. Received: {'172.23.121.123': 1685302} for bucket 'GleamBookUsers0' 2022-02-08 00:07:03,098 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:03,101 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:03,101 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:03,230 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1862098 == 0 2022-02-08 00:07:03,230 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1862098 == 0. Received: {'172.23.120.170': 1862098} for bucket 'GleamBookUsers0' 2022-02-08 00:07:03,230 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:03,233 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:03,233 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:03,486 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1463551 == 0 2022-02-08 00:07:03,486 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1463551 == 0. Received: {'172.23.121.127': 1463551} for bucket 'GleamBookUsers0' 2022-02-08 00:07:03,486 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:03,489 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:03,490 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:03,667 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:04,026 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1581490 == 0 2022-02-08 00:07:04,026 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1581490 == 0. Received: {'172.23.121.126': 1581490} for bucket 'GleamBookUsers0' 2022-02-08 00:07:04,026 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:04,029 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:04,029 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:04,043 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:05,410 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1758543 == 0 2022-02-08 00:07:05,411 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1758543 == 0. Received: {'172.23.121.115': 1758543} for bucket 'GleamBookUsers0' 2022-02-08 00:07:05,411 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:05,413 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:05,413 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:05,423 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:05,750 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:05,789 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1594174 == 0 2022-02-08 00:07:05,790 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1594174 == 0. Received: {'172.23.121.129': 1594174} for bucket 'GleamBookUsers0' 2022-02-08 00:07:05,790 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:05,792 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:05,792 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:06,946 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1415666 == 0 2022-02-08 00:07:06,947 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1415666 == 0. Received: {'172.23.121.124': 1415666} for bucket 'GleamBookUsers0' 2022-02-08 00:07:06,947 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:06,948 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:06,948 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:07,602 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1500920 == 0 2022-02-08 00:07:07,602 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1500920 == 0. Received: {'172.23.121.128': 1500920} for bucket 'GleamBookUsers0' 2022-02-08 00:07:07,602 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:07,605 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:07,605 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:08,102 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:08,233 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:08,490 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:09,029 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:10,089 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1685302 == 0 2022-02-08 00:07:10,089 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1685302 == 0. Received: {'172.23.121.123': 1685302} for bucket 'GleamBookUsers0' 2022-02-08 00:07:10,089 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:10,092 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:10,092 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:10,180 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1822072 == 0 2022-02-08 00:07:10,180 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1822072 == 0. Received: {'172.23.120.170': 1822072} for bucket 'GleamBookUsers0' 2022-02-08 00:07:10,180 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:10,183 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:10,183 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:10,410 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1446195 == 0 2022-02-08 00:07:10,411 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1446195 == 0. Received: {'172.23.121.127': 1446195} for bucket 'GleamBookUsers0' 2022-02-08 00:07:10,411 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:10,413 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:10,414 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:10,417 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:10,789 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1581490 == 0 2022-02-08 00:07:10,789 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1581490 == 0. Received: {'172.23.121.126': 1581490} for bucket 'GleamBookUsers0' 2022-02-08 00:07:10,789 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:10,790 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:10,790 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:10,792 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:11,948 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:12,109 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1758543 == 0 2022-02-08 00:07:12,111 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1758543 == 0. Received: {'172.23.121.115': 1758543} for bucket 'GleamBookUsers0' 2022-02-08 00:07:12,111 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:12,112 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:12,112 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:12,540 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1581216 == 0 2022-02-08 00:07:12,542 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1581216 == 0. Received: {'172.23.121.129': 1581216} for bucket 'GleamBookUsers0' 2022-02-08 00:07:12,542 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:12,543 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:12,543 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:12,605 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:13,484 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1415666 == 0 2022-02-08 00:07:13,484 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1415666 == 0. Received: {'172.23.121.124': 1415666} for bucket 'GleamBookUsers0' 2022-02-08 00:07:13,486 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:13,487 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:13,487 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:14,309 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1480828 == 0 2022-02-08 00:07:14,309 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1480828 == 0. Received: {'172.23.121.128': 1480828} for bucket 'GleamBookUsers0' 2022-02-08 00:07:14,309 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:14,312 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:14,312 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:15,128 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:15,184 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:15,414 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:15,792 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:17,114 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:17,253 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1677086 == 0 2022-02-08 00:07:17,255 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1677086 == 0. Received: {'172.23.121.123': 1677086} for bucket 'GleamBookUsers0' 2022-02-08 00:07:17,255 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:17,256 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:17,257 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:17,273 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1783142 == 0 2022-02-08 00:07:17,273 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1783142 == 0. Received: {'172.23.120.170': 1783142} for bucket 'GleamBookUsers0' 2022-02-08 00:07:17,273 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:17,276 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:17,276 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:17,448 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1411821 == 0 2022-02-08 00:07:17,448 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1411821 == 0. Received: {'172.23.121.127': 1411821} for bucket 'GleamBookUsers0' 2022-02-08 00:07:17,448 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:17,451 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:17,451 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:17,545 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:17,674 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1559382 == 0 2022-02-08 00:07:17,674 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1559382 == 0. Received: {'172.23.121.126': 1559382} for bucket 'GleamBookUsers0' 2022-02-08 00:07:17,674 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:17,677 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:17,677 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:18,487 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:18,891 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1745521 == 0 2022-02-08 00:07:18,891 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1745521 == 0. Received: {'172.23.121.115': 1745521} for bucket 'GleamBookUsers0' 2022-02-08 00:07:18,892 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:18,894 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:18,894 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:19,275 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1560210 == 0 2022-02-08 00:07:19,275 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1560210 == 0. Received: {'172.23.121.129': 1560210} for bucket 'GleamBookUsers0' 2022-02-08 00:07:19,275 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:19,276 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:19,278 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:19,312 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:19,974 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1375730 == 0 2022-02-08 00:07:19,974 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1375730 == 0. Received: {'172.23.121.124': 1375730} for bucket 'GleamBookUsers0' 2022-02-08 00:07:19,974 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:19,976 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:19,976 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:21,040 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1456486 == 0 2022-02-08 00:07:21,040 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1456486 == 0. Received: {'172.23.121.128': 1456486} for bucket 'GleamBookUsers0' 2022-02-08 00:07:21,040 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:21,042 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:21,043 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:22,259 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:22,276 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:22,453 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:22,677 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:23,894 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:24,278 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:24,305 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1767718 == 0 2022-02-08 00:07:24,305 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1767718 == 0. Received: {'172.23.120.170': 1767718} for bucket 'GleamBookUsers0' 2022-02-08 00:07:24,305 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:24,306 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:24,308 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:24,505 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1660772 == 0 2022-02-08 00:07:24,505 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1660772 == 0. Received: {'172.23.121.123': 1660772} for bucket 'GleamBookUsers0' 2022-02-08 00:07:24,505 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:24,507 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:24,507 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:24,551 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1379301 == 0 2022-02-08 00:07:24,552 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1379301 == 0. Received: {'172.23.121.127': 1379301} for bucket 'GleamBookUsers0' 2022-02-08 00:07:24,552 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:24,553 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:24,553 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:24,683 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1548164 == 0 2022-02-08 00:07:24,683 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1548164 == 0. Received: {'172.23.121.126': 1548164} for bucket 'GleamBookUsers0' 2022-02-08 00:07:24,683 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:24,684 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:24,684 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:24,977 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:25,621 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1733529 == 0 2022-02-08 00:07:25,622 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1733529 == 0. Received: {'172.23.121.115': 1733529} for bucket 'GleamBookUsers0' 2022-02-08 00:07:25,622 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:25,625 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:25,625 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:25,993 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1506552 == 0 2022-02-08 00:07:25,993 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1506552 == 0. Received: {'172.23.121.129': 1506552} for bucket 'GleamBookUsers0' 2022-02-08 00:07:25,993 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:25,996 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:25,996 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:26,043 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:26,500 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1315635 == 0 2022-02-08 00:07:26,500 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1315635 == 0. Received: {'172.23.121.124': 1315635} for bucket 'GleamBookUsers0' 2022-02-08 00:07:26,500 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:26,502 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:26,503 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:27,737 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1419477 == 0 2022-02-08 00:07:27,737 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1419477 == 0. Received: {'172.23.121.128': 1419477} for bucket 'GleamBookUsers0' 2022-02-08 00:07:27,737 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:27,740 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:27,740 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:29,309 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:29,507 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:29,555 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:29,686 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:30,627 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:30,996 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:31,081 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1748130 == 0 2022-02-08 00:07:31,082 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1748130 == 0. Received: {'172.23.120.170': 1748130} for bucket 'GleamBookUsers0' 2022-02-08 00:07:31,082 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:31,084 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:31,084 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:31,503 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:31,556 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1631735 == 0 2022-02-08 00:07:31,556 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1631735 == 0. Received: {'172.23.121.123': 1631735} for bucket 'GleamBookUsers0' 2022-02-08 00:07:31,558 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:31,559 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:31,559 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:31,615 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1367711 == 0 2022-02-08 00:07:31,615 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1367711 == 0. Received: {'172.23.121.127': 1367711} for bucket 'GleamBookUsers0' 2022-02-08 00:07:31,617 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:31,618 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:31,618 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:31,664 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1510793 == 0 2022-02-08 00:07:31,665 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1510793 == 0. Received: {'172.23.121.126': 1510793} for bucket 'GleamBookUsers0' 2022-02-08 00:07:31,665 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:31,667 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:31,667 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:32,335 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1701099 == 0 2022-02-08 00:07:32,335 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1701099 == 0. Received: {'172.23.121.115': 1701099} for bucket 'GleamBookUsers0' 2022-02-08 00:07:32,336 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:32,338 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:32,338 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:32,740 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:32,789 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1493976 == 0 2022-02-08 00:07:32,790 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1493976 == 0. Received: {'172.23.121.129': 1493976} for bucket 'GleamBookUsers0' 2022-02-08 00:07:32,790 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:32,793 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:32,793 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:33,003 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1315635 == 0 2022-02-08 00:07:33,005 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1315635 == 0. Received: {'172.23.121.124': 1315635} for bucket 'GleamBookUsers0' 2022-02-08 00:07:33,005 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:33,006 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:33,006 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:34,484 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1407017 == 0 2022-02-08 00:07:34,484 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1407017 == 0. Received: {'172.23.121.128': 1407017} for bucket 'GleamBookUsers0' 2022-02-08 00:07:34,484 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:34,487 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:34,487 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:36,085 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:36,561 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:36,618 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:36,668 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:37,339 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:37,717 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1748130 == 0 2022-02-08 00:07:37,719 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1748130 == 0. Received: {'172.23.120.170': 1748130} for bucket 'GleamBookUsers0' 2022-02-08 00:07:37,719 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:37,723 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:37,723 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:37,793 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:38,006 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:38,645 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1619159 == 0 2022-02-08 00:07:38,645 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1619159 == 0. Received: {'172.23.121.123': 1619159} for bucket 'GleamBookUsers0' 2022-02-08 00:07:38,647 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:38,648 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:38,648 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:38,726 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1367711 == 0 2022-02-08 00:07:38,726 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1367711 == 0. Received: {'172.23.121.127': 1367711} for bucket 'GleamBookUsers0' 2022-02-08 00:07:38,727 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:38,730 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:38,730 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:38,799 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1498879 == 0 2022-02-08 00:07:38,799 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1498879 == 0. Received: {'172.23.121.126': 1498879} for bucket 'GleamBookUsers0' 2022-02-08 00:07:38,799 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:38,802 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:38,802 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:39,147 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1701099 == 0 2022-02-08 00:07:39,147 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1701099 == 0. Received: {'172.23.121.115': 1701099} for bucket 'GleamBookUsers0' 2022-02-08 00:07:39,147 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:39,150 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:39,150 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:39,487 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:39,632 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1315635 == 0 2022-02-08 00:07:39,632 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1315635 == 0. Received: {'172.23.121.124': 1315635} for bucket 'GleamBookUsers0' 2022-02-08 00:07:39,632 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:39,635 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:39,635 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:39,678 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1468537 == 0 2022-02-08 00:07:39,678 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1468537 == 0. Received: {'172.23.121.129': 1468537} for bucket 'GleamBookUsers0' 2022-02-08 00:07:39,678 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:39,680 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:39,680 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:41,167 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1407017 == 0 2022-02-08 00:07:41,168 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1407017 == 0. Received: {'172.23.121.128': 1407017} for bucket 'GleamBookUsers0' 2022-02-08 00:07:41,168 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:41,171 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:41,171 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:42,726 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:43,650 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:43,732 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:43,802 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:44,150 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:44,418 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1726226 == 0 2022-02-08 00:07:44,418 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1726226 == 0. Received: {'172.23.120.170': 1726226} for bucket 'GleamBookUsers0' 2022-02-08 00:07:44,418 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:44,421 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:44,421 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:44,635 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:44,681 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:45,744 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1333689 == 0 2022-02-08 00:07:45,744 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1333689 == 0. Received: {'172.23.121.127': 1333689} for bucket 'GleamBookUsers0' 2022-02-08 00:07:45,744 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:45,747 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:45,747 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:45,756 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1590377 == 0 2022-02-08 00:07:45,756 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1590377 == 0. Received: {'172.23.121.123': 1590377} for bucket 'GleamBookUsers0' 2022-02-08 00:07:45,756 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:45,759 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:45,759 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:45,951 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1471291 == 0 2022-02-08 00:07:45,953 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1471291 == 0. Received: {'172.23.121.126': 1471291} for bucket 'GleamBookUsers0' 2022-02-08 00:07:45,953 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:45,954 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:45,954 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:46,171 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:46,338 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1701099 == 0 2022-02-08 00:07:46,338 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1701099 == 0. Received: {'172.23.121.115': 1701099} for bucket 'GleamBookUsers0' 2022-02-08 00:07:46,338 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:46,342 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:46,342 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:46,388 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1303680 == 0 2022-02-08 00:07:46,388 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1303680 == 0. Received: {'172.23.121.124': 1303680} for bucket 'GleamBookUsers0' 2022-02-08 00:07:46,388 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:46,391 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:46,391 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:46,569 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1468537 == 0 2022-02-08 00:07:46,571 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1468537 == 0. Received: {'172.23.121.129': 1468537} for bucket 'GleamBookUsers0' 2022-02-08 00:07:46,571 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:46,572 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:46,572 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:47,789 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1397417 == 0 2022-02-08 00:07:47,789 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1397417 == 0. Received: {'172.23.121.128': 1397417} for bucket 'GleamBookUsers0' 2022-02-08 00:07:47,790 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:47,792 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:47,792 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:49,421 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:50,749 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:50,759 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:50,956 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:51,022 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1708627 == 0 2022-02-08 00:07:51,022 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1708627 == 0. Received: {'172.23.120.170': 1708627} for bucket 'GleamBookUsers0' 2022-02-08 00:07:51,022 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:51,023 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:51,023 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:51,342 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:51,392 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:51,573 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:52,351 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1271642 == 0 2022-02-08 00:07:52,351 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1271642 == 0. Received: {'172.23.121.127': 1271642} for bucket 'GleamBookUsers0' 2022-02-08 00:07:52,351 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:52,355 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:52,355 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:52,721 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1569757 == 0 2022-02-08 00:07:52,721 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1569757 == 0. Received: {'172.23.121.123': 1569757} for bucket 'GleamBookUsers0' 2022-02-08 00:07:52,721 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:52,724 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:52,724 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:52,793 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:53,101 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1460123 == 0 2022-02-08 00:07:53,101 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1460123 == 0. Received: {'172.23.121.126': 1460123} for bucket 'GleamBookUsers0' 2022-02-08 00:07:53,101 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:53,104 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:53,104 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:53,276 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1281211 == 0 2022-02-08 00:07:53,276 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1281211 == 0. Received: {'172.23.121.124': 1281211} for bucket 'GleamBookUsers0' 2022-02-08 00:07:53,276 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:53,279 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:53,279 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:53,494 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1689605 == 0 2022-02-08 00:07:53,496 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1689605 == 0. Received: {'172.23.121.115': 1689605} for bucket 'GleamBookUsers0' 2022-02-08 00:07:53,496 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:53,499 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:53,500 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:53,530 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1448775 == 0 2022-02-08 00:07:53,530 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1448775 == 0. Received: {'172.23.121.129': 1448775} for bucket 'GleamBookUsers0' 2022-02-08 00:07:53,530 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:53,533 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:53,533 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:54,447 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1390311 == 0 2022-02-08 00:07:54,447 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1390311 == 0. Received: {'172.23.121.128': 1390311} for bucket 'GleamBookUsers0' 2022-02-08 00:07:54,447 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:54,450 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:54,450 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:56,025 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:57,355 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:57,648 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1708627 == 0 2022-02-08 00:07:57,648 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1708627 == 0. Received: {'172.23.120.170': 1708627} for bucket 'GleamBookUsers0' 2022-02-08 00:07:57,648 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:57,651 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:57,651 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:57,724 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:58,105 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:58,279 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:58,500 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:58,533 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:58,948 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1271642 == 0 2022-02-08 00:07:58,948 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1271642 == 0. Received: {'172.23.121.127': 1271642} for bucket 'GleamBookUsers0' 2022-02-08 00:07:58,948 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:58,951 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:58,951 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:59,451 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:07:59,529 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1544287 == 0 2022-02-08 00:07:59,530 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1544287 == 0. Received: {'172.23.121.123': 1544287} for bucket 'GleamBookUsers0' 2022-02-08 00:07:59,530 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:59,532 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:59,533 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:59,871 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1427575 == 0 2022-02-08 00:07:59,872 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1427575 == 0. Received: {'172.23.121.126': 1427575} for bucket 'GleamBookUsers0' 2022-02-08 00:07:59,872 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:59,875 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:59,875 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:07:59,908 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1219531 == 0 2022-02-08 00:07:59,908 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1219531 == 0. Received: {'172.23.121.124': 1219531} for bucket 'GleamBookUsers0' 2022-02-08 00:07:59,908 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:07:59,911 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:07:59,911 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:00,315 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1427019 == 0 2022-02-08 00:08:00,315 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1427019 == 0. Received: {'172.23.121.129': 1427019} for bucket 'GleamBookUsers0' 2022-02-08 00:08:00,315 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:00,318 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:00,318 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:00,415 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1648114 == 0 2022-02-08 00:08:00,417 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1648114 == 0. Received: {'172.23.121.115': 1648114} for bucket 'GleamBookUsers0' 2022-02-08 00:08:00,417 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:00,418 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:00,418 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:01,053 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1378081 == 0 2022-02-08 00:08:01,053 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1378081 == 0. Received: {'172.23.121.128': 1378081} for bucket 'GleamBookUsers0' 2022-02-08 00:08:01,053 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:01,055 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:01,055 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:02,653 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:03,953 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:04,217 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1685339 == 0 2022-02-08 00:08:04,217 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1685339 == 0. Received: {'172.23.120.170': 1685339} for bucket 'GleamBookUsers0' 2022-02-08 00:08:04,219 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:04,220 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:04,220 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:04,533 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:04,875 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:04,911 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:05,319 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:05,420 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:05,539 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1244113 == 0 2022-02-08 00:08:05,540 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1244113 == 0. Received: {'172.23.121.127': 1244113} for bucket 'GleamBookUsers0' 2022-02-08 00:08:05,540 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:05,542 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:05,542 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:06,056 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:06,477 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1506783 == 0 2022-02-08 00:08:06,479 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1506783 == 0. Received: {'172.23.121.123': 1506783} for bucket 'GleamBookUsers0' 2022-02-08 00:08:06,479 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:06,482 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:06,482 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:06,573 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1205817 == 0 2022-02-08 00:08:06,575 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1205817 == 0. Received: {'172.23.121.124': 1205817} for bucket 'GleamBookUsers0' 2022-02-08 00:08:06,575 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:06,576 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:06,576 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:06,661 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1392159 == 0 2022-02-08 00:08:06,661 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1392159 == 0. Received: {'172.23.121.126': 1392159} for bucket 'GleamBookUsers0' 2022-02-08 00:08:06,661 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:06,664 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:06,664 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:07,078 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1415891 == 0 2022-02-08 00:08:07,078 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1415891 == 0. Received: {'172.23.121.129': 1415891} for bucket 'GleamBookUsers0' 2022-02-08 00:08:07,078 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:07,082 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:07,082 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:07,266 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1627474 == 0 2022-02-08 00:08:07,266 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1627474 == 0. Received: {'172.23.121.115': 1627474} for bucket 'GleamBookUsers0' 2022-02-08 00:08:07,266 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:07,269 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:07,269 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:07,647 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1358109 == 0 2022-02-08 00:08:07,648 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1358109 == 0. Received: {'172.23.121.128': 1358109} for bucket 'GleamBookUsers0' 2022-02-08 00:08:07,648 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:07,650 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:07,651 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:09,221 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:10,546 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:10,796 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1656369 == 0 2022-02-08 00:08:10,796 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1656369 == 0. Received: {'172.23.120.170': 1656369} for bucket 'GleamBookUsers0' 2022-02-08 00:08:10,796 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:10,799 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:10,799 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:11,483 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:11,578 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:11,667 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:12,081 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1233090 == 0 2022-02-08 00:08:12,081 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1233090 == 0. Received: {'172.23.121.127': 1233090} for bucket 'GleamBookUsers0' 2022-02-08 00:08:12,082 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:12,082 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:12,084 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:12,084 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:12,269 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:12,651 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:13,213 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1194072 == 0 2022-02-08 00:08:13,213 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1194072 == 0. Received: {'172.23.121.124': 1194072} for bucket 'GleamBookUsers0' 2022-02-08 00:08:13,213 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:13,216 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:13,216 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:13,414 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1506783 == 0 2022-02-08 00:08:13,414 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1506783 == 0. Received: {'172.23.121.123': 1506783} for bucket 'GleamBookUsers0' 2022-02-08 00:08:13,414 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:13,417 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:13,417 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:13,460 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1378917 == 0 2022-02-08 00:08:13,460 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1378917 == 0. Received: {'172.23.121.126': 1378917} for bucket 'GleamBookUsers0' 2022-02-08 00:08:13,460 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:13,463 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:13,463 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:13,811 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1408771 == 0 2022-02-08 00:08:13,811 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1408771 == 0. Received: {'172.23.121.129': 1408771} for bucket 'GleamBookUsers0' 2022-02-08 00:08:13,811 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:13,813 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:13,813 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:14,071 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1606964 == 0 2022-02-08 00:08:14,072 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1606964 == 0. Received: {'172.23.121.115': 1606964} for bucket 'GleamBookUsers0' 2022-02-08 00:08:14,072 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:14,073 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:14,073 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:14,322 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1358109 == 0 2022-02-08 00:08:14,322 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1358109 == 0. Received: {'172.23.121.128': 1358109} for bucket 'GleamBookUsers0' 2022-02-08 00:08:14,322 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:14,325 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:14,325 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:15,799 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:17,084 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:17,273 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1636931 == 0 2022-02-08 00:08:17,273 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1636931 == 0. Received: {'172.23.120.170': 1636931} for bucket 'GleamBookUsers0' 2022-02-08 00:08:17,273 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:17,276 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:17,276 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:18,216 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:18,418 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:18,463 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:18,529 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1182446 == 0 2022-02-08 00:08:18,529 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1182446 == 0. Received: {'172.23.121.127': 1182446} for bucket 'GleamBookUsers0' 2022-02-08 00:08:18,529 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:18,532 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:18,532 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:18,815 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:19,073 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:19,325 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:19,655 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1174934 == 0 2022-02-08 00:08:19,655 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1174934 == 0. Received: {'172.23.121.124': 1174934} for bucket 'GleamBookUsers0' 2022-02-08 00:08:19,655 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:19,658 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:19,658 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:20,232 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1369877 == 0 2022-02-08 00:08:20,233 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1369877 == 0. Received: {'172.23.121.126': 1369877} for bucket 'GleamBookUsers0' 2022-02-08 00:08:20,233 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:20,234 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:20,234 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:20,349 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1506783 == 0 2022-02-08 00:08:20,349 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1506783 == 0. Received: {'172.23.121.123': 1506783} for bucket 'GleamBookUsers0' 2022-02-08 00:08:20,349 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:20,351 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:20,352 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:20,723 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1391269 == 0 2022-02-08 00:08:20,723 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1391269 == 0. Received: {'172.23.121.129': 1391269} for bucket 'GleamBookUsers0' 2022-02-08 00:08:20,723 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:20,726 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:20,726 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:20,933 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1594758 == 0 2022-02-08 00:08:20,933 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1594758 == 0. Received: {'172.23.121.115': 1594758} for bucket 'GleamBookUsers0' 2022-02-08 00:08:20,933 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:20,934 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:20,936 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:21,043 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1321995 == 0 2022-02-08 00:08:21,043 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1321995 == 0. Received: {'172.23.121.128': 1321995} for bucket 'GleamBookUsers0' 2022-02-08 00:08:21,043 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:21,046 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:21,046 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:22,276 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:23,532 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:23,799 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1615445 == 0 2022-02-08 00:08:23,801 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1615445 == 0. Received: {'172.23.120.170': 1615445} for bucket 'GleamBookUsers0' 2022-02-08 00:08:23,801 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:23,802 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:23,802 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:24,658 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:25,016 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1172532 == 0 2022-02-08 00:08:25,016 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1172532 == 0. Received: {'172.23.121.127': 1172532} for bucket 'GleamBookUsers0' 2022-02-08 00:08:25,016 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:25,017 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:25,017 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:25,236 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:25,352 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:25,726 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:25,936 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:26,035 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1131980 == 0 2022-02-08 00:08:26,035 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1131980 == 0. Received: {'172.23.121.124': 1131980} for bucket 'GleamBookUsers0' 2022-02-08 00:08:26,035 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:26,036 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:26,036 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:26,046 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:26,872 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1359739 == 0 2022-02-08 00:08:26,872 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1359739 == 0. Received: {'172.23.121.126': 1359739} for bucket 'GleamBookUsers0' 2022-02-08 00:08:26,872 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:26,875 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:26,875 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:27,108 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1477765 == 0 2022-02-08 00:08:27,108 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1477765 == 0. Received: {'172.23.121.123': 1477765} for bucket 'GleamBookUsers0' 2022-02-08 00:08:27,109 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:27,111 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:27,112 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:27,539 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1370063 == 0 2022-02-08 00:08:27,540 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1370063 == 0. Received: {'172.23.121.129': 1370063} for bucket 'GleamBookUsers0' 2022-02-08 00:08:27,540 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:27,542 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:27,542 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:27,732 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1533568 == 0 2022-02-08 00:08:27,732 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1533568 == 0. Received: {'172.23.121.115': 1533568} for bucket 'GleamBookUsers0' 2022-02-08 00:08:27,733 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:27,734 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:27,734 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:27,812 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1292650 == 0 2022-02-08 00:08:27,813 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1292650 == 0. Received: {'172.23.121.128': 1292650} for bucket 'GleamBookUsers0' 2022-02-08 00:08:27,813 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:27,815 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:27,815 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:28,802 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:30,019 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:30,290 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1608557 == 0 2022-02-08 00:08:30,292 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1608557 == 0. Received: {'172.23.120.170': 1608557} for bucket 'GleamBookUsers0' 2022-02-08 00:08:30,292 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:30,293 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:30,293 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:31,038 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:31,454 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1156634 == 0 2022-02-08 00:08:31,456 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1156634 == 0. Received: {'172.23.121.127': 1156634} for bucket 'GleamBookUsers0' 2022-02-08 00:08:31,456 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:31,457 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:31,457 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:31,877 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:32,114 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:32,426 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1092748 == 0 2022-02-08 00:08:32,426 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1092748 == 0. Received: {'172.23.121.124': 1092748} for bucket 'GleamBookUsers0' 2022-02-08 00:08:32,426 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:32,427 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:32,428 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:32,543 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:32,734 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:32,815 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:33,516 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1332219 == 0 2022-02-08 00:08:33,516 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1332219 == 0. Received: {'172.23.121.126': 1332219} for bucket 'GleamBookUsers0' 2022-02-08 00:08:33,516 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:33,519 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:33,519 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:33,891 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1459649 == 0 2022-02-08 00:08:33,891 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1459649 == 0. Received: {'172.23.121.123': 1459649} for bucket 'GleamBookUsers0' 2022-02-08 00:08:33,891 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:33,892 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:33,894 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:33,986 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1333045 == 0 2022-02-08 00:08:33,986 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1333045 == 0. Received: {'172.23.121.129': 1333045} for bucket 'GleamBookUsers0' 2022-02-08 00:08:33,986 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:33,989 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:33,990 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:34,496 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1533568 == 0 2022-02-08 00:08:34,496 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1533568 == 0. Received: {'172.23.121.115': 1533568} for bucket 'GleamBookUsers0' 2022-02-08 00:08:34,496 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:34,500 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:34,500 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:34,543 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1292650 == 0 2022-02-08 00:08:34,543 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1292650 == 0. Received: {'172.23.121.128': 1292650} for bucket 'GleamBookUsers0' 2022-02-08 00:08:34,543 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:34,546 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:34,546 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:35,293 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:36,459 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:36,799 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1608557 == 0 2022-02-08 00:08:36,799 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1608557 == 0. Received: {'172.23.120.170': 1608557} for bucket 'GleamBookUsers0' 2022-02-08 00:08:36,799 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:36,802 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:36,802 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:37,428 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:37,871 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1113782 == 0 2022-02-08 00:08:37,871 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1113782 == 0. Received: {'172.23.121.127': 1113782} for bucket 'GleamBookUsers0' 2022-02-08 00:08:37,871 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:37,874 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:37,874 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:38,519 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:38,812 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1084644 == 0 2022-02-08 00:08:38,812 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1084644 == 0. Received: {'172.23.121.124': 1084644} for bucket 'GleamBookUsers0' 2022-02-08 00:08:38,812 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:38,815 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:38,815 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:38,894 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:38,990 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:39,500 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:39,546 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:40,119 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1299049 == 0 2022-02-08 00:08:40,119 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1299049 == 0. Received: {'172.23.121.126': 1299049} for bucket 'GleamBookUsers0' 2022-02-08 00:08:40,119 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:40,122 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:40,122 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:40,519 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1312051 == 0 2022-02-08 00:08:40,520 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1312051 == 0. Received: {'172.23.121.129': 1312051} for bucket 'GleamBookUsers0' 2022-02-08 00:08:40,520 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:40,523 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:40,523 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:40,688 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1422923 == 0 2022-02-08 00:08:40,688 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1422923 == 0. Received: {'172.23.121.123': 1422923} for bucket 'GleamBookUsers0' 2022-02-08 00:08:40,690 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:40,691 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:40,691 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:41,220 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1267674 == 0 2022-02-08 00:08:41,220 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1267674 == 0. Received: {'172.23.121.128': 1267674} for bucket 'GleamBookUsers0' 2022-02-08 00:08:41,221 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:41,223 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:41,223 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:41,260 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1533568 == 0 2022-02-08 00:08:41,260 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1533568 == 0. Received: {'172.23.121.115': 1533568} for bucket 'GleamBookUsers0' 2022-02-08 00:08:41,260 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:41,263 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:41,263 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:41,802 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:42,874 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:43,288 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1566499 == 0 2022-02-08 00:08:43,289 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1566499 == 0. Received: {'172.23.120.170': 1566499} for bucket 'GleamBookUsers0' 2022-02-08 00:08:43,289 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:43,290 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:43,290 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:43,815 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:44,280 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1081588 == 0 2022-02-08 00:08:44,282 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1081588 == 0. Received: {'172.23.121.127': 1081588} for bucket 'GleamBookUsers0' 2022-02-08 00:08:44,282 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:44,283 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:44,283 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:45,124 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:45,187 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1073642 == 0 2022-02-08 00:08:45,187 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1073642 == 0. Received: {'172.23.121.124': 1073642} for bucket 'GleamBookUsers0' 2022-02-08 00:08:45,187 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:45,190 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:45,190 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:45,525 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:45,691 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:46,224 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:46,265 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:46,653 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1276863 == 0 2022-02-08 00:08:46,654 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1276863 == 0. Received: {'172.23.121.126': 1276863} for bucket 'GleamBookUsers0' 2022-02-08 00:08:46,654 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:46,655 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:46,655 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:47,035 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1301317 == 0 2022-02-08 00:08:47,036 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1301317 == 0. Received: {'172.23.121.129': 1301317} for bucket 'GleamBookUsers0' 2022-02-08 00:08:47,036 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:47,038 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:47,038 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:47,496 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1407405 == 0 2022-02-08 00:08:47,496 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1407405 == 0. Received: {'172.23.121.123': 1407405} for bucket 'GleamBookUsers0' 2022-02-08 00:08:47,496 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:47,500 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:47,500 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:47,836 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1267674 == 0 2022-02-08 00:08:47,836 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1267674 == 0. Received: {'172.23.121.128': 1267674} for bucket 'GleamBookUsers0' 2022-02-08 00:08:47,836 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:47,838 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:47,839 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:48,003 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1533568 == 0 2022-02-08 00:08:48,003 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1533568 == 0. Received: {'172.23.121.115': 1533568} for bucket 'GleamBookUsers0' 2022-02-08 00:08:48,003 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:48,005 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:48,005 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:48,290 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:49,285 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:49,767 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1548575 == 0 2022-02-08 00:08:49,767 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1548575 == 0. Received: {'172.23.120.170': 1548575} for bucket 'GleamBookUsers0' 2022-02-08 00:08:49,767 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:49,769 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:49,770 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:50,190 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:50,724 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1071612 == 0 2022-02-08 00:08:50,724 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1071612 == 0. Received: {'172.23.121.127': 1071612} for bucket 'GleamBookUsers0' 2022-02-08 00:08:50,724 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:50,726 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:50,727 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:51,572 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1047012 == 0 2022-02-08 00:08:51,572 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1047012 == 0. Received: {'172.23.121.124': 1047012} for bucket 'GleamBookUsers0' 2022-02-08 00:08:51,572 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:51,573 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:51,573 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:51,657 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:52,039 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:52,500 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:52,839 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:53,006 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:53,086 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1267483 == 0 2022-02-08 00:08:53,086 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1267483 == 0. Received: {'172.23.121.126': 1267483} for bucket 'GleamBookUsers0' 2022-02-08 00:08:53,088 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:53,089 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:53,089 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:53,408 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1252589 == 0 2022-02-08 00:08:53,410 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1252589 == 0. Received: {'172.23.121.129': 1252589} for bucket 'GleamBookUsers0' 2022-02-08 00:08:53,410 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:53,411 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:53,411 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:54,490 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1407405 == 0 2022-02-08 00:08:54,490 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1407405 == 0. Received: {'172.23.121.123': 1407405} for bucket 'GleamBookUsers0' 2022-02-08 00:08:54,490 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:54,493 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:54,493 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:54,611 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1253682 == 0 2022-02-08 00:08:54,612 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1253682 == 0. Received: {'172.23.121.128': 1253682} for bucket 'GleamBookUsers0' 2022-02-08 00:08:54,612 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:54,615 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:54,615 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:54,772 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:54,789 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1495272 == 0 2022-02-08 00:08:54,789 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1495272 == 0. Received: {'172.23.121.115': 1495272} for bucket 'GleamBookUsers0' 2022-02-08 00:08:54,790 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:54,792 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:54,792 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:55,727 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:56,197 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1520863 == 0 2022-02-08 00:08:56,198 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1520863 == 0. Received: {'172.23.120.170': 1520863} for bucket 'GleamBookUsers0' 2022-02-08 00:08:56,198 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:56,200 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:56,200 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:56,575 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:57,079 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1037518 == 0 2022-02-08 00:08:57,079 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1037518 == 0. Received: {'172.23.121.127': 1037518} for bucket 'GleamBookUsers0' 2022-02-08 00:08:57,079 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:57,082 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:57,082 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:57,966 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1047012 == 0 2022-02-08 00:08:57,966 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1047012 == 0. Received: {'172.23.121.124': 1047012} for bucket 'GleamBookUsers0' 2022-02-08 00:08:57,966 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:57,969 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:57,969 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:58,089 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:58,411 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:59,494 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:59,615 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:59,660 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1267483 == 0 2022-02-08 00:08:59,660 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1267483 == 0. Received: {'172.23.121.126': 1267483} for bucket 'GleamBookUsers0' 2022-02-08 00:08:59,660 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:59,661 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:59,663 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:08:59,792 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:08:59,836 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1242477 == 0 2022-02-08 00:08:59,836 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1242477 == 0. Received: {'172.23.121.129': 1242477} for bucket 'GleamBookUsers0' 2022-02-08 00:08:59,838 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:08:59,838 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:08:59,839 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:01,200 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:01,388 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1253682 == 0 2022-02-08 00:09:01,388 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1253682 == 0. Received: {'172.23.121.128': 1253682} for bucket 'GleamBookUsers0' 2022-02-08 00:09:01,388 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:01,391 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:01,391 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:01,509 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1345188 == 0 2022-02-08 00:09:01,509 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1345188 == 0. Received: {'172.23.121.123': 1345188} for bucket 'GleamBookUsers0' 2022-02-08 00:09:01,509 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:01,510 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:01,512 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:01,582 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1463912 == 0 2022-02-08 00:09:01,584 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1463912 == 0. Received: {'172.23.121.115': 1463912} for bucket 'GleamBookUsers0' 2022-02-08 00:09:01,584 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:01,585 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:01,585 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:02,082 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:02,641 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1520863 == 0 2022-02-08 00:09:02,641 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1520863 == 0. Received: {'172.23.120.170': 1520863} for bucket 'GleamBookUsers0' 2022-02-08 00:09:02,641 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:02,644 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:02,644 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:02,970 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:03,533 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1029543 == 0 2022-02-08 00:09:03,535 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1029543 == 0. Received: {'172.23.121.127': 1029543} for bucket 'GleamBookUsers0' 2022-02-08 00:09:03,535 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:03,536 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:03,536 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:04,321 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 1001226 == 0 2022-02-08 00:09:04,322 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1001226 == 0. Received: {'172.23.121.124': 1001226} for bucket 'GleamBookUsers0' 2022-02-08 00:09:04,322 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:04,323 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:04,323 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:04,663 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:04,839 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:06,226 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1242477 == 0 2022-02-08 00:09:06,227 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1242477 == 0. Received: {'172.23.121.129': 1242477} for bucket 'GleamBookUsers0' 2022-02-08 00:09:06,227 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:06,230 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:06,230 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:06,250 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1223943 == 0 2022-02-08 00:09:06,252 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1223943 == 0. Received: {'172.23.121.126': 1223943} for bucket 'GleamBookUsers0' 2022-02-08 00:09:06,252 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:06,253 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:06,253 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:06,391 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:06,512 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:06,585 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:07,644 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:07,845 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1215046 == 0 2022-02-08 00:09:07,845 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1215046 == 0. Received: {'172.23.121.128': 1215046} for bucket 'GleamBookUsers0' 2022-02-08 00:09:07,845 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:07,848 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:07,848 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:08,266 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1330070 == 0 2022-02-08 00:09:08,267 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1330070 == 0. Received: {'172.23.121.123': 1330070} for bucket 'GleamBookUsers0' 2022-02-08 00:09:08,267 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:08,269 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:08,269 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:08,272 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1463912 == 0 2022-02-08 00:09:08,272 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1463912 == 0. Received: {'172.23.121.115': 1463912} for bucket 'GleamBookUsers0' 2022-02-08 00:09:08,272 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:08,273 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:08,273 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:08,536 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:09,051 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1492653 == 0 2022-02-08 00:09:09,051 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1492653 == 0. Received: {'172.23.120.170': 1492653} for bucket 'GleamBookUsers0' 2022-02-08 00:09:09,051 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:09,052 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:09,052 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:09,325 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:09,996 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1015239 == 0 2022-02-08 00:09:09,996 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1015239 == 0. Received: {'172.23.121.127': 1015239} for bucket 'GleamBookUsers0' 2022-02-08 00:09:09,996 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:09,999 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:09,999 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:10,671 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 986312 == 0 2022-02-08 00:09:10,671 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 986312 == 0. Received: {'172.23.121.124': 986312} for bucket 'GleamBookUsers0' 2022-02-08 00:09:10,671 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:10,674 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:10,674 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:11,230 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:11,255 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:12,680 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1223833 == 0 2022-02-08 00:09:12,680 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1223833 == 0. Received: {'172.23.121.129': 1223833} for bucket 'GleamBookUsers0' 2022-02-08 00:09:12,680 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:12,683 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:12,683 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:12,835 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1198999 == 0 2022-02-08 00:09:12,836 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1198999 == 0. Received: {'172.23.121.126': 1198999} for bucket 'GleamBookUsers0' 2022-02-08 00:09:12,836 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:12,838 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:12,838 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:12,848 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:13,270 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:13,275 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:14,052 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:14,345 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1215046 == 0 2022-02-08 00:09:14,345 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1215046 == 0. Received: {'172.23.121.128': 1215046} for bucket 'GleamBookUsers0' 2022-02-08 00:09:14,345 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:14,348 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:14,348 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:14,989 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1415564 == 0 2022-02-08 00:09:14,990 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1415564 == 0. Received: {'172.23.121.115': 1415564} for bucket 'GleamBookUsers0' 2022-02-08 00:09:14,990 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:14,992 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:14,992 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:14,999 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:15,003 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1311346 == 0 2022-02-08 00:09:15,003 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1311346 == 0. Received: {'172.23.121.123': 1311346} for bucket 'GleamBookUsers0' 2022-02-08 00:09:15,003 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:15,006 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:15,006 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:15,494 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1479041 == 0 2022-02-08 00:09:15,496 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1479041 == 0. Received: {'172.23.120.170': 1479041} for bucket 'GleamBookUsers0' 2022-02-08 00:09:15,496 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:15,497 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:15,497 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:15,674 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:16,428 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 1004819 == 0 2022-02-08 00:09:16,430 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1004819 == 0. Received: {'172.23.121.127': 1004819} for bucket 'GleamBookUsers0' 2022-02-08 00:09:16,430 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:16,431 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:16,431 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:17,006 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 961770 == 0 2022-02-08 00:09:17,007 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 961770 == 0. Received: {'172.23.121.124': 961770} for bucket 'GleamBookUsers0' 2022-02-08 00:09:17,007 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:17,009 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:17,009 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:17,683 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:17,838 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:19,052 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1223833 == 0 2022-02-08 00:09:19,052 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1223833 == 0. Received: {'172.23.121.129': 1223833} for bucket 'GleamBookUsers0' 2022-02-08 00:09:19,052 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:19,055 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:19,055 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:19,309 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1158109 == 0 2022-02-08 00:09:19,309 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1158109 == 0. Received: {'172.23.121.126': 1158109} for bucket 'GleamBookUsers0' 2022-02-08 00:09:19,309 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:19,312 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:19,312 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:19,349 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:19,993 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:20,006 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:20,499 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:20,756 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1189246 == 0 2022-02-08 00:09:20,756 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1189246 == 0. Received: {'172.23.121.128': 1189246} for bucket 'GleamBookUsers0' 2022-02-08 00:09:20,756 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:20,757 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:20,757 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:21,433 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:21,796 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1406032 == 0 2022-02-08 00:09:21,798 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1406032 == 0. Received: {'172.23.121.115': 1406032} for bucket 'GleamBookUsers0' 2022-02-08 00:09:21,798 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:21,799 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:21,799 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:21,802 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1265598 == 0 2022-02-08 00:09:21,802 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1265598 == 0. Received: {'172.23.121.123': 1265598} for bucket 'GleamBookUsers0' 2022-02-08 00:09:21,802 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:21,805 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:21,805 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:22,009 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:22,049 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1479041 == 0 2022-02-08 00:09:22,051 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1479041 == 0. Received: {'172.23.120.170': 1479041} for bucket 'GleamBookUsers0' 2022-02-08 00:09:22,051 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:22,052 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:22,052 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:22,779 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 974487 == 0 2022-02-08 00:09:22,780 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 974487 == 0. Received: {'172.23.121.127': 974487} for bucket 'GleamBookUsers0' 2022-02-08 00:09:22,780 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:22,782 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:22,782 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:23,341 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 961770 == 0 2022-02-08 00:09:23,342 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 961770 == 0. Received: {'172.23.121.124': 961770} for bucket 'GleamBookUsers0' 2022-02-08 00:09:23,342 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:23,344 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:23,344 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:24,055 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:24,312 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:25,391 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1199463 == 0 2022-02-08 00:09:25,391 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1199463 == 0. Received: {'172.23.121.129': 1199463} for bucket 'GleamBookUsers0' 2022-02-08 00:09:25,391 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:25,394 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:25,394 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:25,759 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:25,792 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1142835 == 0 2022-02-08 00:09:25,793 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1142835 == 0. Received: {'172.23.121.126': 1142835} for bucket 'GleamBookUsers0' 2022-02-08 00:09:25,793 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:25,796 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:25,796 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:26,801 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:26,805 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:27,053 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:27,176 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1183282 == 0 2022-02-08 00:09:27,176 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1183282 == 0. Received: {'172.23.121.128': 1183282} for bucket 'GleamBookUsers0' 2022-02-08 00:09:27,176 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:27,178 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:27,178 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:27,782 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:28,344 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:28,546 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1395520 == 0 2022-02-08 00:09:28,546 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1395520 == 0. Received: {'172.23.121.115': 1395520} for bucket 'GleamBookUsers0' 2022-02-08 00:09:28,546 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:28,549 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:28,549 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:28,641 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1249620 == 0 2022-02-08 00:09:28,641 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1249620 == 0. Received: {'172.23.121.123': 1249620} for bucket 'GleamBookUsers0' 2022-02-08 00:09:28,641 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:28,644 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:28,644 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:28,836 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1453545 == 0 2022-02-08 00:09:28,836 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1453545 == 0. Received: {'172.23.120.170': 1453545} for bucket 'GleamBookUsers0' 2022-02-08 00:09:28,836 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:28,839 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:28,839 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:29,118 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 961081 == 0 2022-02-08 00:09:29,119 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 961081 == 0. Received: {'172.23.121.127': 961081} for bucket 'GleamBookUsers0' 2022-02-08 00:09:29,119 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:29,121 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:29,122 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:29,680 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 933997 == 0 2022-02-08 00:09:29,680 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 933997 == 0. Received: {'172.23.121.124': 933997} for bucket 'GleamBookUsers0' 2022-02-08 00:09:29,680 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:29,683 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:29,683 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:30,394 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:30,796 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:31,687 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1173733 == 0 2022-02-08 00:09:31,687 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1173733 == 0. Received: {'172.23.121.129': 1173733} for bucket 'GleamBookUsers0' 2022-02-08 00:09:31,687 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:31,688 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:31,690 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:32,178 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:32,249 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1118229 == 0 2022-02-08 00:09:32,250 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1118229 == 0. Received: {'172.23.121.126': 1118229} for bucket 'GleamBookUsers0' 2022-02-08 00:09:32,250 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:32,252 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:32,253 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:33,551 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:33,562 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1183282 == 0 2022-02-08 00:09:33,563 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1183282 == 0. Received: {'172.23.121.128': 1183282} for bucket 'GleamBookUsers0' 2022-02-08 00:09:33,563 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:33,565 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:33,565 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:33,644 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:33,839 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:34,122 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:34,683 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:35,259 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1395520 == 0 2022-02-08 00:09:35,259 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1395520 == 0. Received: {'172.23.121.115': 1395520} for bucket 'GleamBookUsers0' 2022-02-08 00:09:35,259 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:35,263 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:35,263 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:35,543 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1435147 == 0 2022-02-08 00:09:35,543 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1435147 == 0. Received: {'172.23.120.170': 1435147} for bucket 'GleamBookUsers0' 2022-02-08 00:09:35,543 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1249620 == 0 2022-02-08 00:09:35,543 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:35,543 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1249620 == 0. Received: {'172.23.121.123': 1249620} for bucket 'GleamBookUsers0' 2022-02-08 00:09:35,543 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:35,551 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:35,551 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:35,551 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:35,552 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:35,736 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 951763 == 0 2022-02-08 00:09:35,736 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 951763 == 0. Received: {'172.23.121.127': 951763} for bucket 'GleamBookUsers0' 2022-02-08 00:09:35,736 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:35,739 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:35,740 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:35,963 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 922553 == 0 2022-02-08 00:09:35,964 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 922553 == 0. Received: {'172.23.121.124': 922553} for bucket 'GleamBookUsers0' 2022-02-08 00:09:35,964 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:35,967 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:35,967 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:36,690 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:37,257 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:37,980 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1154923 == 0 2022-02-08 00:09:37,980 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1154923 == 0. Received: {'172.23.121.129': 1154923} for bucket 'GleamBookUsers0' 2022-02-08 00:09:37,980 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:37,982 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:37,983 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:38,565 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:38,693 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1109787 == 0 2022-02-08 00:09:38,693 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1109787 == 0. Received: {'172.23.121.126': 1109787} for bucket 'GleamBookUsers0' 2022-02-08 00:09:38,694 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:38,696 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:38,696 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:39,944 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1151792 == 0 2022-02-08 00:09:39,944 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1151792 == 0. Received: {'172.23.121.128': 1151792} for bucket 'GleamBookUsers0' 2022-02-08 00:09:39,944 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:39,947 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:39,947 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:40,263 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:40,551 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:40,552 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:40,740 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:40,967 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:41,831 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1355900 == 0 2022-02-08 00:09:41,831 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1355900 == 0. Received: {'172.23.121.115': 1355900} for bucket 'GleamBookUsers0' 2022-02-08 00:09:41,831 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:41,834 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:41,834 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:42,308 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1409633 == 0 2022-02-08 00:09:42,308 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1409633 == 0. Received: {'172.23.120.170': 1409633} for bucket 'GleamBookUsers0' 2022-02-08 00:09:42,309 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:42,311 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:42,312 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:42,487 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1210962 == 0 2022-02-08 00:09:42,487 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1210962 == 0. Received: {'172.23.121.123': 1210962} for bucket 'GleamBookUsers0' 2022-02-08 00:09:42,489 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:42,490 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:42,490 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:42,492 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 937017 == 0 2022-02-08 00:09:42,492 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 937017 == 0. Received: {'172.23.121.127': 937017} for bucket 'GleamBookUsers0' 2022-02-08 00:09:42,492 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:42,493 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:42,493 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:42,522 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 912063 == 0 2022-02-08 00:09:42,522 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 912063 == 0. Received: {'172.23.121.124': 912063} for bucket 'GleamBookUsers0' 2022-02-08 00:09:42,523 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:42,523 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:42,525 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:42,983 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:43,696 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:44,269 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1134608 == 0 2022-02-08 00:09:44,269 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1134608 == 0. Received: {'172.23.121.129': 1134608} for bucket 'GleamBookUsers0' 2022-02-08 00:09:44,269 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:44,272 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:44,272 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:44,947 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:45,094 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1086921 == 0 2022-02-08 00:09:45,095 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1086921 == 0. Received: {'172.23.121.126': 1086921} for bucket 'GleamBookUsers0' 2022-02-08 00:09:45,095 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:45,096 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:45,096 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:46,344 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1126008 == 0 2022-02-08 00:09:46,344 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1126008 == 0. Received: {'172.23.121.128': 1126008} for bucket 'GleamBookUsers0' 2022-02-08 00:09:46,344 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:46,346 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:46,346 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:46,835 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:47,312 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:47,492 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:47,494 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:47,525 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:48,328 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1348867 == 0 2022-02-08 00:09:48,329 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1348867 == 0. Received: {'172.23.121.115': 1348867} for bucket 'GleamBookUsers0' 2022-02-08 00:09:48,329 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:48,332 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:48,332 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:48,971 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1409633 == 0 2022-02-08 00:09:48,973 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1409633 == 0. Received: {'172.23.120.170': 1409633} for bucket 'GleamBookUsers0' 2022-02-08 00:09:48,973 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:48,976 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:48,976 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:49,039 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 868555 == 0 2022-02-08 00:09:49,039 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 868555 == 0. Received: {'172.23.121.124': 868555} for bucket 'GleamBookUsers0' 2022-02-08 00:09:49,039 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:49,042 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:49,042 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:49,154 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 928987 == 0 2022-02-08 00:09:49,154 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 928987 == 0. Received: {'172.23.121.127': 928987} for bucket 'GleamBookUsers0' 2022-02-08 00:09:49,154 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:49,157 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:49,157 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:49,253 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1195432 == 0 2022-02-08 00:09:49,253 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1195432 == 0. Received: {'172.23.121.123': 1195432} for bucket 'GleamBookUsers0' 2022-02-08 00:09:49,253 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:49,256 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:49,256 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:49,273 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:50,096 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:51,348 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:51,460 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1083281 == 0 2022-02-08 00:09:51,461 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1083281 == 0. Received: {'172.23.121.129': 1083281} for bucket 'GleamBookUsers0' 2022-02-08 00:09:51,461 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:51,463 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:51,463 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:51,671 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1061481 == 0 2022-02-08 00:09:51,671 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1061481 == 0. Received: {'172.23.121.126': 1061481} for bucket 'GleamBookUsers0' 2022-02-08 00:09:51,671 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:51,674 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:51,674 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:52,707 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1102844 == 0 2022-02-08 00:09:52,709 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1102844 == 0. Received: {'172.23.121.128': 1102844} for bucket 'GleamBookUsers0' 2022-02-08 00:09:52,709 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:52,710 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:52,710 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:53,332 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:53,976 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:54,043 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:54,157 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:54,256 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:54,841 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1339559 == 0 2022-02-08 00:09:54,841 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1339559 == 0. Received: {'172.23.121.115': 1339559} for bucket 'GleamBookUsers0' 2022-02-08 00:09:54,841 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:54,842 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:54,842 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:55,364 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 852359 == 0 2022-02-08 00:09:55,365 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 852359 == 0. Received: {'172.23.121.124': 852359} for bucket 'GleamBookUsers0' 2022-02-08 00:09:55,365 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:55,367 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:55,367 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:55,730 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1390457 == 0 2022-02-08 00:09:55,730 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1390457 == 0. Received: {'172.23.120.170': 1390457} for bucket 'GleamBookUsers0' 2022-02-08 00:09:55,730 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:55,733 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:55,733 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:55,743 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 909203 == 0 2022-02-08 00:09:55,743 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 909203 == 0. Received: {'172.23.121.127': 909203} for bucket 'GleamBookUsers0' 2022-02-08 00:09:55,743 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:55,746 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:55,746 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:55,921 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1161898 == 0 2022-02-08 00:09:55,921 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1161898 == 0. Received: {'172.23.121.123': 1161898} for bucket 'GleamBookUsers0' 2022-02-08 00:09:55,921 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:55,924 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:55,924 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:56,464 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:56,674 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:57,710 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:09:57,726 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1083281 == 0 2022-02-08 00:09:57,726 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1083281 == 0. Received: {'172.23.121.129': 1083281} for bucket 'GleamBookUsers0' 2022-02-08 00:09:57,727 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:57,730 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:57,730 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:58,075 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1038903 == 0 2022-02-08 00:09:58,076 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1038903 == 0. Received: {'172.23.121.126': 1038903} for bucket 'GleamBookUsers0' 2022-02-08 00:09:58,076 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:58,078 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:58,078 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:58,967 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1083044 == 0 2022-02-08 00:09:58,967 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1083044 == 0. Received: {'172.23.121.128': 1083044} for bucket 'GleamBookUsers0' 2022-02-08 00:09:58,967 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:09:58,969 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:09:58,970 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:09:59,844 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:00,368 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:00,733 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:00,746 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:00,924 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:01,430 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1339559 == 0 2022-02-08 00:10:01,431 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1339559 == 0. Received: {'172.23.121.115': 1339559} for bucket 'GleamBookUsers0' 2022-02-08 00:10:01,431 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:01,434 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:01,434 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:01,621 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 831273 == 0 2022-02-08 00:10:01,621 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 831273 == 0. Received: {'172.23.121.124': 831273} for bucket 'GleamBookUsers0' 2022-02-08 00:10:01,621 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:01,624 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:01,624 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:02,186 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 863803 == 0 2022-02-08 00:10:02,187 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 863803 == 0. Received: {'172.23.121.127': 863803} for bucket 'GleamBookUsers0' 2022-02-08 00:10:02,187 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:02,188 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:02,190 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:02,269 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1352129 == 0 2022-02-08 00:10:02,269 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1352129 == 0. Received: {'172.23.120.170': 1352129} for bucket 'GleamBookUsers0' 2022-02-08 00:10:02,269 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:02,272 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:02,273 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:02,434 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1110908 == 0 2022-02-08 00:10:02,434 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1110908 == 0. Received: {'172.23.121.123': 1110908} for bucket 'GleamBookUsers0' 2022-02-08 00:10:02,434 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:02,436 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:02,436 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:02,730 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:03,078 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:03,970 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:03,980 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1068993 == 0 2022-02-08 00:10:03,980 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1068993 == 0. Received: {'172.23.121.129': 1068993} for bucket 'GleamBookUsers0' 2022-02-08 00:10:03,980 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:03,983 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:03,983 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:04,538 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 1022889 == 0 2022-02-08 00:10:04,538 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1022889 == 0. Received: {'172.23.121.126': 1022889} for bucket 'GleamBookUsers0' 2022-02-08 00:10:04,538 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:04,540 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:04,540 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:05,207 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1067052 == 0 2022-02-08 00:10:05,209 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1067052 == 0. Received: {'172.23.121.128': 1067052} for bucket 'GleamBookUsers0' 2022-02-08 00:10:05,209 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:05,210 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:05,210 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:06,434 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:06,625 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:07,190 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:07,273 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:07,437 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:07,848 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 822287 == 0 2022-02-08 00:10:07,848 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 822287 == 0. Received: {'172.23.121.124': 822287} for bucket 'GleamBookUsers0' 2022-02-08 00:10:07,848 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:07,851 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:07,851 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:07,960 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1293571 == 0 2022-02-08 00:10:07,960 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1293571 == 0. Received: {'172.23.121.115': 1293571} for bucket 'GleamBookUsers0' 2022-02-08 00:10:07,960 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:07,961 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:07,961 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:08,542 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 863803 == 0 2022-02-08 00:10:08,542 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 863803 == 0. Received: {'172.23.121.127': 863803} for bucket 'GleamBookUsers0' 2022-02-08 00:10:08,542 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:08,546 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:08,546 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:08,816 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1343459 == 0 2022-02-08 00:10:08,818 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1343459 == 0. Received: {'172.23.120.170': 1343459} for bucket 'GleamBookUsers0' 2022-02-08 00:10:08,818 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:08,819 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:08,819 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:08,924 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1110908 == 0 2022-02-08 00:10:08,924 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1110908 == 0. Received: {'172.23.121.123': 1110908} for bucket 'GleamBookUsers0' 2022-02-08 00:10:08,924 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:08,927 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:08,927 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:08,983 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:09,540 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:10,211 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:10,230 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1057361 == 0 2022-02-08 00:10:10,232 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1057361 == 0. Received: {'172.23.121.129': 1057361} for bucket 'GleamBookUsers0' 2022-02-08 00:10:10,232 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:10,233 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:10,233 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:10,901 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 975767 == 0 2022-02-08 00:10:10,901 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 975767 == 0. Received: {'172.23.121.126': 975767} for bucket 'GleamBookUsers0' 2022-02-08 00:10:10,901 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:10,904 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:10,904 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:11,447 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1049710 == 0 2022-02-08 00:10:11,447 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1049710 == 0. Received: {'172.23.121.128': 1049710} for bucket 'GleamBookUsers0' 2022-02-08 00:10:11,447 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:11,450 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:11,450 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:12,851 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:12,963 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:13,546 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:13,819 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:13,927 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:13,957 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 794971 == 0 2022-02-08 00:10:13,957 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 794971 == 0. Received: {'172.23.121.124': 794971} for bucket 'GleamBookUsers0' 2022-02-08 00:10:13,957 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:13,960 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:13,960 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:14,398 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1268265 == 0 2022-02-08 00:10:14,398 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1268265 == 0. Received: {'172.23.121.115': 1268265} for bucket 'GleamBookUsers0' 2022-02-08 00:10:14,398 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:14,400 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:14,400 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:14,858 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 840947 == 0 2022-02-08 00:10:14,858 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 840947 == 0. Received: {'172.23.121.127': 840947} for bucket 'GleamBookUsers0' 2022-02-08 00:10:14,858 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:14,861 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:14,861 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:15,236 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:15,244 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1338395 == 0 2022-02-08 00:10:15,244 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1338395 == 0. Received: {'172.23.120.170': 1338395} for bucket 'GleamBookUsers0' 2022-02-08 00:10:15,244 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:15,247 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:15,247 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:15,269 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1076200 == 0 2022-02-08 00:10:15,270 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1076200 == 0. Received: {'172.23.121.123': 1076200} for bucket 'GleamBookUsers0' 2022-02-08 00:10:15,270 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:15,273 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:15,273 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:15,904 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:16,447 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1030619 == 0 2022-02-08 00:10:16,447 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1030619 == 0. Received: {'172.23.121.129': 1030619} for bucket 'GleamBookUsers0' 2022-02-08 00:10:16,447 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:16,450 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:16,450 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:16,450 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:17,250 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 953207 == 0 2022-02-08 00:10:17,250 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 953207 == 0. Received: {'172.23.121.126': 953207} for bucket 'GleamBookUsers0' 2022-02-08 00:10:17,252 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:17,253 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:17,253 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:17,739 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 1015832 == 0 2022-02-08 00:10:17,739 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1015832 == 0. Received: {'172.23.121.128': 1015832} for bucket 'GleamBookUsers0' 2022-02-08 00:10:17,739 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:17,742 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:17,742 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:18,960 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:19,401 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:19,862 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:20,035 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 747177 == 0 2022-02-08 00:10:20,035 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 747177 == 0. Received: {'172.23.121.124': 747177} for bucket 'GleamBookUsers0' 2022-02-08 00:10:20,035 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:20,036 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:20,036 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:20,249 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:20,273 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:20,854 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1240579 == 0 2022-02-08 00:10:20,855 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1240579 == 0. Received: {'172.23.121.115': 1240579} for bucket 'GleamBookUsers0' 2022-02-08 00:10:20,855 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:20,857 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:20,857 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:21,144 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 806913 == 0 2022-02-08 00:10:21,144 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 806913 == 0. Received: {'172.23.121.127': 806913} for bucket 'GleamBookUsers0' 2022-02-08 00:10:21,144 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:21,145 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:21,145 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:21,450 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:21,615 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1065024 == 0 2022-02-08 00:10:21,617 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1065024 == 0. Received: {'172.23.121.123': 1065024} for bucket 'GleamBookUsers0' 2022-02-08 00:10:21,617 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:21,618 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:21,618 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:21,743 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1284964 == 0 2022-02-08 00:10:21,743 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1284964 == 0. Received: {'172.23.120.170': 1284964} for bucket 'GleamBookUsers0' 2022-02-08 00:10:21,743 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:21,746 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:21,746 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:22,253 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:22,648 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1020565 == 0 2022-02-08 00:10:22,648 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1020565 == 0. Received: {'172.23.121.129': 1020565} for bucket 'GleamBookUsers0' 2022-02-08 00:10:22,648 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:22,651 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:22,651 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:22,742 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:23,634 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 931263 == 0 2022-02-08 00:10:23,634 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 931263 == 0. Received: {'172.23.121.126': 931263} for bucket 'GleamBookUsers0' 2022-02-08 00:10:23,634 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:23,638 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:23,638 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:23,894 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 975286 == 0 2022-02-08 00:10:23,894 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 975286 == 0. Received: {'172.23.121.128': 975286} for bucket 'GleamBookUsers0' 2022-02-08 00:10:23,894 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:23,895 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:23,895 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:25,038 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:25,857 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:26,112 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 738457 == 0 2022-02-08 00:10:26,114 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 738457 == 0. Received: {'172.23.121.124': 738457} for bucket 'GleamBookUsers0' 2022-02-08 00:10:26,114 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:26,115 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:26,115 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:26,147 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:26,621 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:26,746 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:27,342 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1214897 == 0 2022-02-08 00:10:27,342 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1214897 == 0. Received: {'172.23.121.115': 1214897} for bucket 'GleamBookUsers0' 2022-02-08 00:10:27,342 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:27,344 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:27,345 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:27,517 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 806913 == 0 2022-02-08 00:10:27,517 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 806913 == 0. Received: {'172.23.121.127': 806913} for bucket 'GleamBookUsers0' 2022-02-08 00:10:27,517 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:27,519 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:27,520 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:27,651 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:27,934 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1040544 == 0 2022-02-08 00:10:27,936 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1040544 == 0. Received: {'172.23.121.123': 1040544} for bucket 'GleamBookUsers0' 2022-02-08 00:10:27,936 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:27,937 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:27,937 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:28,119 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1272038 == 0 2022-02-08 00:10:28,119 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1272038 == 0. Received: {'172.23.120.170': 1272038} for bucket 'GleamBookUsers0' 2022-02-08 00:10:28,119 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:28,121 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:28,121 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:28,638 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:28,842 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 1016035 == 0 2022-02-08 00:10:28,842 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1016035 == 0. Received: {'172.23.121.129': 1016035} for bucket 'GleamBookUsers0' 2022-02-08 00:10:28,842 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:28,844 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:28,844 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:28,897 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:30,013 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 910441 == 0 2022-02-08 00:10:30,013 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 910441 == 0. Received: {'172.23.121.126': 910441} for bucket 'GleamBookUsers0' 2022-02-08 00:10:30,013 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:30,016 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:30,016 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:30,062 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 975286 == 0 2022-02-08 00:10:30,062 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 975286 == 0. Received: {'172.23.121.128': 975286} for bucket 'GleamBookUsers0' 2022-02-08 00:10:30,062 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:30,065 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:30,065 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:31,115 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:32,184 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 720115 == 0 2022-02-08 00:10:32,184 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 720115 == 0. Received: {'172.23.121.124': 720115} for bucket 'GleamBookUsers0' 2022-02-08 00:10:32,184 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:32,187 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:32,187 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:32,345 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:32,522 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:32,938 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:33,122 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:33,844 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:33,914 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 786609 == 0 2022-02-08 00:10:33,914 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 786609 == 0. Received: {'172.23.121.127': 786609} for bucket 'GleamBookUsers0' 2022-02-08 00:10:33,914 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:33,917 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:33,917 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:33,938 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1169489 == 0 2022-02-08 00:10:33,938 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1169489 == 0. Received: {'172.23.121.115': 1169489} for bucket 'GleamBookUsers0' 2022-02-08 00:10:33,938 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:33,940 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:33,940 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:34,198 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 1010204 == 0 2022-02-08 00:10:34,200 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1010204 == 0. Received: {'172.23.121.123': 1010204} for bucket 'GleamBookUsers0' 2022-02-08 00:10:34,200 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:34,201 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:34,201 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:34,532 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1264843 == 0 2022-02-08 00:10:34,533 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1264843 == 0. Received: {'172.23.120.170': 1264843} for bucket 'GleamBookUsers0' 2022-02-08 00:10:34,533 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:34,540 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:34,540 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:35,005 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 995643 == 0 2022-02-08 00:10:35,005 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 995643 == 0. Received: {'172.23.121.129': 995643} for bucket 'GleamBookUsers0' 2022-02-08 00:10:35,005 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:35,006 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:35,006 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:35,016 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:35,065 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:36,309 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 887713 == 0 2022-02-08 00:10:36,309 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 887713 == 0. Received: {'172.23.121.126': 887713} for bucket 'GleamBookUsers0' 2022-02-08 00:10:36,309 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:36,312 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:36,312 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:36,345 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 937646 == 0 2022-02-08 00:10:36,345 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 937646 == 0. Received: {'172.23.121.128': 937646} for bucket 'GleamBookUsers0' 2022-02-08 00:10:36,345 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:36,348 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:36,348 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:37,191 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:38,234 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 712665 == 0 2022-02-08 00:10:38,234 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 712665 == 0. Received: {'172.23.121.124': 712665} for bucket 'GleamBookUsers0' 2022-02-08 00:10:38,234 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:38,236 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:38,237 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:38,917 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:38,941 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:39,201 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:39,546 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:40,007 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:40,266 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 771105 == 0 2022-02-08 00:10:40,266 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 771105 == 0. Received: {'172.23.121.127': 771105} for bucket 'GleamBookUsers0' 2022-02-08 00:10:40,266 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:40,269 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:40,269 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:40,486 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1160891 == 0 2022-02-08 00:10:40,486 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1160891 == 0. Received: {'172.23.121.115': 1160891} for bucket 'GleamBookUsers0' 2022-02-08 00:10:40,486 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:40,489 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:40,489 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:40,589 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 997258 == 0 2022-02-08 00:10:40,589 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 997258 == 0. Received: {'172.23.121.123': 997258} for bucket 'GleamBookUsers0' 2022-02-08 00:10:40,589 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:40,592 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:40,592 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:40,894 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1241111 == 0 2022-02-08 00:10:40,894 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1241111 == 0. Received: {'172.23.120.170': 1241111} for bucket 'GleamBookUsers0' 2022-02-08 00:10:40,894 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:40,897 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:40,897 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:41,180 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 950761 == 0 2022-02-08 00:10:41,181 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 950761 == 0. Received: {'172.23.121.129': 950761} for bucket 'GleamBookUsers0' 2022-02-08 00:10:41,181 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:41,184 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:41,184 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:41,312 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:41,348 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:42,619 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 937646 == 0 2022-02-08 00:10:42,619 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 937646 == 0. Received: {'172.23.121.128': 937646} for bucket 'GleamBookUsers0' 2022-02-08 00:10:42,619 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:42,622 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:42,622 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:42,634 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 866115 == 0 2022-02-08 00:10:42,634 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 866115 == 0. Received: {'172.23.121.126': 866115} for bucket 'GleamBookUsers0' 2022-02-08 00:10:42,634 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:42,637 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:42,637 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:43,237 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:44,293 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 667015 == 0 2022-02-08 00:10:44,295 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 667015 == 0. Received: {'172.23.121.124': 667015} for bucket 'GleamBookUsers0' 2022-02-08 00:10:44,295 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:44,296 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:44,296 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:45,269 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:45,489 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:45,592 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:45,898 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:46,184 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:46,453 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 744802 == 0 2022-02-08 00:10:46,453 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 744802 == 0. Received: {'172.23.121.127': 744802} for bucket 'GleamBookUsers0' 2022-02-08 00:10:46,453 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:46,456 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:46,456 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:46,869 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1160891 == 0 2022-02-08 00:10:46,871 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1160891 == 0. Received: {'172.23.121.115': 1160891} for bucket 'GleamBookUsers0' 2022-02-08 00:10:46,871 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:46,872 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:46,872 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:46,895 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 992811 == 0 2022-02-08 00:10:46,897 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 992811 == 0. Received: {'172.23.121.123': 992811} for bucket 'GleamBookUsers0' 2022-02-08 00:10:46,897 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:46,898 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:46,898 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:47,292 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1223181 == 0 2022-02-08 00:10:47,292 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1223181 == 0. Received: {'172.23.120.170': 1223181} for bucket 'GleamBookUsers0' 2022-02-08 00:10:47,292 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:47,296 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:47,296 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:47,483 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 942123 == 0 2022-02-08 00:10:47,483 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 942123 == 0. Received: {'172.23.121.129': 942123} for bucket 'GleamBookUsers0' 2022-02-08 00:10:47,483 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:47,486 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:47,486 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:47,622 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:47,638 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:48,802 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 904004 == 0 2022-02-08 00:10:48,802 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 904004 == 0. Received: {'172.23.121.128': 904004} for bucket 'GleamBookUsers0' 2022-02-08 00:10:48,802 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:48,805 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:48,805 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:48,905 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 859557 == 0 2022-02-08 00:10:48,905 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 859557 == 0. Received: {'172.23.121.126': 859557} for bucket 'GleamBookUsers0' 2022-02-08 00:10:48,907 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:48,908 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:48,908 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:49,296 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:50,325 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 659327 == 0 2022-02-08 00:10:50,325 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 659327 == 0. Received: {'172.23.121.124': 659327} for bucket 'GleamBookUsers0' 2022-02-08 00:10:50,325 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:50,328 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:50,328 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:51,461 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:51,874 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:51,898 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:52,296 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:52,487 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:52,654 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 737326 == 0 2022-02-08 00:10:52,654 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 737326 == 0. Received: {'172.23.121.127': 737326} for bucket 'GleamBookUsers0' 2022-02-08 00:10:52,654 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:52,657 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:52,657 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:53,124 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 979245 == 0 2022-02-08 00:10:53,124 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 979245 == 0. Received: {'172.23.121.123': 979245} for bucket 'GleamBookUsers0' 2022-02-08 00:10:53,124 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:53,127 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:53,127 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:53,250 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1103886 == 0 2022-02-08 00:10:53,250 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1103886 == 0. Received: {'172.23.121.115': 1103886} for bucket 'GleamBookUsers0' 2022-02-08 00:10:53,250 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:53,253 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:53,253 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:53,766 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1223181 == 0 2022-02-08 00:10:53,767 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1223181 == 0. Received: {'172.23.120.170': 1223181} for bucket 'GleamBookUsers0' 2022-02-08 00:10:53,767 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:53,770 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:53,770 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:53,805 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:53,890 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 914823 == 0 2022-02-08 00:10:53,890 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 914823 == 0. Received: {'172.23.121.129': 914823} for bucket 'GleamBookUsers0' 2022-02-08 00:10:53,890 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:53,891 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:53,891 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:53,908 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:54,917 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 876046 == 0 2022-02-08 00:10:54,917 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 876046 == 0. Received: {'172.23.121.128': 876046} for bucket 'GleamBookUsers0' 2022-02-08 00:10:54,917 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:54,920 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:54,921 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:55,138 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 852191 == 0 2022-02-08 00:10:55,138 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 852191 == 0. Received: {'172.23.121.126': 852191} for bucket 'GleamBookUsers0' 2022-02-08 00:10:55,140 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:55,141 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:55,141 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:55,328 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:56,510 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 642599 == 0 2022-02-08 00:10:56,510 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 642599 == 0. Received: {'172.23.121.124': 642599} for bucket 'GleamBookUsers0' 2022-02-08 00:10:56,510 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:56,513 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:56,513 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:57,657 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:58,127 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:58,255 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:58,770 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:58,785 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 717716 == 0 2022-02-08 00:10:58,786 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 717716 == 0. Received: {'172.23.121.127': 717716} for bucket 'GleamBookUsers0' 2022-02-08 00:10:58,786 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:58,786 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:58,788 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:58,892 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:10:59,427 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 964970 == 0 2022-02-08 00:10:59,427 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 964970 == 0. Received: {'172.23.121.123': 964970} for bucket 'GleamBookUsers0' 2022-02-08 00:10:59,427 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:59,430 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:59,430 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:59,578 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1098268 == 0 2022-02-08 00:10:59,578 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1098268 == 0. Received: {'172.23.121.115': 1098268} for bucket 'GleamBookUsers0' 2022-02-08 00:10:59,578 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:10:59,579 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:10:59,579 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:10:59,921 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:00,036 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 894741 == 0 2022-02-08 00:11:00,036 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 894741 == 0. Received: {'172.23.121.129': 894741} for bucket 'GleamBookUsers0' 2022-02-08 00:11:00,036 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:00,039 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:00,039 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:00,141 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1204527 == 0 2022-02-08 00:11:00,141 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1204527 == 0. Received: {'172.23.120.170': 1204527} for bucket 'GleamBookUsers0' 2022-02-08 00:11:00,141 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:00,144 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:00,144 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:00,144 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:00,969 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 866310 == 0 2022-02-08 00:11:00,969 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 866310 == 0. Received: {'172.23.121.128': 866310} for bucket 'GleamBookUsers0' 2022-02-08 00:11:00,969 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:00,970 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:00,970 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:01,359 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 793254 == 0 2022-02-08 00:11:01,359 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 793254 == 0. Received: {'172.23.121.126': 793254} for bucket 'GleamBookUsers0' 2022-02-08 00:11:01,361 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:01,365 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:01,365 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:01,513 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:02,602 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 635669 == 0 2022-02-08 00:11:02,602 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 635669 == 0. Received: {'172.23.121.124': 635669} for bucket 'GleamBookUsers0' 2022-02-08 00:11:02,602 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:02,605 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:02,605 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:03,788 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:04,430 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:04,581 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:04,911 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 695258 == 0 2022-02-08 00:11:04,911 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 695258 == 0. Received: {'172.23.121.127': 695258} for bucket 'GleamBookUsers0' 2022-02-08 00:11:04,911 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:04,913 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:04,914 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:05,039 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:05,144 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:05,703 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 947732 == 0 2022-02-08 00:11:05,703 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 947732 == 0. Received: {'172.23.121.123': 947732} for bucket 'GleamBookUsers0' 2022-02-08 00:11:05,703 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:05,706 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:05,706 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:05,851 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1072332 == 0 2022-02-08 00:11:05,851 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1072332 == 0. Received: {'172.23.121.115': 1072332} for bucket 'GleamBookUsers0' 2022-02-08 00:11:05,851 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:05,854 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:05,854 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:05,971 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:06,190 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 882625 == 0 2022-02-08 00:11:06,190 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 882625 == 0. Received: {'172.23.121.129': 882625} for bucket 'GleamBookUsers0' 2022-02-08 00:11:06,190 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:06,191 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:06,193 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:06,368 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:06,451 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1130557 == 0 2022-02-08 00:11:06,453 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1130557 == 0. Received: {'172.23.120.170': 1130557} for bucket 'GleamBookUsers0' 2022-02-08 00:11:06,453 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:06,454 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:06,454 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:07,006 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 861310 == 0 2022-02-08 00:11:07,006 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 861310 == 0. Received: {'172.23.121.128': 861310} for bucket 'GleamBookUsers0' 2022-02-08 00:11:07,006 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:07,009 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:07,009 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:07,607 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:07,618 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 780352 == 0 2022-02-08 00:11:07,618 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 780352 == 0. Received: {'172.23.121.126': 780352} for bucket 'GleamBookUsers0' 2022-02-08 00:11:07,618 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:07,619 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:07,619 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:08,667 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 635669 == 0 2022-02-08 00:11:08,667 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 635669 == 0. Received: {'172.23.121.124': 635669} for bucket 'GleamBookUsers0' 2022-02-08 00:11:08,667 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:08,670 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:08,670 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:09,914 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:10,707 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:10,854 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:11,026 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 680571 == 0 2022-02-08 00:11:11,028 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 680571 == 0. Received: {'172.23.121.127': 680571} for bucket 'GleamBookUsers0' 2022-02-08 00:11:11,028 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:11,029 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:11,029 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:11,193 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:11,454 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:11,836 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 916814 == 0 2022-02-08 00:11:11,836 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 916814 == 0. Received: {'172.23.121.123': 916814} for bucket 'GleamBookUsers0' 2022-02-08 00:11:11,836 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:11,838 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:11,839 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:12,009 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:12,118 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1072332 == 0 2022-02-08 00:11:12,118 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1072332 == 0. Received: {'172.23.121.115': 1072332} for bucket 'GleamBookUsers0' 2022-02-08 00:11:12,118 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:12,121 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:12,121 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:12,378 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 865853 == 0 2022-02-08 00:11:12,378 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 865853 == 0. Received: {'172.23.121.129': 865853} for bucket 'GleamBookUsers0' 2022-02-08 00:11:12,378 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:12,381 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:12,381 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:12,621 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:12,785 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1130557 == 0 2022-02-08 00:11:12,785 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1130557 == 0. Received: {'172.23.120.170': 1130557} for bucket 'GleamBookUsers0' 2022-02-08 00:11:12,785 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:12,788 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:12,788 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:13,055 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 833676 == 0 2022-02-08 00:11:13,055 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 833676 == 0. Received: {'172.23.121.128': 833676} for bucket 'GleamBookUsers0' 2022-02-08 00:11:13,055 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:13,059 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:13,059 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:13,673 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:13,789 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 767534 == 0 2022-02-08 00:11:13,789 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 767534 == 0. Received: {'172.23.121.126': 767534} for bucket 'GleamBookUsers0' 2022-02-08 00:11:13,789 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:13,790 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:13,790 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:14,700 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 584683 == 0 2022-02-08 00:11:14,700 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 584683 == 0. Received: {'172.23.121.124': 584683} for bucket 'GleamBookUsers0' 2022-02-08 00:11:14,700 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:14,703 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:14,703 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:16,030 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:16,839 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:17,121 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:17,130 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 660715 == 0 2022-02-08 00:11:17,130 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 660715 == 0. Received: {'172.23.121.127': 660715} for bucket 'GleamBookUsers0' 2022-02-08 00:11:17,130 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:17,131 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:17,131 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:17,381 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:17,788 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:17,907 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 890200 == 0 2022-02-08 00:11:17,907 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 890200 == 0. Received: {'172.23.121.123': 890200} for bucket 'GleamBookUsers0' 2022-02-08 00:11:17,908 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:17,910 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:17,911 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:18,062 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:18,500 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1058842 == 0 2022-02-08 00:11:18,502 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1058842 == 0. Received: {'172.23.121.115': 1058842} for bucket 'GleamBookUsers0' 2022-02-08 00:11:18,502 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:18,506 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:18,506 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:18,632 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 836332 == 0 2022-02-08 00:11:18,634 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 836332 == 0. Received: {'172.23.121.129': 836332} for bucket 'GleamBookUsers0' 2022-02-08 00:11:18,634 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:18,635 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:18,635 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:18,792 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:19,131 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1130557 == 0 2022-02-08 00:11:19,131 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1130557 == 0. Received: {'172.23.120.170': 1130557} for bucket 'GleamBookUsers0' 2022-02-08 00:11:19,131 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:19,134 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:19,134 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:19,142 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 826134 == 0 2022-02-08 00:11:19,142 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 826134 == 0. Received: {'172.23.121.128': 826134} for bucket 'GleamBookUsers0' 2022-02-08 00:11:19,142 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:19,144 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:19,144 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:19,703 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:19,987 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 767534 == 0 2022-02-08 00:11:19,989 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 767534 == 0. Received: {'172.23.121.126': 767534} for bucket 'GleamBookUsers0' 2022-02-08 00:11:19,989 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:19,990 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:19,990 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:20,723 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 568993 == 0 2022-02-08 00:11:20,723 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 568993 == 0. Received: {'172.23.121.124': 568993} for bucket 'GleamBookUsers0' 2022-02-08 00:11:20,723 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:20,724 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:20,726 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:22,132 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:22,911 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:23,220 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 654475 == 0 2022-02-08 00:11:23,221 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 654475 == 0. Received: {'172.23.121.127': 654475} for bucket 'GleamBookUsers0' 2022-02-08 00:11:23,221 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:23,224 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:23,224 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:23,506 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:23,637 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:23,970 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 869210 == 0 2022-02-08 00:11:23,971 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 869210 == 0. Received: {'172.23.121.123': 869210} for bucket 'GleamBookUsers0' 2022-02-08 00:11:23,971 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:23,973 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:23,973 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:24,134 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:24,145 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:24,756 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 803304 == 0 2022-02-08 00:11:24,756 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 803304 == 0. Received: {'172.23.121.129': 803304} for bucket 'GleamBookUsers0' 2022-02-08 00:11:24,756 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:24,759 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:24,759 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:24,990 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:25,187 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 804704 == 0 2022-02-08 00:11:25,187 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 804704 == 0. Received: {'172.23.121.128': 804704} for bucket 'GleamBookUsers0' 2022-02-08 00:11:25,187 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:25,190 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:25,190 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:25,414 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1109627 == 0 2022-02-08 00:11:25,415 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1109627 == 0. Received: {'172.23.120.170': 1109627} for bucket 'GleamBookUsers0' 2022-02-08 00:11:25,415 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:25,417 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:25,417 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:25,726 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:26,150 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 709058 == 0 2022-02-08 00:11:26,150 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 709058 == 0. Received: {'172.23.121.126': 709058} for bucket 'GleamBookUsers0' 2022-02-08 00:11:26,150 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:26,151 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:26,151 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:26,759 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 562167 == 0 2022-02-08 00:11:26,760 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 562167 == 0. Received: {'172.23.121.124': 562167} for bucket 'GleamBookUsers0' 2022-02-08 00:11:26,760 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:26,763 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:26,763 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:28,226 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:28,973 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:29,266 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 632907 == 0 2022-02-08 00:11:29,266 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 632907 == 0. Received: {'172.23.121.127': 632907} for bucket 'GleamBookUsers0' 2022-02-08 00:11:29,266 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:29,269 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:29,269 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:29,760 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:30,059 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 847190 == 0 2022-02-08 00:11:30,059 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 847190 == 0. Received: {'172.23.121.123': 847190} for bucket 'GleamBookUsers0' 2022-02-08 00:11:30,059 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:30,062 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:30,062 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:30,190 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:30,417 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:30,795 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 796154 == 0 2022-02-08 00:11:30,795 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 796154 == 0. Received: {'172.23.121.129': 796154} for bucket 'GleamBookUsers0' 2022-02-08 00:11:30,795 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:30,796 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:30,796 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:31,153 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:31,259 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 796372 == 0 2022-02-08 00:11:31,260 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 796372 == 0. Received: {'172.23.121.128': 796372} for bucket 'GleamBookUsers0' 2022-02-08 00:11:31,260 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:31,263 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:31,263 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:31,696 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1087705 == 0 2022-02-08 00:11:31,696 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1087705 == 0. Received: {'172.23.120.170': 1087705} for bucket 'GleamBookUsers0' 2022-02-08 00:11:31,696 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:31,697 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:31,697 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:31,763 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:32,305 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 709058 == 0 2022-02-08 00:11:32,338 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 709058 == 0. Received: {'172.23.121.126': 709058} for bucket 'GleamBookUsers0' 2022-02-08 00:11:32,338 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:32,339 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:32,341 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:32,743 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 562167 == 0 2022-02-08 00:11:32,743 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 562167 == 0. Received: {'172.23.121.124': 562167} for bucket 'GleamBookUsers0' 2022-02-08 00:11:32,743 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:32,746 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:32,746 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:34,269 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:35,062 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:35,312 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 615587 == 0 2022-02-08 00:11:35,313 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 615587 == 0. Received: {'172.23.121.127': 615587} for bucket 'GleamBookUsers0' 2022-02-08 00:11:35,313 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:35,319 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:35,319 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:35,798 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:36,125 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 831522 == 0 2022-02-08 00:11:36,125 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 831522 == 0. Received: {'172.23.121.123': 831522} for bucket 'GleamBookUsers0' 2022-02-08 00:11:36,125 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:36,127 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:36,127 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:36,263 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:36,697 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:36,823 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 784714 == 0 2022-02-08 00:11:36,825 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 784714 == 0. Received: {'172.23.121.129': 784714} for bucket 'GleamBookUsers0' 2022-02-08 00:11:36,825 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:36,826 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:36,826 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:37,230 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 763958 == 0 2022-02-08 00:11:37,230 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 763958 == 0. Received: {'172.23.121.128': 763958} for bucket 'GleamBookUsers0' 2022-02-08 00:11:37,230 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:37,233 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:37,233 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:37,341 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:37,746 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:37,951 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1059130 == 0 2022-02-08 00:11:37,953 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1059130 == 0. Received: {'172.23.120.170': 1059130} for bucket 'GleamBookUsers0' 2022-02-08 00:11:37,953 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:37,954 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:37,954 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:38,539 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 702572 == 0 2022-02-08 00:11:38,540 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 702572 == 0. Received: {'172.23.121.126': 702572} for bucket 'GleamBookUsers0' 2022-02-08 00:11:38,540 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:38,542 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:38,542 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:38,783 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 541777 == 0 2022-02-08 00:11:38,785 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 541777 == 0. Received: {'172.23.121.124': 541777} for bucket 'GleamBookUsers0' 2022-02-08 00:11:38,785 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:38,786 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:38,786 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:40,101 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 991636 == 0 2022-02-08 00:11:40,101 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 991636 == 0. Received: {'172.23.121.115': 991636} for bucket 'GleamBookUsers0' 2022-02-08 00:11:40,102 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:40,104 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:40,104 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:40,322 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:41,128 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:41,355 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 615587 == 0 2022-02-08 00:11:41,355 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 615587 == 0. Received: {'172.23.121.127': 615587} for bucket 'GleamBookUsers0' 2022-02-08 00:11:41,355 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:41,358 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:41,358 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:41,826 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:42,181 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 817654 == 0 2022-02-08 00:11:42,181 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 817654 == 0. Received: {'172.23.121.123': 817654} for bucket 'GleamBookUsers0' 2022-02-08 00:11:42,183 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:42,184 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:42,184 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:42,233 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:42,855 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 767768 == 0 2022-02-08 00:11:42,855 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 767768 == 0. Received: {'172.23.121.129': 767768} for bucket 'GleamBookUsers0' 2022-02-08 00:11:42,855 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:42,858 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:42,858 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:42,954 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:43,240 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 763958 == 0 2022-02-08 00:11:43,242 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 763958 == 0. Received: {'172.23.121.128': 763958} for bucket 'GleamBookUsers0' 2022-02-08 00:11:43,242 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:43,243 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:43,243 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:43,543 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:43,788 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:44,220 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1059130 == 0 2022-02-08 00:11:44,221 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1059130 == 0. Received: {'172.23.120.170': 1059130} for bucket 'GleamBookUsers0' 2022-02-08 00:11:44,221 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:44,223 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:44,223 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:44,792 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 678256 == 0 2022-02-08 00:11:44,792 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 678256 == 0. Received: {'172.23.121.126': 678256} for bucket 'GleamBookUsers0' 2022-02-08 00:11:44,792 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:44,795 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:44,795 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:44,832 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 499707 == 0 2022-02-08 00:11:44,832 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 499707 == 0. Received: {'172.23.121.124': 499707} for bucket 'GleamBookUsers0' 2022-02-08 00:11:44,832 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:44,835 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:44,835 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:45,105 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:46,311 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 953968 == 0 2022-02-08 00:11:46,311 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 953968 == 0. Received: {'172.23.121.115': 953968} for bucket 'GleamBookUsers0' 2022-02-08 00:11:46,311 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:46,312 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:46,312 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:46,358 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:47,184 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:47,355 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 591511 == 0 2022-02-08 00:11:47,357 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 591511 == 0. Received: {'172.23.121.127': 591511} for bucket 'GleamBookUsers0' 2022-02-08 00:11:47,357 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:47,358 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:47,358 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:47,858 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:48,203 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 817654 == 0 2022-02-08 00:11:48,203 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 817654 == 0. Received: {'172.23.121.123': 817654} for bucket 'GleamBookUsers0' 2022-02-08 00:11:48,203 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:48,206 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:48,206 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:48,243 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:48,884 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 734487 == 0 2022-02-08 00:11:48,885 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 734487 == 0. Received: {'172.23.121.129': 734487} for bucket 'GleamBookUsers0' 2022-02-08 00:11:48,885 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:48,888 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:48,888 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:49,216 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 756096 == 0 2022-02-08 00:11:49,216 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 756096 == 0. Received: {'172.23.121.128': 756096} for bucket 'GleamBookUsers0' 2022-02-08 00:11:49,216 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:49,217 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:49,219 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:49,223 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:49,796 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:49,835 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:50,489 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1024652 == 0 2022-02-08 00:11:50,489 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1024652 == 0. Received: {'172.23.120.170': 1024652} for bucket 'GleamBookUsers0' 2022-02-08 00:11:50,490 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:50,492 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:50,492 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:50,832 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 489339 == 0 2022-02-08 00:11:50,834 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 489339 == 0. Received: {'172.23.121.124': 489339} for bucket 'GleamBookUsers0' 2022-02-08 00:11:50,834 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:50,835 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:50,835 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:50,940 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 642216 == 0 2022-02-08 00:11:50,940 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 642216 == 0. Received: {'172.23.121.126': 642216} for bucket 'GleamBookUsers0' 2022-02-08 00:11:50,940 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:50,941 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:50,943 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:51,313 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:52,361 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:52,516 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 947420 == 0 2022-02-08 00:11:52,516 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 947420 == 0. Received: {'172.23.121.115': 947420} for bucket 'GleamBookUsers0' 2022-02-08 00:11:52,516 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:52,520 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:52,520 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:53,206 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:53,368 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 573745 == 0 2022-02-08 00:11:53,368 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 573745 == 0. Received: {'172.23.121.127': 573745} for bucket 'GleamBookUsers0' 2022-02-08 00:11:53,368 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:53,371 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:53,371 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:53,888 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:54,219 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:54,278 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 780538 == 0 2022-02-08 00:11:54,278 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 780538 == 0. Received: {'172.23.121.123': 780538} for bucket 'GleamBookUsers0' 2022-02-08 00:11:54,279 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:54,279 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:54,280 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:54,871 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 704049 == 0 2022-02-08 00:11:54,872 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 704049 == 0. Received: {'172.23.121.129': 704049} for bucket 'GleamBookUsers0' 2022-02-08 00:11:54,872 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:54,875 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:54,875 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:55,160 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 723376 == 0 2022-02-08 00:11:55,160 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 723376 == 0. Received: {'172.23.121.128': 723376} for bucket 'GleamBookUsers0' 2022-02-08 00:11:55,160 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:55,161 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:55,161 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:55,492 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:55,836 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:55,943 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:56,799 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 965564 == 0 2022-02-08 00:11:56,799 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 965564 == 0. Received: {'172.23.120.170': 965564} for bucket 'GleamBookUsers0' 2022-02-08 00:11:56,799 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:56,802 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:56,802 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:56,888 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 475399 == 0 2022-02-08 00:11:56,888 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 475399 == 0. Received: {'172.23.121.124': 475399} for bucket 'GleamBookUsers0' 2022-02-08 00:11:56,888 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:56,891 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:56,891 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:57,036 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 620956 == 0 2022-02-08 00:11:57,036 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 620956 == 0. Received: {'172.23.121.126': 620956} for bucket 'GleamBookUsers0' 2022-02-08 00:11:57,036 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:57,038 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:57,038 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:57,520 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:58,371 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:58,651 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 924314 == 0 2022-02-08 00:11:58,651 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 924314 == 0. Received: {'172.23.121.115': 924314} for bucket 'GleamBookUsers0' 2022-02-08 00:11:58,653 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:58,654 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:58,654 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:59,282 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:11:59,357 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 568077 == 0 2022-02-08 00:11:59,358 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 568077 == 0. Received: {'172.23.121.127': 568077} for bucket 'GleamBookUsers0' 2022-02-08 00:11:59,358 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:11:59,359 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:11:59,359 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:11:59,875 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:00,163 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:00,286 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 763998 == 0 2022-02-08 00:12:00,286 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 763998 == 0. Received: {'172.23.121.123': 763998} for bucket 'GleamBookUsers0' 2022-02-08 00:12:00,286 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:00,289 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:00,289 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:00,868 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 698245 == 0 2022-02-08 00:12:00,868 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 698245 == 0. Received: {'172.23.121.129': 698245} for bucket 'GleamBookUsers0' 2022-02-08 00:12:00,868 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:00,871 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:00,871 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:01,117 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 704667 == 0 2022-02-08 00:12:01,117 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 704667 == 0. Received: {'172.23.121.128': 704667} for bucket 'GleamBookUsers0' 2022-02-08 00:12:01,117 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:01,118 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:01,118 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:01,802 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:01,891 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:02,039 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:02,819 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 458276 == 0 2022-02-08 00:12:02,819 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 458276 == 0. Received: {'172.23.121.124': 458276} for bucket 'GleamBookUsers0' 2022-02-08 00:12:02,821 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:02,822 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:02,822 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:03,130 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 955198 == 0 2022-02-08 00:12:03,130 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 955198 == 0. Received: {'172.23.120.170': 955198} for bucket 'GleamBookUsers0' 2022-02-08 00:12:03,131 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:03,132 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:03,132 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 597994 == 0 2022-02-08 00:12:03,132 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:03,132 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 597994 == 0. Received: {'172.23.121.126': 597994} for bucket 'GleamBookUsers0' 2022-02-08 00:12:03,132 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:03,134 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:03,134 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:03,655 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:04,359 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:04,888 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 920300 == 0 2022-02-08 00:12:04,888 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 920300 == 0. Received: {'172.23.121.115': 920300} for bucket 'GleamBookUsers0' 2022-02-08 00:12:04,890 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:04,891 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:04,891 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:05,290 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:05,348 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 542443 == 0 2022-02-08 00:12:05,348 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 542443 == 0. Received: {'172.23.121.127': 542443} for bucket 'GleamBookUsers0' 2022-02-08 00:12:05,348 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:05,351 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:05,351 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:05,874 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:06,119 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:06,288 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 730915 == 0 2022-02-08 00:12:06,289 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 730915 == 0. Received: {'172.23.121.123': 730915} for bucket 'GleamBookUsers0' 2022-02-08 00:12:06,289 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:06,290 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:06,290 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:06,894 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 698245 == 0 2022-02-08 00:12:06,894 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 698245 == 0. Received: {'172.23.121.129': 698245} for bucket 'GleamBookUsers0' 2022-02-08 00:12:06,894 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:06,895 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:06,895 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:07,092 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 679891 == 0 2022-02-08 00:12:07,092 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 679891 == 0. Received: {'172.23.121.128': 679891} for bucket 'GleamBookUsers0' 2022-02-08 00:12:07,092 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:07,095 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:07,095 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:07,822 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:08,134 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:08,135 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:08,704 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 425026 == 0 2022-02-08 00:12:08,704 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 425026 == 0. Received: {'172.23.121.124': 425026} for bucket 'GleamBookUsers0' 2022-02-08 00:12:08,704 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:08,707 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:08,707 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:09,257 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 576664 == 0 2022-02-08 00:12:09,257 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 576664 == 0. Received: {'172.23.121.126': 576664} for bucket 'GleamBookUsers0' 2022-02-08 00:12:09,257 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:09,260 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:09,260 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:09,450 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 938007 == 0 2022-02-08 00:12:09,450 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 938007 == 0. Received: {'172.23.120.170': 938007} for bucket 'GleamBookUsers0' 2022-02-08 00:12:09,450 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:09,453 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:09,453 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:09,891 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:10,351 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:10,957 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 920300 == 0 2022-02-08 00:12:10,959 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 920300 == 0. Received: {'172.23.121.115': 920300} for bucket 'GleamBookUsers0' 2022-02-08 00:12:10,959 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:10,960 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:10,960 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:11,298 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:11,332 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 537373 == 0 2022-02-08 00:12:11,332 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 537373 == 0. Received: {'172.23.121.127': 537373} for bucket 'GleamBookUsers0' 2022-02-08 00:12:11,332 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:11,335 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:11,335 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:11,897 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:12,096 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:12,272 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 716305 == 0 2022-02-08 00:12:12,272 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 716305 == 0. Received: {'172.23.121.123': 716305} for bucket 'GleamBookUsers0' 2022-02-08 00:12:12,272 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:12,273 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:12,273 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:12,894 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 673517 == 0 2022-02-08 00:12:12,895 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 673517 == 0. Received: {'172.23.121.129': 673517} for bucket 'GleamBookUsers0' 2022-02-08 00:12:12,895 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:12,898 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:12,898 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:13,035 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 655987 == 0 2022-02-08 00:12:13,036 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 655987 == 0. Received: {'172.23.121.128': 655987} for bucket 'GleamBookUsers0' 2022-02-08 00:12:13,036 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:13,038 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:13,038 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:13,707 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:14,260 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:14,453 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:14,601 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 425026 == 0 2022-02-08 00:12:14,601 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 425026 == 0. Received: {'172.23.121.124': 425026} for bucket 'GleamBookUsers0' 2022-02-08 00:12:14,601 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:14,602 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:14,602 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:15,328 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 558980 == 0 2022-02-08 00:12:15,328 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 558980 == 0. Received: {'172.23.121.126': 558980} for bucket 'GleamBookUsers0' 2022-02-08 00:12:15,329 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:15,332 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:15,332 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:15,694 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 929265 == 0 2022-02-08 00:12:15,694 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 929265 == 0. Received: {'172.23.120.170': 929265} for bucket 'GleamBookUsers0' 2022-02-08 00:12:15,694 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:15,696 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:15,697 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:15,960 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:16,335 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:17,012 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 873326 == 0 2022-02-08 00:12:17,012 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 873326 == 0. Received: {'172.23.121.115': 873326} for bucket 'GleamBookUsers0' 2022-02-08 00:12:17,013 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:17,015 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:17,015 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:17,275 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:17,280 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 519279 == 0 2022-02-08 00:12:17,280 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 519279 == 0. Received: {'172.23.121.127': 519279} for bucket 'GleamBookUsers0' 2022-02-08 00:12:17,280 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:17,282 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:17,283 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:17,898 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:18,039 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:18,247 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 713433 == 0 2022-02-08 00:12:18,249 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 713433 == 0. Received: {'172.23.121.123': 713433} for bucket 'GleamBookUsers0' 2022-02-08 00:12:18,249 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:18,250 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:18,250 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:18,947 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 653961 == 0 2022-02-08 00:12:18,948 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 653961 == 0. Received: {'172.23.121.129': 653961} for bucket 'GleamBookUsers0' 2022-02-08 00:12:18,948 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:18,950 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:18,950 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:19,059 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 636159 == 0 2022-02-08 00:12:19,061 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 636159 == 0. Received: {'172.23.121.128': 636159} for bucket 'GleamBookUsers0' 2022-02-08 00:12:19,061 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:19,062 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:19,062 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:19,604 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:20,332 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:20,490 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 411872 == 0 2022-02-08 00:12:20,492 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 411872 == 0. Received: {'172.23.121.124': 411872} for bucket 'GleamBookUsers0' 2022-02-08 00:12:20,492 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:20,493 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:20,493 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:20,697 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:21,334 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 548128 == 0 2022-02-08 00:12:21,334 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 548128 == 0. Received: {'172.23.121.126': 548128} for bucket 'GleamBookUsers0' 2022-02-08 00:12:21,334 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:21,335 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:21,335 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:21,907 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 883837 == 0 2022-02-08 00:12:21,907 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 883837 == 0. Received: {'172.23.120.170': 883837} for bucket 'GleamBookUsers0' 2022-02-08 00:12:21,907 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:21,908 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:21,908 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:22,016 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:22,283 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:23,181 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 833702 == 0 2022-02-08 00:12:23,181 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 833702 == 0. Received: {'172.23.121.115': 833702} for bucket 'GleamBookUsers0' 2022-02-08 00:12:23,183 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:23,184 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:23,184 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:23,250 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:23,257 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 495439 == 0 2022-02-08 00:12:23,257 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 495439 == 0. Received: {'172.23.121.127': 495439} for bucket 'GleamBookUsers0' 2022-02-08 00:12:23,257 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:23,259 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:23,259 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:23,951 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:24,062 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:24,230 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 664014 == 0 2022-02-08 00:12:24,230 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 664014 == 0. Received: {'172.23.121.123': 664014} for bucket 'GleamBookUsers0' 2022-02-08 00:12:24,230 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:24,232 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:24,232 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:24,976 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 636987 == 0 2022-02-08 00:12:24,976 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 636987 == 0. Received: {'172.23.121.129': 636987} for bucket 'GleamBookUsers0' 2022-02-08 00:12:24,976 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:24,980 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:24,980 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:25,009 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 629605 == 0 2022-02-08 00:12:25,010 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 629605 == 0. Received: {'172.23.121.128': 629605} for bucket 'GleamBookUsers0' 2022-02-08 00:12:25,010 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:25,013 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:25,013 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:25,494 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:26,336 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:26,346 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 391278 == 0 2022-02-08 00:12:26,346 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 391278 == 0. Received: {'172.23.121.124': 391278} for bucket 'GleamBookUsers0' 2022-02-08 00:12:26,346 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:26,348 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:26,348 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:26,910 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:27,341 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 548128 == 0 2022-02-08 00:12:27,341 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 548128 == 0. Received: {'172.23.121.126': 548128} for bucket 'GleamBookUsers0' 2022-02-08 00:12:27,341 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:27,342 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:27,342 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:28,061 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 883837 == 0 2022-02-08 00:12:28,062 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 883837 == 0. Received: {'172.23.120.170': 883837} for bucket 'GleamBookUsers0' 2022-02-08 00:12:28,062 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:28,063 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:28,063 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:28,186 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:28,260 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:29,233 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:29,255 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 468847 == 0 2022-02-08 00:12:29,255 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 468847 == 0. Received: {'172.23.121.127': 468847} for bucket 'GleamBookUsers0' 2022-02-08 00:12:29,256 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:29,257 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:29,257 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:29,302 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 829444 == 0 2022-02-08 00:12:29,303 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 829444 == 0. Received: {'172.23.121.115': 829444} for bucket 'GleamBookUsers0' 2022-02-08 00:12:29,303 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:29,305 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:29,305 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:29,980 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:30,013 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:30,197 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 664014 == 0 2022-02-08 00:12:30,198 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 664014 == 0. Received: {'172.23.121.123': 664014} for bucket 'GleamBookUsers0' 2022-02-08 00:12:30,198 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:30,200 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:30,200 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:31,006 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 596040 == 0 2022-02-08 00:12:31,006 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 596040 == 0. Received: {'172.23.121.128': 596040} for bucket 'GleamBookUsers0' 2022-02-08 00:12:31,006 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:31,007 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:31,009 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:31,062 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 633829 == 0 2022-02-08 00:12:31,062 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 633829 == 0. Received: {'172.23.121.129': 633829} for bucket 'GleamBookUsers0' 2022-02-08 00:12:31,062 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:31,063 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:31,065 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:31,349 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:32,176 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 386030 == 0 2022-02-08 00:12:32,177 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 386030 == 0. Received: {'172.23.121.124': 386030} for bucket 'GleamBookUsers0' 2022-02-08 00:12:32,177 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:32,178 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:32,178 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:32,344 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:33,063 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:33,315 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 533098 == 0 2022-02-08 00:12:33,315 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 533098 == 0. Received: {'172.23.121.126': 533098} for bucket 'GleamBookUsers0' 2022-02-08 00:12:33,315 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:33,318 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:33,318 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:34,227 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 857091 == 0 2022-02-08 00:12:34,227 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 857091 == 0. Received: {'172.23.120.170': 857091} for bucket 'GleamBookUsers0' 2022-02-08 00:12:34,227 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:34,230 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:34,230 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:34,259 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:34,305 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:35,200 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:35,259 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 468847 == 0 2022-02-08 00:12:35,259 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 468847 == 0. Received: {'172.23.121.127': 468847} for bucket 'GleamBookUsers0' 2022-02-08 00:12:35,259 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:35,262 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:35,263 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:35,378 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 808491 == 0 2022-02-08 00:12:35,378 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 808491 == 0. Received: {'172.23.121.115': 808491} for bucket 'GleamBookUsers0' 2022-02-08 00:12:35,378 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:35,381 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:35,381 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:36,009 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:36,065 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:36,163 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 636382 == 0 2022-02-08 00:12:36,163 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 636382 == 0. Received: {'172.23.121.123': 636382} for bucket 'GleamBookUsers0' 2022-02-08 00:12:36,163 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:36,164 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:36,164 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:36,974 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 583468 == 0 2022-02-08 00:12:36,974 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 583468 == 0. Received: {'172.23.121.128': 583468} for bucket 'GleamBookUsers0' 2022-02-08 00:12:36,974 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:36,977 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:36,977 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:37,033 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 610517 == 0 2022-02-08 00:12:37,033 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 610517 == 0. Received: {'172.23.121.129': 610517} for bucket 'GleamBookUsers0' 2022-02-08 00:12:37,033 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:37,036 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:37,036 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:37,178 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:37,990 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 363950 == 0 2022-02-08 00:12:37,992 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 363950 == 0. Received: {'172.23.121.124': 363950} for bucket 'GleamBookUsers0' 2022-02-08 00:12:37,992 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:37,993 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:37,993 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:38,318 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:39,220 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 515378 == 0 2022-02-08 00:12:39,220 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 515378 == 0. Received: {'172.23.121.126': 515378} for bucket 'GleamBookUsers0' 2022-02-08 00:12:39,221 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:39,223 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:39,223 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:39,230 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:40,263 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:40,382 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:40,391 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 852005 == 0 2022-02-08 00:12:40,391 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 852005 == 0. Received: {'172.23.120.170': 852005} for bucket 'GleamBookUsers0' 2022-02-08 00:12:40,391 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:40,394 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:40,394 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:41,161 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 444906 == 0 2022-02-08 00:12:41,161 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 444906 == 0. Received: {'172.23.121.127': 444906} for bucket 'GleamBookUsers0' 2022-02-08 00:12:41,161 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:41,164 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:41,164 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:41,164 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:41,441 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 761162 == 0 2022-02-08 00:12:41,441 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 761162 == 0. Received: {'172.23.121.115': 761162} for bucket 'GleamBookUsers0' 2022-02-08 00:12:41,441 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:41,444 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:41,444 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:41,977 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:42,036 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:42,088 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 624519 == 0 2022-02-08 00:12:42,088 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 624519 == 0. Received: {'172.23.121.123': 624519} for bucket 'GleamBookUsers0' 2022-02-08 00:12:42,088 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:42,089 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:42,089 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:42,901 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 583468 == 0 2022-02-08 00:12:42,901 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 583468 == 0. Received: {'172.23.121.128': 583468} for bucket 'GleamBookUsers0' 2022-02-08 00:12:42,901 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:42,904 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:42,904 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:42,980 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 605638 == 0 2022-02-08 00:12:42,980 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 605638 == 0. Received: {'172.23.121.129': 605638} for bucket 'GleamBookUsers0' 2022-02-08 00:12:42,982 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:42,983 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:42,983 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:42,994 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:43,809 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 359724 == 0 2022-02-08 00:12:43,811 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 359724 == 0. Received: {'172.23.121.124': 359724} for bucket 'GleamBookUsers0' 2022-02-08 00:12:43,811 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:43,812 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:43,812 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:44,223 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:45,092 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 492378 == 0 2022-02-08 00:12:45,092 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 492378 == 0. Received: {'172.23.121.126': 492378} for bucket 'GleamBookUsers0' 2022-02-08 00:12:45,092 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:45,095 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:45,095 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:45,395 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:46,164 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:46,444 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:46,558 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 821299 == 0 2022-02-08 00:12:46,558 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 821299 == 0. Received: {'172.23.120.170': 821299} for bucket 'GleamBookUsers0' 2022-02-08 00:12:46,559 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:46,561 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:46,561 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:47,049 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 430090 == 0 2022-02-08 00:12:47,049 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 430090 == 0. Received: {'172.23.121.127': 430090} for bucket 'GleamBookUsers0' 2022-02-08 00:12:47,049 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:47,051 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:47,051 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:47,091 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:47,470 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 761162 == 0 2022-02-08 00:12:47,470 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 761162 == 0. Received: {'172.23.121.115': 761162} for bucket 'GleamBookUsers0' 2022-02-08 00:12:47,470 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:47,473 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:47,473 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:47,904 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:47,943 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 599681 == 0 2022-02-08 00:12:47,943 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 599681 == 0. Received: {'172.23.121.123': 599681} for bucket 'GleamBookUsers0' 2022-02-08 00:12:47,943 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:47,944 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:47,944 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:47,984 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:48,801 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 557975 == 0 2022-02-08 00:12:48,802 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 557975 == 0. Received: {'172.23.121.128': 557975} for bucket 'GleamBookUsers0' 2022-02-08 00:12:48,802 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:48,803 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:48,805 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:48,812 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:48,933 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 574420 == 0 2022-02-08 00:12:48,933 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 574420 == 0. Received: {'172.23.121.129': 574420} for bucket 'GleamBookUsers0' 2022-02-08 00:12:48,933 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:48,934 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:48,936 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:49,634 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 329833 == 0 2022-02-08 00:12:49,645 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 329833 == 0. Received: {'172.23.121.124': 329833} for bucket 'GleamBookUsers0' 2022-02-08 00:12:49,645 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:49,648 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:49,648 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:50,095 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:50,940 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 481316 == 0 2022-02-08 00:12:50,940 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 481316 == 0. Received: {'172.23.121.126': 481316} for bucket 'GleamBookUsers0' 2022-02-08 00:12:50,941 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:50,943 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:50,944 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:51,561 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:52,052 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:52,473 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:52,700 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 805657 == 0 2022-02-08 00:12:52,700 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 805657 == 0. Received: {'172.23.120.170': 805657} for bucket 'GleamBookUsers0' 2022-02-08 00:12:52,700 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:52,703 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:52,703 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:52,924 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 409686 == 0 2022-02-08 00:12:52,924 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 409686 == 0. Received: {'172.23.121.127': 409686} for bucket 'GleamBookUsers0' 2022-02-08 00:12:52,924 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:52,927 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:52,927 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:52,946 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:53,509 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 741844 == 0 2022-02-08 00:12:53,509 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 741844 == 0. Received: {'172.23.121.115': 741844} for bucket 'GleamBookUsers0' 2022-02-08 00:12:53,509 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:53,512 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:53,512 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:53,805 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:53,812 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 590039 == 0 2022-02-08 00:12:53,812 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 590039 == 0. Received: {'172.23.121.123': 590039} for bucket 'GleamBookUsers0' 2022-02-08 00:12:53,812 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:53,815 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:53,815 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:53,936 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:54,654 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:54,677 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 525641 == 0 2022-02-08 00:12:54,677 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 525641 == 0. Received: {'172.23.121.128': 525641} for bucket 'GleamBookUsers0' 2022-02-08 00:12:54,678 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:54,680 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:54,680 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:54,921 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 559802 == 0 2022-02-08 00:12:54,921 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 559802 == 0. Received: {'172.23.121.129': 559802} for bucket 'GleamBookUsers0' 2022-02-08 00:12:54,921 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:54,923 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:54,923 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:55,430 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 303389 == 0 2022-02-08 00:12:55,430 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 303389 == 0. Received: {'172.23.121.124': 303389} for bucket 'GleamBookUsers0' 2022-02-08 00:12:55,430 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:55,431 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:55,431 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:55,944 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:56,798 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 451724 == 0 2022-02-08 00:12:56,799 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 451724 == 0. Received: {'172.23.121.126': 451724} for bucket 'GleamBookUsers0' 2022-02-08 00:12:56,799 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:56,801 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:56,801 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:57,703 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:57,927 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:58,516 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:58,815 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:58,868 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 400318 == 0 2022-02-08 00:12:58,868 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 400318 == 0. Received: {'172.23.121.127': 400318} for bucket 'GleamBookUsers0' 2022-02-08 00:12:58,868 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:58,871 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:58,871 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:59,036 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 796755 == 0 2022-02-08 00:12:59,038 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 796755 == 0. Received: {'172.23.120.170': 796755} for bucket 'GleamBookUsers0' 2022-02-08 00:12:59,038 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:59,039 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:59,039 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:59,617 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 713614 == 0 2022-02-08 00:12:59,617 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 713614 == 0. Received: {'172.23.121.115': 713614} for bucket 'GleamBookUsers0' 2022-02-08 00:12:59,617 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:59,618 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:59,618 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:59,684 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:12:59,776 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 566085 == 0 2022-02-08 00:12:59,776 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 566085 == 0. Received: {'172.23.121.123': 566085} for bucket 'GleamBookUsers0' 2022-02-08 00:12:59,776 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:12:59,779 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:12:59,779 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:12:59,934 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:00,431 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:00,444 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 510401 == 0 2022-02-08 00:13:00,444 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 510401 == 0. Received: {'172.23.121.128': 510401} for bucket 'GleamBookUsers0' 2022-02-08 00:13:00,444 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:00,447 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:00,447 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:00,868 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 552746 == 0 2022-02-08 00:13:00,868 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 552746 == 0. Received: {'172.23.121.129': 552746} for bucket 'GleamBookUsers0' 2022-02-08 00:13:00,868 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:00,871 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:00,871 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:01,187 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 293529 == 0 2022-02-08 00:13:01,187 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 293529 == 0. Received: {'172.23.121.124': 293529} for bucket 'GleamBookUsers0' 2022-02-08 00:13:01,187 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:01,188 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:01,190 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:01,802 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:02,648 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 445824 == 0 2022-02-08 00:13:02,648 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 445824 == 0. Received: {'172.23.121.126': 445824} for bucket 'GleamBookUsers0' 2022-02-08 00:13:02,648 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:02,650 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:02,651 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:03,871 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:04,040 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:04,619 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:04,749 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 383544 == 0 2022-02-08 00:13:04,750 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 383544 == 0. Received: {'172.23.121.127': 383544} for bucket 'GleamBookUsers0' 2022-02-08 00:13:04,750 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:04,752 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:04,752 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:04,779 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:05,190 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 774363 == 0 2022-02-08 00:13:05,191 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 774363 == 0. Received: {'172.23.120.170': 774363} for bucket 'GleamBookUsers0' 2022-02-08 00:13:05,191 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:05,194 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:05,194 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:05,447 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:05,687 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 531366 == 0 2022-02-08 00:13:05,687 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 531366 == 0. Received: {'172.23.121.123': 531366} for bucket 'GleamBookUsers0' 2022-02-08 00:13:05,687 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:05,690 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:05,690 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:05,755 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 677368 == 0 2022-02-08 00:13:05,756 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 677368 == 0. Received: {'172.23.121.115': 677368} for bucket 'GleamBookUsers0' 2022-02-08 00:13:05,756 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:05,757 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:05,757 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:05,871 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:06,197 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:06,234 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 507391 == 0 2022-02-08 00:13:06,236 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 507391 == 0. Received: {'172.23.121.128': 507391} for bucket 'GleamBookUsers0' 2022-02-08 00:13:06,236 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:06,237 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:06,237 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:06,680 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 520334 == 0 2022-02-08 00:13:06,680 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 520334 == 0. Received: {'172.23.121.129': 520334} for bucket 'GleamBookUsers0' 2022-02-08 00:13:06,680 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:06,683 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:06,683 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:06,954 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 279745 == 0 2022-02-08 00:13:06,956 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 279745 == 0. Received: {'172.23.121.124': 279745} for bucket 'GleamBookUsers0' 2022-02-08 00:13:06,956 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:06,957 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:06,957 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:07,651 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:08,456 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 422792 == 0 2022-02-08 00:13:08,456 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 422792 == 0. Received: {'172.23.121.126': 422792} for bucket 'GleamBookUsers0' 2022-02-08 00:13:08,457 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:08,457 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:08,459 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:09,753 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:10,194 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:10,588 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 370534 == 0 2022-02-08 00:13:10,588 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 370534 == 0. Received: {'172.23.121.127': 370534} for bucket 'GleamBookUsers0' 2022-02-08 00:13:10,588 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:10,589 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:10,589 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:10,691 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:10,757 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:11,213 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 743512 == 0 2022-02-08 00:13:11,213 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 743512 == 0. Received: {'172.23.120.170': 743512} for bucket 'GleamBookUsers0' 2022-02-08 00:13:11,213 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:11,216 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:11,216 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:11,237 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:11,516 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 522638 == 0 2022-02-08 00:13:11,517 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 522638 == 0. Received: {'172.23.121.123': 522638} for bucket 'GleamBookUsers0' 2022-02-08 00:13:11,517 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:11,519 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:11,519 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:11,684 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:11,766 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 661898 == 0 2022-02-08 00:13:11,766 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 661898 == 0. Received: {'172.23.121.115': 661898} for bucket 'GleamBookUsers0' 2022-02-08 00:13:11,766 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:11,769 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:11,769 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:11,957 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:11,979 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 480953 == 0 2022-02-08 00:13:11,979 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 480953 == 0. Received: {'172.23.121.128': 480953} for bucket 'GleamBookUsers0' 2022-02-08 00:13:11,979 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:11,980 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:11,980 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:12,473 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 498175 == 0 2022-02-08 00:13:12,473 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 498175 == 0. Received: {'172.23.121.129': 498175} for bucket 'GleamBookUsers0' 2022-02-08 00:13:12,473 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:12,476 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:12,476 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:12,691 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 262076 == 0 2022-02-08 00:13:12,691 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 262076 == 0. Received: {'172.23.121.124': 262076} for bucket 'GleamBookUsers0' 2022-02-08 00:13:12,693 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:12,694 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:12,694 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:13,459 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:14,280 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 391376 == 0 2022-02-08 00:13:14,280 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 391376 == 0. Received: {'172.23.121.126': 391376} for bucket 'GleamBookUsers0' 2022-02-08 00:13:14,280 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:14,282 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:14,282 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:15,592 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:16,217 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:16,423 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 344468 == 0 2022-02-08 00:13:16,423 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 344468 == 0. Received: {'172.23.121.127': 344468} for bucket 'GleamBookUsers0' 2022-02-08 00:13:16,423 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:16,426 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:16,426 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:16,520 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:16,769 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:16,982 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:17,276 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 727240 == 0 2022-02-08 00:13:17,276 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 727240 == 0. Received: {'172.23.120.170': 727240} for bucket 'GleamBookUsers0' 2022-02-08 00:13:17,276 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:17,278 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:17,279 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:17,351 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 515740 == 0 2022-02-08 00:13:17,351 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 515740 == 0. Received: {'172.23.121.123': 515740} for bucket 'GleamBookUsers0' 2022-02-08 00:13:17,351 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:17,352 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:17,352 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:17,476 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:17,694 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:17,865 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 651590 == 0 2022-02-08 00:13:17,867 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 651590 == 0. Received: {'172.23.121.115': 651590} for bucket 'GleamBookUsers0' 2022-02-08 00:13:17,867 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:17,868 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:17,868 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:17,869 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 464079 == 0 2022-02-08 00:13:17,869 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 464079 == 0. Received: {'172.23.121.128': 464079} for bucket 'GleamBookUsers0' 2022-02-08 00:13:17,869 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:17,871 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:17,871 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:18,272 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 474559 == 0 2022-02-08 00:13:18,272 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 474559 == 0. Received: {'172.23.121.129': 474559} for bucket 'GleamBookUsers0' 2022-02-08 00:13:18,272 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:18,275 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:18,275 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:18,450 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 262076 == 0 2022-02-08 00:13:18,450 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 262076 == 0. Received: {'172.23.121.124': 262076} for bucket 'GleamBookUsers0' 2022-02-08 00:13:18,450 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:18,451 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:18,453 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:19,283 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:20,092 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 384384 == 0 2022-02-08 00:13:20,092 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 384384 == 0. Received: {'172.23.121.126': 384384} for bucket 'GleamBookUsers0' 2022-02-08 00:13:20,092 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:20,094 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:20,094 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:21,426 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:22,229 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 341442 == 0 2022-02-08 00:13:22,230 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 341442 == 0. Received: {'172.23.121.127': 341442} for bucket 'GleamBookUsers0' 2022-02-08 00:13:22,230 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:22,232 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:22,232 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:22,279 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:22,354 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:22,869 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:22,872 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:23,207 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 515740 == 0 2022-02-08 00:13:23,207 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 515740 == 0. Received: {'172.23.121.123': 515740} for bucket 'GleamBookUsers0' 2022-02-08 00:13:23,207 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:23,209 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:23,209 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:23,276 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:23,335 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 725720 == 0 2022-02-08 00:13:23,335 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 725720 == 0. Received: {'172.23.120.170': 725720} for bucket 'GleamBookUsers0' 2022-02-08 00:13:23,335 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:23,338 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:23,338 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:23,456 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:23,653 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 443436 == 0 2022-02-08 00:13:23,653 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 443436 == 0. Received: {'172.23.121.128': 443436} for bucket 'GleamBookUsers0' 2022-02-08 00:13:23,653 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:23,655 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:23,655 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:23,892 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 624200 == 0 2022-02-08 00:13:23,894 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 624200 == 0. Received: {'172.23.121.115': 624200} for bucket 'GleamBookUsers0' 2022-02-08 00:13:23,894 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:23,895 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:23,895 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:24,096 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 474559 == 0 2022-02-08 00:13:24,096 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 474559 == 0. Received: {'172.23.121.129': 474559} for bucket 'GleamBookUsers0' 2022-02-08 00:13:24,096 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:24,099 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:24,099 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:24,203 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 229287 == 0 2022-02-08 00:13:24,203 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 229287 == 0. Received: {'172.23.121.124': 229287} for bucket 'GleamBookUsers0' 2022-02-08 00:13:24,203 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:24,207 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:24,207 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:25,094 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:25,895 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 377638 == 0 2022-02-08 00:13:25,895 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 377638 == 0. Received: {'172.23.121.126': 377638} for bucket 'GleamBookUsers0' 2022-02-08 00:13:25,895 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:25,898 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:25,898 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:27,232 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:28,017 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 323090 == 0 2022-02-08 00:13:28,019 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 323090 == 0. Received: {'172.23.121.127': 323090} for bucket 'GleamBookUsers0' 2022-02-08 00:13:28,019 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:28,020 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:28,020 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:28,210 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:28,338 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:28,657 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:28,898 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:29,003 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 476890 == 0 2022-02-08 00:13:29,005 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 476890 == 0. Received: {'172.23.121.123': 476890} for bucket 'GleamBookUsers0' 2022-02-08 00:13:29,005 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:29,006 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:29,006 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:29,099 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:29,207 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:29,414 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 436344 == 0 2022-02-08 00:13:29,414 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 436344 == 0. Received: {'172.23.121.128': 436344} for bucket 'GleamBookUsers0' 2022-02-08 00:13:29,414 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:29,417 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:29,417 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:29,427 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 699377 == 0 2022-02-08 00:13:29,427 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 699377 == 0. Received: {'172.23.120.170': 699377} for bucket 'GleamBookUsers0' 2022-02-08 00:13:29,427 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:29,428 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:29,428 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:29,953 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 448566 == 0 2022-02-08 00:13:29,954 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 448566 == 0. Received: {'172.23.121.129': 448566} for bucket 'GleamBookUsers0' 2022-02-08 00:13:29,954 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:29,957 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:29,957 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:29,974 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 586116 == 0 2022-02-08 00:13:29,974 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 586116 == 0. Received: {'172.23.121.115': 586116} for bucket 'GleamBookUsers0' 2022-02-08 00:13:29,974 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:29,976 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:29,977 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:30,005 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 224661 == 0 2022-02-08 00:13:30,005 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 224661 == 0. Received: {'172.23.121.124': 224661} for bucket 'GleamBookUsers0' 2022-02-08 00:13:30,005 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:30,006 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:30,006 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:30,898 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:31,647 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 344638 == 0 2022-02-08 00:13:31,648 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 344638 == 0. Received: {'172.23.121.126': 344638} for bucket 'GleamBookUsers0' 2022-02-08 00:13:31,648 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:31,650 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:31,650 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:33,020 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:33,822 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 307980 == 0 2022-02-08 00:13:33,822 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 307980 == 0. Received: {'172.23.121.127': 307980} for bucket 'GleamBookUsers0' 2022-02-08 00:13:33,822 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:33,825 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:33,825 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:34,007 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:34,418 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:34,430 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:34,802 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 445978 == 0 2022-02-08 00:13:34,802 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 445978 == 0. Received: {'172.23.121.123': 445978} for bucket 'GleamBookUsers0' 2022-02-08 00:13:34,802 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:34,805 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:34,805 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:34,957 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:34,977 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:35,006 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:35,111 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 417636 == 0 2022-02-08 00:13:35,111 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 417636 == 0. Received: {'172.23.121.128': 417636} for bucket 'GleamBookUsers0' 2022-02-08 00:13:35,111 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:35,112 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:35,114 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:35,457 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 687463 == 0 2022-02-08 00:13:35,457 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 687463 == 0. Received: {'172.23.120.170': 687463} for bucket 'GleamBookUsers0' 2022-02-08 00:13:35,457 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:35,460 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:35,460 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:35,759 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 427606 == 0 2022-02-08 00:13:35,759 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 427606 == 0. Received: {'172.23.121.129': 427606} for bucket 'GleamBookUsers0' 2022-02-08 00:13:35,759 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:35,763 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:35,763 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:35,901 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 222631 == 0 2022-02-08 00:13:35,901 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 222631 == 0. Received: {'172.23.121.124': 222631} for bucket 'GleamBookUsers0' 2022-02-08 00:13:35,901 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:35,904 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:35,904 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:36,006 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 575838 == 0 2022-02-08 00:13:36,007 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 575838 == 0. Received: {'172.23.121.115': 575838} for bucket 'GleamBookUsers0' 2022-02-08 00:13:36,007 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:36,009 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:36,009 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:36,650 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:37,354 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 333890 == 0 2022-02-08 00:13:37,355 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 333890 == 0. Received: {'172.23.121.126': 333890} for bucket 'GleamBookUsers0' 2022-02-08 00:13:37,355 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:37,357 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:37,357 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:38,825 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:39,565 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 286360 == 0 2022-02-08 00:13:39,566 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 286360 == 0. Received: {'172.23.121.127': 286360} for bucket 'GleamBookUsers0' 2022-02-08 00:13:39,566 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:39,569 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:39,569 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:39,805 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:40,114 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:40,461 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:40,615 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 445366 == 0 2022-02-08 00:13:40,615 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 445366 == 0. Received: {'172.23.121.123': 445366} for bucket 'GleamBookUsers0' 2022-02-08 00:13:40,615 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:40,618 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:40,618 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:40,763 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:40,831 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 416552 == 0 2022-02-08 00:13:40,832 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 416552 == 0. Received: {'172.23.121.128': 416552} for bucket 'GleamBookUsers0' 2022-02-08 00:13:40,832 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:40,834 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:40,834 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:40,904 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:41,009 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:41,592 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 659805 == 0 2022-02-08 00:13:41,592 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 659805 == 0. Received: {'172.23.120.170': 659805} for bucket 'GleamBookUsers0' 2022-02-08 00:13:41,594 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:41,595 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:41,595 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:41,668 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 424592 == 0 2022-02-08 00:13:41,668 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 424592 == 0. Received: {'172.23.121.129': 424592} for bucket 'GleamBookUsers0' 2022-02-08 00:13:41,668 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:41,671 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:41,671 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:41,776 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 211743 == 0 2022-02-08 00:13:41,776 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 211743 == 0. Received: {'172.23.121.124': 211743} for bucket 'GleamBookUsers0' 2022-02-08 00:13:41,776 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:41,779 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:41,779 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:41,982 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 554298 == 0 2022-02-08 00:13:41,983 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 554298 == 0. Received: {'172.23.121.115': 554298} for bucket 'GleamBookUsers0' 2022-02-08 00:13:41,983 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:41,984 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:41,984 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:42,367 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:43,071 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 325312 == 0 2022-02-08 00:13:43,071 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 325312 == 0. Received: {'172.23.121.126': 325312} for bucket 'GleamBookUsers0' 2022-02-08 00:13:43,071 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:43,072 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:43,072 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:44,569 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:45,275 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 275732 == 0 2022-02-08 00:13:45,276 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 275732 == 0. Received: {'172.23.121.127': 275732} for bucket 'GleamBookUsers0' 2022-02-08 00:13:45,276 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:45,278 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:45,278 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:45,625 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:45,834 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:46,400 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 425168 == 0 2022-02-08 00:13:46,401 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 425168 == 0. Received: {'172.23.121.123': 425168} for bucket 'GleamBookUsers0' 2022-02-08 00:13:46,401 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:46,403 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:46,403 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:46,522 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 394792 == 0 2022-02-08 00:13:46,522 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 394792 == 0. Received: {'172.23.121.128': 394792} for bucket 'GleamBookUsers0' 2022-02-08 00:13:46,522 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:46,523 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:46,523 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:46,596 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:46,673 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:46,779 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:46,986 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:47,559 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 406738 == 0 2022-02-08 00:13:47,559 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 406738 == 0. Received: {'172.23.121.129': 406738} for bucket 'GleamBookUsers0' 2022-02-08 00:13:47,559 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:47,561 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:47,561 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:47,588 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 185155 == 0 2022-02-08 00:13:47,588 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 185155 == 0. Received: {'172.23.121.124': 185155} for bucket 'GleamBookUsers0' 2022-02-08 00:13:47,588 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:47,591 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:47,591 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:47,806 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 634627 == 0 2022-02-08 00:13:47,806 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 634627 == 0. Received: {'172.23.120.170': 634627} for bucket 'GleamBookUsers0' 2022-02-08 00:13:47,806 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:47,809 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:47,809 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:47,933 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 541502 == 0 2022-02-08 00:13:47,934 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 541502 == 0. Received: {'172.23.121.115': 541502} for bucket 'GleamBookUsers0' 2022-02-08 00:13:47,934 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:47,936 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:47,936 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:48,072 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:48,753 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 307232 == 0 2022-02-08 00:13:48,753 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 307232 == 0. Received: {'172.23.121.126': 307232} for bucket 'GleamBookUsers0' 2022-02-08 00:13:48,753 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:48,756 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:48,756 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:50,278 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:50,984 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 260664 == 0 2022-02-08 00:13:50,984 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 260664 == 0. Received: {'172.23.121.127': 260664} for bucket 'GleamBookUsers0' 2022-02-08 00:13:50,984 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:50,986 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:50,986 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:51,407 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:51,525 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:52,252 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 404444 == 0 2022-02-08 00:13:52,252 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 404444 == 0. Received: {'172.23.121.123': 404444} for bucket 'GleamBookUsers0' 2022-02-08 00:13:52,252 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:52,253 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:52,255 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:52,276 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 381454 == 0 2022-02-08 00:13:52,278 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 381454 == 0. Received: {'172.23.121.128': 381454} for bucket 'GleamBookUsers0' 2022-02-08 00:13:52,278 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:52,279 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:52,279 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:52,562 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:52,591 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:52,809 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:52,936 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:53,305 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 182665 == 0 2022-02-08 00:13:53,305 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 182665 == 0. Received: {'172.23.121.124': 182665} for bucket 'GleamBookUsers0' 2022-02-08 00:13:53,305 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:53,308 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:53,309 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:53,349 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 380901 == 0 2022-02-08 00:13:53,349 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 380901 == 0. Received: {'172.23.121.129': 380901} for bucket 'GleamBookUsers0' 2022-02-08 00:13:53,351 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:53,351 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:53,352 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:53,759 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:53,871 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 610317 == 0 2022-02-08 00:13:53,872 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 610317 == 0. Received: {'172.23.120.170': 610317} for bucket 'GleamBookUsers0' 2022-02-08 00:13:53,872 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:53,875 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:53,875 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:53,930 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 539988 == 0 2022-02-08 00:13:53,930 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 539988 == 0. Received: {'172.23.121.115': 539988} for bucket 'GleamBookUsers0' 2022-02-08 00:13:53,930 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:53,933 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:53,933 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:54,453 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 307232 == 0 2022-02-08 00:13:54,453 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 307232 == 0. Received: {'172.23.121.126': 307232} for bucket 'GleamBookUsers0' 2022-02-08 00:13:54,453 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:54,454 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:54,456 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:55,987 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:56,661 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 243295 == 0 2022-02-08 00:13:56,661 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 243295 == 0. Received: {'172.23.121.127': 243295} for bucket 'GleamBookUsers0' 2022-02-08 00:13:56,661 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:56,664 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:56,664 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:57,255 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:57,279 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:57,957 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 364468 == 0 2022-02-08 00:13:57,957 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 364468 == 0. Received: {'172.23.121.128': 364468} for bucket 'GleamBookUsers0' 2022-02-08 00:13:57,957 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:57,960 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:57,960 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:58,046 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 404444 == 0 2022-02-08 00:13:58,046 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 404444 == 0. Received: {'172.23.121.123': 404444} for bucket 'GleamBookUsers0' 2022-02-08 00:13:58,046 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:58,049 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:58,049 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:58,309 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:58,352 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:58,877 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:58,933 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:58,948 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 170563 == 0 2022-02-08 00:13:58,948 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 170563 == 0. Received: {'172.23.121.124': 170563} for bucket 'GleamBookUsers0' 2022-02-08 00:13:58,948 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:58,951 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:58,951 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:59,075 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 365987 == 0 2022-02-08 00:13:59,075 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 365987 == 0. Received: {'172.23.121.129': 365987} for bucket 'GleamBookUsers0' 2022-02-08 00:13:59,075 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:59,078 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:59,078 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:59,456 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:13:59,841 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 593849 == 0 2022-02-08 00:13:59,841 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 593849 == 0. Received: {'172.23.120.170': 593849} for bucket 'GleamBookUsers0' 2022-02-08 00:13:59,841 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:59,842 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:59,844 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:13:59,956 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 499566 == 0 2022-02-08 00:13:59,956 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 499566 == 0. Received: {'172.23.121.115': 499566} for bucket 'GleamBookUsers0' 2022-02-08 00:13:59,957 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:13:59,959 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:13:59,959 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:00,134 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 270256 == 0 2022-02-08 00:14:00,135 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 270256 == 0. Received: {'172.23.121.126': 270256} for bucket 'GleamBookUsers0' 2022-02-08 00:14:00,135 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:00,138 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:00,138 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:01,671 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:02,348 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 232307 == 0 2022-02-08 00:14:02,348 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 232307 == 0. Received: {'172.23.121.127': 232307} for bucket 'GleamBookUsers0' 2022-02-08 00:14:02,348 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:02,352 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:02,354 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:02,960 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:03,049 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:03,592 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 347178 == 0 2022-02-08 00:14:03,592 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 347178 == 0. Received: {'172.23.121.128': 347178} for bucket 'GleamBookUsers0' 2022-02-08 00:14:03,592 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:03,594 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:03,594 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:03,838 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 377380 == 0 2022-02-08 00:14:03,838 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 377380 == 0. Received: {'172.23.121.123': 377380} for bucket 'GleamBookUsers0' 2022-02-08 00:14:03,838 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:03,841 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:03,841 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:03,951 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:04,078 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:04,565 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 144233 == 0 2022-02-08 00:14:04,565 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 144233 == 0. Received: {'172.23.121.124': 144233} for bucket 'GleamBookUsers0' 2022-02-08 00:14:04,565 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:04,568 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:04,568 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:04,780 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 342689 == 0 2022-02-08 00:14:04,780 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 342689 == 0. Received: {'172.23.121.129': 342689} for bucket 'GleamBookUsers0' 2022-02-08 00:14:04,780 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:04,782 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:04,782 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:04,844 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:04,960 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:05,138 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:06,009 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 254244 == 0 2022-02-08 00:14:06,009 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 254244 == 0. Received: {'172.23.121.126': 254244} for bucket 'GleamBookUsers0' 2022-02-08 00:14:06,009 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:06,012 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:06,013 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:06,023 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 567349 == 0 2022-02-08 00:14:06,023 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 567349 == 0. Received: {'172.23.120.170': 567349} for bucket 'GleamBookUsers0' 2022-02-08 00:14:06,023 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:06,026 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:06,026 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:06,062 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 481794 == 0 2022-02-08 00:14:06,063 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 481794 == 0. Received: {'172.23.121.115': 481794} for bucket 'GleamBookUsers0' 2022-02-08 00:14:06,063 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:06,065 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:06,065 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:07,354 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:08,033 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 229076 == 0 2022-02-08 00:14:08,035 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 229076 == 0. Received: {'172.23.121.127': 229076} for bucket 'GleamBookUsers0' 2022-02-08 00:14:08,035 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:08,036 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:08,036 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:08,595 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:08,842 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:09,213 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 327558 == 0 2022-02-08 00:14:09,213 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 327558 == 0. Received: {'172.23.121.128': 327558} for bucket 'GleamBookUsers0' 2022-02-08 00:14:09,213 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:09,216 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:09,216 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:09,523 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 361969 == 0 2022-02-08 00:14:09,525 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 361969 == 0. Received: {'172.23.121.123': 361969} for bucket 'GleamBookUsers0' 2022-02-08 00:14:09,525 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:09,526 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:09,526 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:09,568 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:09,783 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:10,154 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 144233 == 0 2022-02-08 00:14:10,154 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 144233 == 0. Received: {'172.23.121.124': 144233} for bucket 'GleamBookUsers0' 2022-02-08 00:14:10,155 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:10,157 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:10,157 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:10,464 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 328933 == 0 2022-02-08 00:14:10,464 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 328933 == 0. Received: {'172.23.121.129': 328933} for bucket 'GleamBookUsers0' 2022-02-08 00:14:10,464 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:10,467 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:10,467 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:11,013 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:11,028 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:11,066 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:11,674 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 254244 == 0 2022-02-08 00:14:11,674 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 254244 == 0. Received: {'172.23.121.126': 254244} for bucket 'GleamBookUsers0' 2022-02-08 00:14:11,674 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:11,677 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:11,677 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:11,979 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 537481 == 0 2022-02-08 00:14:11,980 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 537481 == 0. Received: {'172.23.120.170': 537481} for bucket 'GleamBookUsers0' 2022-02-08 00:14:11,980 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:11,982 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:11,982 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:12,038 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 472912 == 0 2022-02-08 00:14:12,038 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 472912 == 0. Received: {'172.23.121.115': 472912} for bucket 'GleamBookUsers0' 2022-02-08 00:14:12,038 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:12,039 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:12,039 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:13,038 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:13,716 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 209282 == 0 2022-02-08 00:14:13,716 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 209282 == 0. Received: {'172.23.121.127': 209282} for bucket 'GleamBookUsers0' 2022-02-08 00:14:13,717 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:13,719 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:13,719 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:14,217 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:14,526 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:14,812 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 315326 == 0 2022-02-08 00:14:14,812 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 315326 == 0. Received: {'172.23.121.128': 315326} for bucket 'GleamBookUsers0' 2022-02-08 00:14:14,813 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:14,815 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:14,815 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:15,157 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:15,206 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 361969 == 0 2022-02-08 00:14:15,207 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 361969 == 0. Received: {'172.23.121.123': 361969} for bucket 'GleamBookUsers0' 2022-02-08 00:14:15,207 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:15,209 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:15,209 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:15,467 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:15,730 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 128072 == 0 2022-02-08 00:14:15,730 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 128072 == 0. Received: {'172.23.121.124': 128072} for bucket 'GleamBookUsers0' 2022-02-08 00:14:15,730 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:15,733 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:15,733 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:16,134 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 304437 == 0 2022-02-08 00:14:16,135 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 304437 == 0. Received: {'172.23.121.129': 304437} for bucket 'GleamBookUsers0' 2022-02-08 00:14:16,135 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:16,138 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:16,138 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:16,677 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:16,983 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:17,040 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:17,263 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 224826 == 0 2022-02-08 00:14:17,263 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 224826 == 0. Received: {'172.23.121.126': 224826} for bucket 'GleamBookUsers0' 2022-02-08 00:14:17,263 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:17,266 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:17,266 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:17,944 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 522404 == 0 2022-02-08 00:14:17,944 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 522404 == 0. Received: {'172.23.120.170': 522404} for bucket 'GleamBookUsers0' 2022-02-08 00:14:17,946 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:17,947 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:17,947 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:17,980 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 461482 == 0 2022-02-08 00:14:17,980 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 461482 == 0. Received: {'172.23.121.115': 461482} for bucket 'GleamBookUsers0' 2022-02-08 00:14:17,980 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:17,983 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:17,983 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:18,720 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:19,355 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 194230 == 0 2022-02-08 00:14:19,355 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 194230 == 0. Received: {'172.23.121.127': 194230} for bucket 'GleamBookUsers0' 2022-02-08 00:14:19,355 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:19,378 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:19,378 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:19,816 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:20,210 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:20,381 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 300896 == 0 2022-02-08 00:14:20,381 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 300896 == 0. Received: {'172.23.121.128': 300896} for bucket 'GleamBookUsers0' 2022-02-08 00:14:20,382 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:20,384 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:20,384 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:20,733 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:20,900 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 339824 == 0 2022-02-08 00:14:20,900 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 339824 == 0. Received: {'172.23.121.123': 339824} for bucket 'GleamBookUsers0' 2022-02-08 00:14:20,900 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:20,901 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:20,901 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:21,138 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:21,278 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 107022 == 0 2022-02-08 00:14:21,278 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 107022 == 0. Received: {'172.23.121.124': 107022} for bucket 'GleamBookUsers0' 2022-02-08 00:14:21,278 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:21,279 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:21,279 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:21,845 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 283705 == 0 2022-02-08 00:14:21,845 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 283705 == 0. Received: {'172.23.121.129': 283705} for bucket 'GleamBookUsers0' 2022-02-08 00:14:21,845 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:21,848 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:21,848 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:22,266 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:22,864 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 212266 == 0 2022-02-08 00:14:22,864 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 212266 == 0. Received: {'172.23.121.126': 212266} for bucket 'GleamBookUsers0' 2022-02-08 00:14:22,864 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:22,867 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:22,867 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:22,947 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:22,983 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:23,832 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 412805 == 0 2022-02-08 00:14:23,832 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 412805 == 0. Received: {'172.23.121.115': 412805} for bucket 'GleamBookUsers0' 2022-02-08 00:14:23,832 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:23,835 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:23,835 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:23,864 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 489730 == 0 2022-02-08 00:14:23,864 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 489730 == 0. Received: {'172.23.120.170': 489730} for bucket 'GleamBookUsers0' 2022-02-08 00:14:23,865 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:23,867 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:23,867 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:24,380 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:24,989 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 169574 == 0 2022-02-08 00:14:24,989 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 169574 == 0. Received: {'172.23.121.127': 169574} for bucket 'GleamBookUsers0' 2022-02-08 00:14:24,989 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:25,048 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:25,048 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:25,391 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:25,903 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:26,009 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 285058 == 0 2022-02-08 00:14:26,009 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 285058 == 0. Received: {'172.23.121.128': 285058} for bucket 'GleamBookUsers0' 2022-02-08 00:14:26,010 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:26,012 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:26,013 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:26,280 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:26,516 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 316070 == 0 2022-02-08 00:14:26,516 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 316070 == 0. Received: {'172.23.121.123': 316070} for bucket 'GleamBookUsers0' 2022-02-08 00:14:26,517 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:26,519 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:26,519 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:26,811 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 93877 == 0 2022-02-08 00:14:26,811 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 93877 == 0. Received: {'172.23.121.124': 93877} for bucket 'GleamBookUsers0' 2022-02-08 00:14:26,812 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:26,812 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:26,813 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:26,848 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:27,513 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 261761 == 0 2022-02-08 00:14:27,515 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 261761 == 0. Received: {'172.23.121.129': 261761} for bucket 'GleamBookUsers0' 2022-02-08 00:14:27,515 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:27,516 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:27,516 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:27,868 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:28,417 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 179906 == 0 2022-02-08 00:14:28,417 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 179906 == 0. Received: {'172.23.121.126': 179906} for bucket 'GleamBookUsers0' 2022-02-08 00:14:28,417 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:28,420 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:28,421 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:28,835 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:28,868 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:29,658 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 397475 == 0 2022-02-08 00:14:29,660 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 397475 == 0. Received: {'172.23.121.115': 397475} for bucket 'GleamBookUsers0' 2022-02-08 00:14:29,660 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:29,661 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:29,661 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:29,750 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 454161 == 0 2022-02-08 00:14:29,750 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 454161 == 0. Received: {'172.23.120.170': 454161} for bucket 'GleamBookUsers0' 2022-02-08 00:14:29,750 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:29,753 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:29,753 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:30,049 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:30,637 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 161438 == 0 2022-02-08 00:14:30,637 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 161438 == 0. Received: {'172.23.121.127': 161438} for bucket 'GleamBookUsers0' 2022-02-08 00:14:30,638 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:30,638 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:30,640 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:31,013 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:31,520 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:31,562 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 267524 == 0 2022-02-08 00:14:31,562 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 267524 == 0. Received: {'172.23.121.128': 267524} for bucket 'GleamBookUsers0' 2022-02-08 00:14:31,562 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:31,565 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:31,565 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:31,815 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:32,108 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 283671 == 0 2022-02-08 00:14:32,108 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 283671 == 0. Received: {'172.23.121.123': 283671} for bucket 'GleamBookUsers0' 2022-02-08 00:14:32,108 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:32,109 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:32,111 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:32,302 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 79783 == 0 2022-02-08 00:14:32,302 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 79783 == 0. Received: {'172.23.121.124': 79783} for bucket 'GleamBookUsers0' 2022-02-08 00:14:32,302 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:32,305 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:32,305 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:32,516 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:33,154 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 243985 == 0 2022-02-08 00:14:33,154 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 243985 == 0. Received: {'172.23.121.129': 243985} for bucket 'GleamBookUsers0' 2022-02-08 00:14:33,154 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:33,157 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:33,157 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:33,421 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:33,970 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 164475 == 0 2022-02-08 00:14:33,970 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 164475 == 0. Received: {'172.23.121.126': 164475} for bucket 'GleamBookUsers0' 2022-02-08 00:14:33,970 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:33,973 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:33,973 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:34,663 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:34,755 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:35,417 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 357587 == 0 2022-02-08 00:14:35,417 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 357587 == 0. Received: {'172.23.121.115': 357587} for bucket 'GleamBookUsers0' 2022-02-08 00:14:35,417 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:35,420 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:35,420 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:35,637 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 433091 == 0 2022-02-08 00:14:35,638 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 433091 == 0. Received: {'172.23.120.170': 433091} for bucket 'GleamBookUsers0' 2022-02-08 00:14:35,638 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:35,640 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:35,640 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:35,640 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:36,177 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 139194 == 0 2022-02-08 00:14:36,177 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 139194 == 0. Received: {'172.23.121.127': 139194} for bucket 'GleamBookUsers0' 2022-02-08 00:14:36,177 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:36,180 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:36,180 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:36,565 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:37,102 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 242104 == 0 2022-02-08 00:14:37,102 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 242104 == 0. Received: {'172.23.121.128': 242104} for bucket 'GleamBookUsers0' 2022-02-08 00:14:37,102 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:37,105 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:37,105 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:37,111 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:37,305 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:37,717 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 271873 == 0 2022-02-08 00:14:37,717 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 271873 == 0. Received: {'172.23.121.123': 271873} for bucket 'GleamBookUsers0' 2022-02-08 00:14:37,717 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:37,720 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:37,720 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:37,819 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 71983 == 0 2022-02-08 00:14:37,819 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 71983 == 0. Received: {'172.23.121.124': 71983} for bucket 'GleamBookUsers0' 2022-02-08 00:14:37,819 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:37,821 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:37,822 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:38,157 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:38,819 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 243985 == 0 2022-02-08 00:14:38,821 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 243985 == 0. Received: {'172.23.121.129': 243985} for bucket 'GleamBookUsers0' 2022-02-08 00:14:38,821 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:38,822 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:38,822 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:38,973 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:39,500 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 160219 == 0 2022-02-08 00:14:39,500 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 160219 == 0. Received: {'172.23.121.126': 160219} for bucket 'GleamBookUsers0' 2022-02-08 00:14:39,500 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:39,502 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:39,503 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:40,421 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:40,640 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:41,180 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:41,197 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 349371 == 0 2022-02-08 00:14:41,197 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 349371 == 0. Received: {'172.23.121.115': 349371} for bucket 'GleamBookUsers0' 2022-02-08 00:14:41,197 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:41,198 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:41,200 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:41,471 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 413226 == 0 2022-02-08 00:14:41,471 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 413226 == 0. Received: {'172.23.120.170': 413226} for bucket 'GleamBookUsers0' 2022-02-08 00:14:41,471 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:41,473 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:41,473 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:41,684 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 132918 == 0 2022-02-08 00:14:41,684 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 132918 == 0. Received: {'172.23.121.127': 132918} for bucket 'GleamBookUsers0' 2022-02-08 00:14:41,684 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:41,686 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:41,687 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:42,105 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:42,632 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 231444 == 0 2022-02-08 00:14:42,634 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 231444 == 0. Received: {'172.23.121.128': 231444} for bucket 'GleamBookUsers0' 2022-02-08 00:14:42,634 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:42,635 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:42,635 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:42,720 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:42,822 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:43,305 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 60849 == 0 2022-02-08 00:14:43,306 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 60849 == 0. Received: {'172.23.121.124': 60849} for bucket 'GleamBookUsers0' 2022-02-08 00:14:43,306 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:43,309 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:43,309 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:43,368 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 242277 == 0 2022-02-08 00:14:43,368 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 242277 == 0. Received: {'172.23.121.123': 242277} for bucket 'GleamBookUsers0' 2022-02-08 00:14:43,368 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:43,371 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:43,371 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:43,823 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:44,463 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 218137 == 0 2022-02-08 00:14:44,464 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 218137 == 0. Received: {'172.23.121.129': 218137} for bucket 'GleamBookUsers0' 2022-02-08 00:14:44,464 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:44,467 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:44,467 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:44,503 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:45,029 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 155845 == 0 2022-02-08 00:14:45,029 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 155845 == 0. Received: {'172.23.121.126': 155845} for bucket 'GleamBookUsers0' 2022-02-08 00:14:45,029 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:45,032 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:45,032 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:46,200 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:46,474 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:46,687 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:46,997 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 324361 == 0 2022-02-08 00:14:46,997 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 324361 == 0. Received: {'172.23.121.115': 324361} for bucket 'GleamBookUsers0' 2022-02-08 00:14:46,997 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:47,000 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:47,000 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:47,213 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 124304 == 0 2022-02-08 00:14:47,213 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 124304 == 0. Received: {'172.23.121.127': 124304} for bucket 'GleamBookUsers0' 2022-02-08 00:14:47,213 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:47,216 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:47,216 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:47,312 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 392656 == 0 2022-02-08 00:14:47,312 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 392656 == 0. Received: {'172.23.120.170': 392656} for bucket 'GleamBookUsers0' 2022-02-08 00:14:47,312 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:47,313 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:47,313 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:47,635 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:48,115 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 208864 == 0 2022-02-08 00:14:48,115 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 208864 == 0. Received: {'172.23.121.128': 208864} for bucket 'GleamBookUsers0' 2022-02-08 00:14:48,117 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:48,118 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:48,118 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:48,309 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:48,371 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:48,766 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 54493 == 0 2022-02-08 00:14:48,766 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 54493 == 0. Received: {'172.23.121.124': 54493} for bucket 'GleamBookUsers0' 2022-02-08 00:14:48,766 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:48,767 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:48,767 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:48,986 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 230823 == 0 2022-02-08 00:14:48,987 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 230823 == 0. Received: {'172.23.121.123': 230823} for bucket 'GleamBookUsers0' 2022-02-08 00:14:48,987 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:48,989 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:48,990 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:49,467 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:50,036 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:50,138 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 207295 == 0 2022-02-08 00:14:50,138 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 207295 == 0. Received: {'172.23.121.129': 207295} for bucket 'GleamBookUsers0' 2022-02-08 00:14:50,138 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:50,141 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:50,141 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:50,551 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 136009 == 0 2022-02-08 00:14:50,551 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 136009 == 0. Received: {'172.23.121.126': 136009} for bucket 'GleamBookUsers0' 2022-02-08 00:14:50,552 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:50,552 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:50,553 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:52,000 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:52,217 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:52,313 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:52,743 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 106694 == 0 2022-02-08 00:14:52,744 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 106694 == 0. Received: {'172.23.121.127': 106694} for bucket 'GleamBookUsers0' 2022-02-08 00:14:52,744 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:52,746 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:52,746 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:52,786 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 297187 == 0 2022-02-08 00:14:52,786 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 297187 == 0. Received: {'172.23.121.115': 297187} for bucket 'GleamBookUsers0' 2022-02-08 00:14:52,786 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:52,788 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:52,788 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:53,112 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 362656 == 0 2022-02-08 00:14:53,114 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 362656 == 0. Received: {'172.23.120.170': 362656} for bucket 'GleamBookUsers0' 2022-02-08 00:14:53,114 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:53,115 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:53,115 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:53,121 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:53,579 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 178464 == 0 2022-02-08 00:14:53,579 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 178464 == 0. Received: {'172.23.121.128': 178464} for bucket 'GleamBookUsers0' 2022-02-08 00:14:53,579 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:53,582 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:53,582 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:53,769 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:53,990 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:54,164 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 39839 == 0 2022-02-08 00:14:54,164 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 39839 == 0. Received: {'172.23.121.124': 39839} for bucket 'GleamBookUsers0' 2022-02-08 00:14:54,164 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:54,165 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:54,165 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:54,536 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 204447 == 0 2022-02-08 00:14:54,536 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 204447 == 0. Received: {'172.23.121.123': 204447} for bucket 'GleamBookUsers0' 2022-02-08 00:14:54,536 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:54,539 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:54,539 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:55,141 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:55,553 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:55,786 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 184265 == 0 2022-02-08 00:14:55,786 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 184265 == 0. Received: {'172.23.121.129': 184265} for bucket 'GleamBookUsers0' 2022-02-08 00:14:55,786 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:55,789 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:55,789 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:56,059 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 120603 == 0 2022-02-08 00:14:56,059 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 120603 == 0. Received: {'172.23.121.126': 120603} for bucket 'GleamBookUsers0' 2022-02-08 00:14:56,059 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:56,062 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:56,062 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:57,747 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:57,789 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:58,117 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:58,219 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 89208 == 0 2022-02-08 00:14:58,220 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 89208 == 0. Received: {'172.23.121.127': 89208} for bucket 'GleamBookUsers0' 2022-02-08 00:14:58,220 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:58,221 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:58,221 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:58,536 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 271416 == 0 2022-02-08 00:14:58,536 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 271416 == 0. Received: {'172.23.121.115': 271416} for bucket 'GleamBookUsers0' 2022-02-08 00:14:58,536 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:58,539 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:58,539 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:58,582 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:58,898 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 341273 == 0 2022-02-08 00:14:58,898 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 341273 == 0. Received: {'172.23.120.170': 341273} for bucket 'GleamBookUsers0' 2022-02-08 00:14:58,898 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:58,901 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:58,901 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:59,032 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 178464 == 0 2022-02-08 00:14:59,032 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 178464 == 0. Received: {'172.23.121.128': 178464} for bucket 'GleamBookUsers0' 2022-02-08 00:14:59,032 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:59,033 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:59,033 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:14:59,167 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:59,539 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:14:59,551 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 32916 == 0 2022-02-08 00:14:59,551 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 32916 == 0. Received: {'172.23.121.124': 32916} for bucket 'GleamBookUsers0' 2022-02-08 00:14:59,552 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:14:59,553 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:14:59,553 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:00,084 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 193635 == 0 2022-02-08 00:15:00,085 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 193635 == 0. Received: {'172.23.121.123': 193635} for bucket 'GleamBookUsers0' 2022-02-08 00:15:00,085 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:00,086 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:00,086 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:00,789 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:01,062 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:01,411 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 173849 == 0 2022-02-08 00:15:01,411 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 173849 == 0. Received: {'172.23.121.129': 173849} for bucket 'GleamBookUsers0' 2022-02-08 00:15:01,411 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:01,414 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:01,414 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:01,551 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 112641 == 0 2022-02-08 00:15:01,551 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 112641 == 0. Received: {'172.23.121.126': 112641} for bucket 'GleamBookUsers0' 2022-02-08 00:15:01,551 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:01,552 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:01,552 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:03,221 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:03,539 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:03,680 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 83734 == 0 2022-02-08 00:15:03,680 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 83734 == 0. Received: {'172.23.121.127': 83734} for bucket 'GleamBookUsers0' 2022-02-08 00:15:03,680 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:03,683 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:03,683 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:03,901 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:04,038 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:04,265 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 253128 == 0 2022-02-08 00:15:04,265 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 253128 == 0. Received: {'172.23.121.115': 253128} for bucket 'GleamBookUsers0' 2022-02-08 00:15:04,265 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:04,266 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:04,267 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:04,486 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 157556 == 0 2022-02-08 00:15:04,486 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 157556 == 0. Received: {'172.23.121.128': 157556} for bucket 'GleamBookUsers0' 2022-02-08 00:15:04,486 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:04,487 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:04,489 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:04,553 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:04,690 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 325862 == 0 2022-02-08 00:15:04,690 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 325862 == 0. Received: {'172.23.120.170': 325862} for bucket 'GleamBookUsers0' 2022-02-08 00:15:04,690 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:04,693 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:04,694 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:04,974 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 22640 == 0 2022-02-08 00:15:04,974 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 22640 == 0. Received: {'172.23.121.124': 22640} for bucket 'GleamBookUsers0' 2022-02-08 00:15:04,974 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:04,977 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:04,977 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:05,086 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:05,601 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 180585 == 0 2022-02-08 00:15:05,601 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 180585 == 0. Received: {'172.23.121.123': 180585} for bucket 'GleamBookUsers0' 2022-02-08 00:15:05,601 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:05,604 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:05,604 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:06,414 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:06,553 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:06,996 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 152525 == 0 2022-02-08 00:15:06,997 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 152525 == 0. Received: {'172.23.121.129': 152525} for bucket 'GleamBookUsers0' 2022-02-08 00:15:06,997 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:07,000 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:07,000 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:07,079 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 95119 == 0 2022-02-08 00:15:07,079 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 95119 == 0. Received: {'172.23.121.126': 95119} for bucket 'GleamBookUsers0' 2022-02-08 00:15:07,079 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:07,082 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:07,082 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:08,684 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:09,096 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 62806 == 0 2022-02-08 00:15:09,098 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 62806 == 0. Received: {'172.23.121.127': 62806} for bucket 'GleamBookUsers0' 2022-02-08 00:15:09,098 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:09,099 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:09,099 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:09,267 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:09,489 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:09,694 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:09,951 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 228542 == 0 2022-02-08 00:15:09,951 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 228542 == 0. Received: {'172.23.121.115': 228542} for bucket 'GleamBookUsers0' 2022-02-08 00:15:09,951 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:09,953 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:09,954 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:09,959 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 148156 == 0 2022-02-08 00:15:09,959 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 148156 == 0. Received: {'172.23.121.128': 148156} for bucket 'GleamBookUsers0' 2022-02-08 00:15:09,959 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:09,960 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:09,960 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:09,979 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:10,381 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 22082 == 0 2022-02-08 00:15:10,382 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 22082 == 0. Received: {'172.23.121.124': 22082} for bucket 'GleamBookUsers0' 2022-02-08 00:15:10,382 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:10,384 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:10,384 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:10,447 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 286509 == 0 2022-02-08 00:15:10,447 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 286509 == 0. Received: {'172.23.120.170': 286509} for bucket 'GleamBookUsers0' 2022-02-08 00:15:10,447 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:10,448 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:10,448 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:10,604 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:11,117 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 168165 == 0 2022-02-08 00:15:11,117 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 168165 == 0. Received: {'172.23.121.123': 168165} for bucket 'GleamBookUsers0' 2022-02-08 00:15:11,118 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:11,118 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:11,119 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:12,000 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:12,082 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:12,561 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 143287 == 0 2022-02-08 00:15:12,561 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 143287 == 0. Received: {'172.23.121.129': 143287} for bucket 'GleamBookUsers0' 2022-02-08 00:15:12,561 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:12,563 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:12,563 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:12,612 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 88889 == 0 2022-02-08 00:15:12,612 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 88889 == 0. Received: {'172.23.121.126': 88889} for bucket 'GleamBookUsers0' 2022-02-08 00:15:12,612 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:12,615 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:12,615 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:14,101 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:14,510 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 59958 == 0 2022-02-08 00:15:14,510 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 59958 == 0. Received: {'172.23.121.127': 59958} for bucket 'GleamBookUsers0' 2022-02-08 00:15:14,510 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:14,513 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:14,513 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:14,954 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:14,961 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:15,384 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 122376 == 0 2022-02-08 00:15:15,385 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 122376 == 0. Received: {'172.23.121.128': 122376} for bucket 'GleamBookUsers0' 2022-02-08 00:15:15,385 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:15,387 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:15,388 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:15,388 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:15,450 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:15,578 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 212312 == 0 2022-02-08 00:15:15,578 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 212312 == 0. Received: {'172.23.121.115': 212312} for bucket 'GleamBookUsers0' 2022-02-08 00:15:15,578 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:15,581 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:15,581 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:15,766 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 12900 == 0 2022-02-08 00:15:15,766 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 12900 == 0. Received: {'172.23.121.124': 12900} for bucket 'GleamBookUsers0' 2022-02-08 00:15:15,767 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:15,769 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:15,769 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:16,121 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:16,154 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 274323 == 0 2022-02-08 00:15:16,154 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 274323 == 0. Received: {'172.23.120.170': 274323} for bucket 'GleamBookUsers0' 2022-02-08 00:15:16,155 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:16,157 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:16,157 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:16,634 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 149473 == 0 2022-02-08 00:15:16,634 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 149473 == 0. Received: {'172.23.121.123': 149473} for bucket 'GleamBookUsers0' 2022-02-08 00:15:16,634 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:16,637 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:16,637 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:17,565 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:17,615 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:18,085 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 74163 == 0 2022-02-08 00:15:18,085 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 74163 == 0. Received: {'172.23.121.126': 74163} for bucket 'GleamBookUsers0' 2022-02-08 00:15:18,085 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:18,088 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:18,088 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:18,101 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 116173 == 0 2022-02-08 00:15:18,101 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 116173 == 0. Received: {'172.23.121.129': 116173} for bucket 'GleamBookUsers0' 2022-02-08 00:15:18,101 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:18,102 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:18,102 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:19,513 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:19,911 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 37812 == 0 2022-02-08 00:15:19,911 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 37812 == 0. Received: {'172.23.121.127': 37812} for bucket 'GleamBookUsers0' 2022-02-08 00:15:19,911 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:19,913 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:19,913 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:20,388 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:20,582 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:20,773 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:20,813 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 114142 == 0 2022-02-08 00:15:20,815 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 114142 == 0. Received: {'172.23.121.128': 114142} for bucket 'GleamBookUsers0' 2022-02-08 00:15:20,815 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:20,816 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:20,818 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:21,157 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:21,165 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 7790 == 0 2022-02-08 00:15:21,165 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 7790 == 0. Received: {'172.23.121.124': 7790} for bucket 'GleamBookUsers0' 2022-02-08 00:15:21,167 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:21,168 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:21,168 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:21,194 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 177406 == 0 2022-02-08 00:15:21,194 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 177406 == 0. Received: {'172.23.121.115': 177406} for bucket 'GleamBookUsers0' 2022-02-08 00:15:21,194 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:21,196 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:21,196 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:21,638 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:21,841 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 247993 == 0 2022-02-08 00:15:21,842 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 247993 == 0. Received: {'172.23.120.170': 247993} for bucket 'GleamBookUsers0' 2022-02-08 00:15:21,842 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:21,844 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:21,844 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:22,121 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 123881 == 0 2022-02-08 00:15:22,121 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 123881 == 0. Received: {'172.23.121.123': 123881} for bucket 'GleamBookUsers0' 2022-02-08 00:15:22,121 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:22,124 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:22,124 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:23,092 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:23,102 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:23,492 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 61233 == 0 2022-02-08 00:15:23,493 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 61233 == 0. Received: {'172.23.121.126': 61233} for bucket 'GleamBookUsers0' 2022-02-08 00:15:23,493 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:23,494 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:23,494 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:23,555 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 96848 == 0 2022-02-08 00:15:23,555 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 96848 == 0. Received: {'172.23.121.129': 96848} for bucket 'GleamBookUsers0' 2022-02-08 00:15:23,555 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:23,556 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:23,556 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:24,914 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:25,286 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 24584 == 0 2022-02-08 00:15:25,288 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 24584 == 0. Received: {'172.23.121.127': 24584} for bucket 'GleamBookUsers0' 2022-02-08 00:15:25,288 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:25,289 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:25,289 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:25,818 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:26,168 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:26,197 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:26,227 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 91926 == 0 2022-02-08 00:15:26,229 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 91926 == 0. Received: {'172.23.121.128': 91926} for bucket 'GleamBookUsers0' 2022-02-08 00:15:26,229 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:26,230 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:26,230 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:26,525 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 622 == 0 2022-02-08 00:15:26,526 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 622 == 0. Received: {'172.23.121.124': 622} for bucket 'GleamBookUsers0' 2022-02-08 00:15:26,526 | infra | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:26,528 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:26,528 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:26,740 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 148072 == 0 2022-02-08 00:15:26,740 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 148072 == 0. Received: {'172.23.121.115': 148072} for bucket 'GleamBookUsers0' 2022-02-08 00:15:26,740 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:26,743 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:26,743 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:26,845 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:27,125 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:27,568 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 218891 == 0 2022-02-08 00:15:27,568 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 218891 == 0. Received: {'172.23.120.170': 218891} for bucket 'GleamBookUsers0' 2022-02-08 00:15:27,568 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:27,569 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:27,569 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:27,598 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 100269 == 0 2022-02-08 00:15:27,599 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 100269 == 0. Received: {'172.23.121.123': 100269} for bucket 'GleamBookUsers0' 2022-02-08 00:15:27,599 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:27,601 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:27,601 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:28,496 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:28,558 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:28,885 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 49152 == 0 2022-02-08 00:15:28,885 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 49152 == 0. Received: {'172.23.121.126': 49152} for bucket 'GleamBookUsers0' 2022-02-08 00:15:28,885 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:28,888 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:28,888 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:29,003 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 73634 == 0 2022-02-08 00:15:29,003 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 73634 == 0. Received: {'172.23.121.129': 73634} for bucket 'GleamBookUsers0' 2022-02-08 00:15:29,003 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:29,006 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:29,006 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:30,289 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:30,677 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 16000 == 0 2022-02-08 00:15:30,677 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 16000 == 0. Received: {'172.23.121.127': 16000} for bucket 'GleamBookUsers0' 2022-02-08 00:15:30,678 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:30,680 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:30,680 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:31,230 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:31,528 | infra | DEBUG | pool-3-thread-23 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:31,638 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 71380 == 0 2022-02-08 00:15:31,638 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 71380 == 0. Received: {'172.23.121.128': 71380} for bucket 'GleamBookUsers0' 2022-02-08 00:15:31,638 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:31,640 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:31,640 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:31,743 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:31,921 | test | DEBUG | pool-3-thread-23 | [task:_compare:2755] Comparing 0 == 0 2022-02-08 00:15:31,921 | test | DEBUG | pool-3-thread-23 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.124': 0} for bucket 'GleamBookUsers0' 2022-02-08 00:15:31,921 | infra | INFO | pool-3-thread-23 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307509.16' completed 2022-02-08 00:15:32,269 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 120290 == 0 2022-02-08 00:15:32,269 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 120290 == 0. Received: {'172.23.121.115': 120290} for bucket 'GleamBookUsers0' 2022-02-08 00:15:32,269 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:32,272 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:32,272 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:32,571 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:32,601 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:33,039 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 78779 == 0 2022-02-08 00:15:33,039 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 78779 == 0. Received: {'172.23.121.123': 78779} for bucket 'GleamBookUsers0' 2022-02-08 00:15:33,039 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:33,042 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:33,042 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:33,246 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 183057 == 0 2022-02-08 00:15:33,246 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 183057 == 0. Received: {'172.23.120.170': 183057} for bucket 'GleamBookUsers0' 2022-02-08 00:15:33,246 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:33,249 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:33,249 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:33,891 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:34,006 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:34,267 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 37868 == 0 2022-02-08 00:15:34,269 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 37868 == 0. Received: {'172.23.121.126': 37868} for bucket 'GleamBookUsers0' 2022-02-08 00:15:34,269 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:34,273 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:34,275 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:34,436 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 55369 == 0 2022-02-08 00:15:34,436 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 55369 == 0. Received: {'172.23.121.129': 55369} for bucket 'GleamBookUsers0' 2022-02-08 00:15:34,436 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:34,437 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:34,437 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:35,680 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:36,052 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 8482 == 0 2022-02-08 00:15:36,052 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 8482 == 0. Received: {'172.23.121.127': 8482} for bucket 'GleamBookUsers0' 2022-02-08 00:15:36,052 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:36,053 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:36,053 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:36,645 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:37,049 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 52674 == 0 2022-02-08 00:15:37,051 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 52674 == 0. Received: {'172.23.121.128': 52674} for bucket 'GleamBookUsers0' 2022-02-08 00:15:37,051 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:37,052 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:37,052 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:37,273 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:37,828 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 103458 == 0 2022-02-08 00:15:37,828 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 103458 == 0. Received: {'172.23.121.115': 103458} for bucket 'GleamBookUsers0' 2022-02-08 00:15:37,828 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:37,829 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:37,829 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:38,042 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:38,249 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:38,467 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 64099 == 0 2022-02-08 00:15:38,467 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 64099 == 0. Received: {'172.23.121.123': 64099} for bucket 'GleamBookUsers0' 2022-02-08 00:15:38,467 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:38,470 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:38,470 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:38,832 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 147826 == 0 2022-02-08 00:15:38,834 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 147826 == 0. Received: {'172.23.120.170': 147826} for bucket 'GleamBookUsers0' 2022-02-08 00:15:38,834 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:38,835 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:38,835 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:39,275 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:39,438 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:39,651 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 30528 == 0 2022-02-08 00:15:39,651 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 30528 == 0. Received: {'172.23.121.126': 30528} for bucket 'GleamBookUsers0' 2022-02-08 00:15:39,651 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:39,654 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:39,654 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:39,862 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 39349 == 0 2022-02-08 00:15:39,862 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 39349 == 0. Received: {'172.23.121.129': 39349} for bucket 'GleamBookUsers0' 2022-02-08 00:15:39,864 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:39,865 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:39,865 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:41,055 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:41,430 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 686 == 0 2022-02-08 00:15:41,430 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 686 == 0. Received: {'172.23.121.127': 686} for bucket 'GleamBookUsers0' 2022-02-08 00:15:41,430 | infra | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:41,433 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:41,433 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:42,052 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:42,430 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 30408 == 0 2022-02-08 00:15:42,430 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 30408 == 0. Received: {'172.23.121.128': 30408} for bucket 'GleamBookUsers0' 2022-02-08 00:15:42,430 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:42,433 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:42,433 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:42,831 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:43,328 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 72769 == 0 2022-02-08 00:15:43,328 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 72769 == 0. Received: {'172.23.121.115': 72769} for bucket 'GleamBookUsers0' 2022-02-08 00:15:43,328 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:43,331 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:43,331 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:43,470 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:43,838 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:43,887 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 50599 == 0 2022-02-08 00:15:43,888 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 50599 == 0. Received: {'172.23.121.123': 50599} for bucket 'GleamBookUsers0' 2022-02-08 00:15:43,888 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:43,890 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:43,890 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:44,362 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 112172 == 0 2022-02-08 00:15:44,364 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 112172 == 0. Received: {'172.23.120.170': 112172} for bucket 'GleamBookUsers0' 2022-02-08 00:15:44,364 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:44,365 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:44,365 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:44,654 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:44,865 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:45,029 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 23488 == 0 2022-02-08 00:15:45,029 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 23488 == 0. Received: {'172.23.121.126': 23488} for bucket 'GleamBookUsers0' 2022-02-08 00:15:45,029 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:45,030 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:45,030 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:45,283 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 10882 == 0 2022-02-08 00:15:45,285 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 10882 == 0. Received: {'172.23.121.129': 10882} for bucket 'GleamBookUsers0' 2022-02-08 00:15:45,285 | infra | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:45,286 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:45,286 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:46,440 | infra | DEBUG | pool-3-thread-29 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:46,792 | test | DEBUG | pool-3-thread-29 | [task:_compare:2755] Comparing 0 == 0 2022-02-08 00:15:46,792 | test | DEBUG | pool-3-thread-29 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.127': 0} for bucket 'GleamBookUsers0' 2022-02-08 00:15:46,793 | infra | INFO | pool-3-thread-29 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307512.01' completed 2022-02-08 00:15:47,433 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:47,786 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 7828 == 0 2022-02-08 00:15:47,786 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 7828 == 0. Received: {'172.23.121.128': 7828} for bucket 'GleamBookUsers0' 2022-02-08 00:15:47,786 | infra | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:47,789 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:47,789 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:48,331 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:48,773 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 41021 == 0 2022-02-08 00:15:48,773 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 41021 == 0. Received: {'172.23.121.115': 41021} for bucket 'GleamBookUsers0' 2022-02-08 00:15:48,773 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:48,776 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:48,776 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:48,890 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:49,296 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 34467 == 0 2022-02-08 00:15:49,296 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 34467 == 0. Received: {'172.23.121.123': 34467} for bucket 'GleamBookUsers0' 2022-02-08 00:15:49,296 | infra | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:49,299 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:49,299 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:49,365 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:49,884 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 83316 == 0 2022-02-08 00:15:49,884 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 83316 == 0. Received: {'172.23.120.170': 83316} for bucket 'GleamBookUsers0' 2022-02-08 00:15:49,884 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:49,887 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:49,887 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:50,032 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:50,286 | infra | DEBUG | pool-3-thread-3 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:50,407 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 3880 == 0 2022-02-08 00:15:50,407 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 3880 == 0. Received: {'172.23.121.126': 3880} for bucket 'GleamBookUsers0' 2022-02-08 00:15:50,407 | infra | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:50,410 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:50,410 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:50,635 | test | DEBUG | pool-3-thread-3 | [task:_compare:2755] Comparing 0 == 0 2022-02-08 00:15:50,635 | test | DEBUG | pool-3-thread-3 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.129': 0} for bucket 'GleamBookUsers0' 2022-02-08 00:15:50,637 | infra | INFO | pool-3-thread-3 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307514.87' completed 2022-02-08 00:15:52,789 | infra | DEBUG | pool-3-thread-30 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:53,142 | test | DEBUG | pool-3-thread-30 | [task:_compare:2755] Comparing 0 == 0 2022-02-08 00:15:53,144 | test | DEBUG | pool-3-thread-30 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.128': 0} for bucket 'GleamBookUsers0' 2022-02-08 00:15:53,144 | infra | INFO | pool-3-thread-30 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307513.44' completed 2022-02-08 00:15:53,778 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:54,144 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 1904 == 0 2022-02-08 00:15:54,144 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1904 == 0. Received: {'172.23.121.115': 1904} for bucket 'GleamBookUsers0' 2022-02-08 00:15:54,144 | infra | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:54,145 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:54,145 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:54,299 | infra | DEBUG | pool-3-thread-24 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:54,667 | test | DEBUG | pool-3-thread-24 | [task:_compare:2755] Comparing 0 == 0 2022-02-08 00:15:54,667 | test | DEBUG | pool-3-thread-24 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.123': 0} for bucket 'GleamBookUsers0' 2022-02-08 00:15:54,667 | infra | INFO | pool-3-thread-24 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307507.73' completed 2022-02-08 00:15:54,887 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:55,244 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 1356 == 0 2022-02-08 00:15:55,246 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2737] Not Ready: num_items_for_persistence 1356 == 0. Received: {'172.23.120.170': 1356} for bucket 'GleamBookUsers0' 2022-02-08 00:15:55,246 | infra | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2741] Wait before next StatWaitTask check 2022-02-08 00:15:55,247 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2742 2022-02-08 00:15:55,247 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:15:55,410 | infra | DEBUG | pool-3-thread-28 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:55,760 | test | DEBUG | pool-3-thread-28 | [task:_compare:2755] Comparing 0 == 0 2022-02-08 00:15:55,762 | test | DEBUG | pool-3-thread-28 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.126': 0} for bucket 'GleamBookUsers0' 2022-02-08 00:15:55,762 | infra | INFO | pool-3-thread-28 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307510.58' completed 2022-02-08 00:15:59,147 | infra | DEBUG | pool-3-thread-27 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:15:59,496 | test | DEBUG | pool-3-thread-27 | [task:_compare:2755] Comparing 0 == 0 2022-02-08 00:15:59,496 | test | DEBUG | pool-3-thread-27 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.121.115': 0} for bucket 'GleamBookUsers0' 2022-02-08 00:15:59,496 | infra | INFO | pool-3-thread-27 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307505.93' completed 2022-02-08 00:16:00,249 | infra | DEBUG | pool-3-thread-26 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2022-02-08 00:16:00,592 | test | DEBUG | pool-3-thread-26 | [task:_compare:2755] Comparing 0 == 0 2022-02-08 00:16:00,592 | test | DEBUG | pool-3-thread-26 | [task:_get_checkpoint_stats_and_compare:2744] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.120.170': 0} for bucket 'GleamBookUsers0' 2022-02-08 00:16:00,592 | infra | INFO | pool-3-thread-26 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307505.37' completed 2022-02-08 00:16:00,595 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:16:00,608 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307505.93 2022-02-08 00:16:00,609 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:16:00,609 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307506.4 2022-02-08 00:16:00,611 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:16:00,611 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307507.73 2022-02-08 00:16:00,611 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:16:00,611 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307509.16 2022-02-08 00:16:00,611 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:16:00,611 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307510.58 2022-02-08 00:16:00,612 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:16:00,612 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307512.01 2022-02-08 00:16:00,612 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:16:00,614 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307513.44 2022-02-08 00:16:00,614 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:16:00,614 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1644307514.87 2022-02-08 00:16:00,614 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:16:00,690 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:651] All nodes in cluster: [ip:172.23.121.128 port:8091 ssh_username:root, ip:172.23.121.115 port:8091 ssh_username:root, ip:172.23.120.170 port:8091 ssh_username:root, ip:172.23.121.129 port:8091 ssh_username:root, ip:172.23.121.124 port:8091 ssh_username:root, ip:172.23.121.126 port:8091 ssh_username:root, ip:172.23.121.116 port:8091 ssh_username:root, ip:172.23.121.127 port:8091 ssh_username:root, ip:172.23.121.123 port:8091 ssh_username:root] 2022-02-08 00:16:00,698 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:16:00,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:00,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:16:01,049 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:01,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:16:01,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:16:01,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:16:01,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:16:01,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:16:01,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:16:01,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:16:01,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-08 00:16:01,335 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:16:01,338 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:16:01,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:01,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:16:01,690 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:01,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:16:01,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:16:01,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:16:01,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:16:01,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:16:01,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:16:01,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:16:01,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-08 00:16:01,970 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:16:01,971 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:16:02,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:02,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:16:02,319 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:02,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:16:02,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:16:02,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:16:02,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:16:02,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:16:02,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:16:02,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:16:02,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-08 00:16:02,598 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:16:02,601 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:16:02,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:02,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:16:02,959 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:02,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:16:02,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:16:02,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:16:02,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:16:03,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:16:03,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:16:03,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:16:03,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-08 00:16:03,234 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:16:03,236 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:16:03,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:03,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:16:03,621 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:03,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:16:03,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:16:03,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:16:03,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:16:03,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:16:03,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:16:03,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:16:03,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-08 00:16:03,926 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:16:03,940 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:16:04,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:04,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:16:04,282 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:04,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:16:04,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:16:04,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:16:04,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:16:04,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:16:04,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:16:04,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:16:04,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-08 00:16:04,559 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:16:04,562 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:16:04,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:04,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:16:04,887 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:04,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:16:04,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:16:04,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:16:04,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:16:04,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:16:04,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:16:04,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:16:04,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-08 00:16:05,170 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:16:05,173 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:16:05,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:05,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:16:05,502 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:05,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:16:05,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:16:05,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:16:05,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:16:05,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:16:05,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:16:05,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:16:05,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-08 00:16:05,779 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:16:05,782 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:16:05,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:06,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:16:06,125 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:06,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:16:06,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:16:06,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:16:06,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:16:06,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:16:06,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:16:06,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:16:06,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 vbucket-seqno 2022-02-08 00:16:06,394 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:16:06,394 | test | DEBUG | MainThread | [bucket_ready_functions:verify_stats_for_bucket:2318] Verifying stats for bucket GleamBookUsers0 2022-02-08 00:16:06,473 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:651] All nodes in cluster: [ip:172.23.121.128 port:8091 ssh_username:root, ip:172.23.121.115 port:8091 ssh_username:root, ip:172.23.120.170 port:8091 ssh_username:root, ip:172.23.121.129 port:8091 ssh_username:root, ip:172.23.121.124 port:8091 ssh_username:root, ip:172.23.121.126 port:8091 ssh_username:root, ip:172.23.121.116 port:8091 ssh_username:root, ip:172.23.121.127 port:8091 ssh_username:root, ip:172.23.121.123 port:8091 ssh_username:root] 2022-02-08 00:16:06,474 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:16:06,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:06,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:16:06,812 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:06,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:16:06,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:16:06,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:16:06,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:16:06,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:16:06,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:16:06,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:16:06,904 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:16:06,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:07,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:16:07,233 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:07,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:16:07,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:16:07,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:16:07,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:16:07,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:16:07,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:16:07,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:16:07,328 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:16:07,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:07,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:16:07,686 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:07,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:16:07,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:16:07,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:16:07,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:16:07,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:16:07,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:16:07,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:16:07,783 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:16:07,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:08,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:16:08,118 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:08,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:16:08,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:16:08,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:16:08,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:16:08,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:16:08,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:16:08,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:16:08,213 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:16:08,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:08,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:16:08,553 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:08,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:16:08,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:16:08,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:16:08,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:16:08,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:16:08,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:16:08,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:16:08,654 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:16:08,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:08,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:16:08,980 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:08,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:16:08,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:16:09,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:16:09,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:16:09,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:16:09,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:16:09,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:16:09,082 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:16:09,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:09,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:16:09,437 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:09,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:16:09,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:16:09,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:16:09,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:16:09,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:16:09,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:16:09,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:16:09,529 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:16:09,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:09,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:16:09,874 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:09,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:16:09,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:16:09,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:16:09,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:16:09,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:16:09,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:16:09,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:16:09,976 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:16:10,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:10,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:16:10,339 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:10,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:16:10,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:16:10,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:16:10,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:16:10,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:16:10,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:16:10,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:16:10,447 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:16:10,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:10,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:16:10,795 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:10,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:16:10,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:16:10,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:16:10,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:16:10,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:16:10,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:16:10,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:16:10,914 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:16:10,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:11,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:16:11,243 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:11,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:16:11,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:16:11,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:16:11,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:16:11,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:16:11,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:16:11,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:16:11,335 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:16:11,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:11,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:16:11,703 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:11,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:16:11,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:16:11,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:16:11,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:16:11,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:16:11,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:16:11,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:16:11,799 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:16:11,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:12,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:16:12,124 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:12,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:16:12,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:16:12,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:16:12,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:16:12,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:16:12,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:16:12,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:16:12,213 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:16:12,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:12,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:16:12,525 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:12,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:16:12,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:16:12,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:16:12,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:16:12,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:16:12,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:16:12,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:16:12,619 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:16:12,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:12,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:16:12,950 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:12,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:16:12,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:16:12,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:16:12,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:16:13,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:16:13,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:16:13,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:16:13,040 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:16:13,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:13,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:16:13,348 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:13,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:16:13,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:16:13,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:16:13,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:16:13,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:16:13,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:16:13,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:16:13,434 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:16:13,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:13,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:16:13,762 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:13,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:16:13,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:16:13,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:16:13,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:16:13,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:16:13,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:16:13,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:16:13,855 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:16:13,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:14,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:16:14,181 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:14,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:16:14,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:16:14,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:16:14,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:16:14,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:16:14,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:16:14,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:16:14,269 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for vb_replica_curr_items on bucket GleamBookUsers0 2022-02-08 00:16:14,269 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_vb_replica_curr_items_1644308174.27 2022-02-08 00:16:14,269 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for curr_items_tot on bucket GleamBookUsers0 2022-02-08 00:16:14,269 | infra | INFO | pool-3-thread-2 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_vb_replica_curr_items_1644308174.27' started 2022-02-08 00:16:14,269 | infra | INFO | pool-3-thread-4 | [task:start_task:91] Thread 'StatsWaitTask_GleamBookUsers0_curr_items_tot_1644308174.27' started 2022-02-08 00:16:14,269 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_curr_items_tot_1644308174.27 2022-02-08 00:16:14,273 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:14,273 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_vb_replica_curr_items_1644308174.27 2022-02-08 00:16:14,273 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:14,411 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:14,414 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:14,553 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:14,559 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:14,690 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:14,690 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:14,832 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:14,839 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:14,974 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:14,974 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:15,115 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:15,118 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:15,250 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:15,253 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:15,388 | infra | DEBUG | pool-3-thread-4 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:15,391 | infra | DEBUG | pool-3-thread-2 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:16:15,525 | test | DEBUG | pool-3-thread-2 | [task:_compare:2755] Comparing 40000000 == 40000000 2022-02-08 00:16:15,525 | test | DEBUG | pool-3-thread-2 | [task:_get_all_stats_and_compare:2707] Ready: vb_replica_curr_items 40000000 == 40000000. Received: {'172.23.121.123': '4447200', '172.23.121.124': '4462540', '172.23.121.115': '4450040', '172.23.121.126': '4445220', '172.23.121.116': '4457960', '172.23.121.127': '4477040', '172.23.121.128': '4398720', '172.23.121.129': '4408860', '172.23.120.170': '4452420'} for bucket 'GleamBookUsers0' 2022-02-08 00:16:15,525 | infra | INFO | pool-3-thread-2 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_vb_replica_curr_items_1644308174.27' completed 2022-02-08 00:16:15,526 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_curr_items_tot_1644308174.27 2022-02-08 00:16:15,529 | test | DEBUG | pool-3-thread-4 | [task:_compare:2755] Comparing 80000000 == 80000000 2022-02-08 00:16:15,529 | test | DEBUG | pool-3-thread-4 | [task:_get_all_stats_and_compare:2707] Ready: curr_items_tot 80000000 == 80000000. Received: {'172.23.121.123': '8912760', '172.23.121.124': '8908380', '172.23.121.115': '8893920', '172.23.121.126': '8891820', '172.23.121.116': '8918580', '172.23.121.127': '8932540', '172.23.121.128': '8807560', '172.23.121.129': '8819600', '172.23.120.170': '8914840'} for bucket 'GleamBookUsers0' 2022-02-08 00:16:15,529 | infra | INFO | pool-3-thread-4 | [task:complete_task:106] Thread 'StatsWaitTask_GleamBookUsers0_curr_items_tot_1644308174.27' completed 2022-02-08 00:16:15,529 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:16:15,529 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:16:15,529 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:16:15,529 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:16:15,530 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:16:15,530 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:16:15,530 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:16:15,530 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:16:15,530 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:16:15,530 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:16:15,532 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:16:15,532 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:16:15,532 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:16:15,532 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:16:15,532 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:16:15,532 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:16:15,532 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:16:15,532 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:16:15,608 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3970] Stats: {u'opsPerSec': 0, u'diskFetches': 0, u'dataUsed': 49487698381L, u'itemCount': 40000000, u'quotaPercentUsed': 73.8811032797332, u'storageTotals': {u'hdd': {u'usedByData': 85082897282L, u'used': 179077636786L, u'quotaTotal': 1168799956992L, u'total': 1168799956992L, u'free': 866228006340L}, u'ram': {u'usedByData': 57528411472L, u'used': 90990505984L, u'quotaTotal': 78809923584L, u'total': 110647066624L, u'quotaUsed': 77866205184L, u'quotaUsedPerNode': 8651800576L, u'quotaTotalPerNode': 8756658176L}}, u'diskUsed': 85082897282L, u'vbActiveNumNonResident': 0, u'memUsed': 57528411472L} 2022-02-08 00:16:15,868 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+-----+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 40000000 | 72.52 GiB | 53.58 GiB | 79.24 GiB | 100 | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+-----+ 2022-02-08 00:16:15,869 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+-----------------+---------------+---------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+-----------------+---------------+---------------+---------------+ | 40000000 | 40000000 | 1000000-2000000 | 0-0 | 0-0 | 0-0 | +---------------+---------------+-----------------+---------------+---------------+---------------+ 2022-02-08 00:16:16,135 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 100 2022-02-08 00:16:16,138 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:16:16,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:16,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:16:16,469 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:16,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:16:16,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:16:16,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:16:16,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:16:16,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:16:16,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:16:16,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:16:16,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:16:16,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:16:16,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:16:16,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:16:16,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:16:16,931 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:16:16,934 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:16:17,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:17,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:16:17,263 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:17,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:16:17,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:16:17,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:16:17,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:16:17,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:16:17,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:16:17,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:16:17,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:16:17,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:16:17,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:16:17,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:16:17,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:16:17,710 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:16:17,713 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:16:17,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:17,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:16:18,061 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:18,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:16:18,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:16:18,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:16:18,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:16:18,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:16:18,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:16:18,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:16:18,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:16:18,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:16:18,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:16:18,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:16:18,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:16:18,641 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:16:18,644 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:16:18,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:18,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:16:19,002 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:19,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:16:19,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:16:19,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:16:19,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:16:19,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:16:19,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:16:19,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:16:19,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:16:19,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:16:19,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:16:19,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:16:19,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:16:19,457 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:16:19,460 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:16:19,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:19,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:16:19,805 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:19,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:16:19,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:16:19,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:16:19,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:16:19,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:16:19,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:16:19,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:16:19,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:16:20,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:16:20,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:16:20,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:16:20,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:16:20,295 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:16:20,296 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:16:20,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:20,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:16:20,677 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:20,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:16:20,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:16:20,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:16:20,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:16:20,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:16:20,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:16:20,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:16:20,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:16:20,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:16:20,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:16:20,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:16:21,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:16:21,177 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:16:21,180 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:16:21,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:21,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:16:21,552 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:21,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:16:21,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:16:21,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:16:21,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:16:21,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:16:21,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:16:21,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:16:21,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:16:21,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:16:21,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:16:21,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:16:21,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:16:22,007 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:16:22,012 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:16:22,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:22,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:16:22,345 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:22,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:16:22,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:16:22,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:16:22,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:16:22,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:16:22,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:16:22,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:16:22,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:16:22,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:16:22,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:16:22,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:16:22,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:16:22,769 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:16:22,773 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:16:22,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:23,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:16:23,111 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:23,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:16:23,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:16:23,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:16:23,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:16:23,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:16:23,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:16:23,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:16:23,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:16:23,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:16:23,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:16:23,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:16:23,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:16:23,579 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:16:23,582 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 00:16:23,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:23,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 00:16:23,964 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:23,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 00:16:23,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 00:16:23,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 00:16:24,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 00:16:24,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 00:16:24,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 00:16:24,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 00:16:24,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:16:24,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:16:24,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:16:24,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:16:24,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:16:24,157 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 00:16:24,158 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 00:16:24,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:24,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 00:16:24,522 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:24,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 00:16:24,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 00:16:24,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 00:16:24,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 00:16:24,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 00:16:24,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 00:16:24,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 00:16:24,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:16:24,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:16:24,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:16:24,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:16:24,711 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:16:24,724 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 00:16:24,727 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 00:16:24,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:25,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 00:16:25,109 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:25,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 00:16:25,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 00:16:25,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 00:16:25,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 00:16:25,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 00:16:25,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 00:16:25,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 00:16:25,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:16:25,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:16:25,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:16:25,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:16:25,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:16:25,289 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 00:16:25,292 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 00:16:25,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:25,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 00:16:25,628 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:25,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 00:16:25,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 00:16:25,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 00:16:25,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 00:16:25,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 00:16:25,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 00:16:25,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 00:16:25,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:16:25,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:16:25,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:16:25,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:16:25,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:16:25,864 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 00:16:25,865 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 00:16:25,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:26,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 00:16:26,203 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:26,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 00:16:26,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 00:16:26,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 00:16:26,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 00:16:26,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 00:16:26,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 00:16:26,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 00:16:26,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:16:26,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:16:26,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:16:26,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:16:26,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:16:26,377 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 00:16:26,377 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for kvstore is 77824MB 2022-02-08 00:16:26,377 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for wal is 4059MB 2022-02-08 00:16:26,378 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:642] Total Disk usage for keyTree is 5039MB 2022-02-08 00:16:26,378 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:643] Total Disk usage for seqTree is 72685MB 2022-02-08 00:16:26,381 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:16:26,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:26,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:16:26,721 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:26,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:16:26,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:16:26,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:16:26,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:16:26,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:16:26,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:16:26,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:16:26,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:16:26,829 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.115 - core(s): 8 2022-02-08 00:16:26,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:16:26,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:16:27,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:16:27,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:16:27,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:16:27,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:16:27,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:16:27,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:16:27,907 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:16:27,910 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:16:27,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:28,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:16:28,239 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:28,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:16:28,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:16:28,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:16:28,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:16:28,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:16:28,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:16:28,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:16:28,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:16:28,351 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.116 - core(s): 8 2022-02-08 00:16:28,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:16:28,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:16:28,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:16:28,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:16:28,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:16:29,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:16:29,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:16:29,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:16:29,430 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:16:29,433 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:16:29,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:29,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:16:29,782 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:29,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:16:29,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:16:29,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:16:29,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:16:29,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:16:29,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:16:29,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:16:29,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:16:29,930 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.123 - core(s): 8 2022-02-08 00:16:29,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:16:30,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:16:30,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:16:30,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:16:30,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:16:30,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:16:30,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:16:30,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:16:31,016 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:16:31,019 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:16:31,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:31,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:16:31,362 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:31,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:16:31,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:16:31,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:16:31,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:16:31,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:16:31,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:16:31,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:16:31,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:16:31,482 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.124 - core(s): 8 2022-02-08 00:16:31,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:16:31,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:16:31,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:16:31,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:16:32,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:16:32,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:16:32,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:16:32,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:16:32,563 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:16:32,565 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:16:32,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:32,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:16:32,901 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:32,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:16:32,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:16:32,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:16:32,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:16:32,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:16:32,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:16:32,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:16:33,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:16:33,033 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.126 - core(s): 8 2022-02-08 00:16:33,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:16:33,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:16:33,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:16:33,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:16:33,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:16:33,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:16:33,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:16:33,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:16:34,121 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:16:34,125 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:16:34,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:34,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:16:34,437 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:34,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:16:34,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:16:34,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:16:34,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:16:34,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:16:34,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:16:34,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:16:34,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:16:34,545 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.127 - core(s): 8 2022-02-08 00:16:34,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:16:34,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:16:34,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:16:34,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:16:35,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:16:35,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:16:35,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:16:35,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:16:35,612 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:16:35,615 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:16:35,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:35,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:16:35,950 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:35,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:16:35,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:16:35,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:16:35,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:16:36,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:16:36,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:16:36,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:16:36,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:16:36,062 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.128 - core(s): 8 2022-02-08 00:16:36,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:16:36,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:16:36,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:16:36,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:16:36,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:16:36,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:16:36,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:16:36,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:16:37,104 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:16:37,107 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:16:37,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:37,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:16:37,447 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:37,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:16:37,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:16:37,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:16:37,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:16:37,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:16:37,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:16:37,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:16:37,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:16:37,611 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.129 - core(s): 8 2022-02-08 00:16:37,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:16:37,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:16:37,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:16:38,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:16:38,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:16:38,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:16:38,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:16:38,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:16:38,671 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:16:38,674 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:16:38,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:38,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:16:39,026 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:39,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:16:39,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:16:39,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:16:39,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:16:39,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:16:39,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:16:39,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:16:39,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:16:39,140 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.120.170 - core(s): 8 2022-02-08 00:16:39,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:16:39,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:16:39,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:16:39,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:16:39,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:16:39,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:16:39,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:16:40,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:16:40,203 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:16:40,203 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:723] {'172.23.121.115': {'rw_0:magma': {u'NReadBytesGetPerSec': 107277680.0, u'ReplayWALOffset': u'213:8531968', u'NTablesCreated': 6331, u'RecentReadIOAmpSet': 0.09341825544834137, u'ReadIOAmp': 757.57080078125, u'BlockCacheMisses': 146150, u'NFlushes': 3486, u'RecentWriteAmp': 2.8927907943725586, u'NumSyncFlushes': 0, u'TotalMemUsed': 49866297, u'NBloomFilterHits': 19712248, u'Fragmentation': 0.3586973249912262, u'NumNormalFlushes': 491, u'RecentBlockCacheHitRatio': 93.00057983398438, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 4039, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 5791528, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 7899238, u'CheckpointOverhead': 28241920, u'NItemsGet': 0, u'NWriteIOs': 13771, u'NReadBytes': 2954280280L, u'LevelStats': [{u'TargetSize': 121634816, u'LogicalSize': 29044667, u'SizeRatio': 0.2387858012626911, u'NWriteBytes': 74231808, u'BloomFilterMemUsed': 0, u'NumTables': 11, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 774144}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 25373738, u'SizeRatio': 0.2086058813950111, u'NWriteBytes': 66064384, u'BloomFilterMemUsed': 0, u'NumTables': 16, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 430080}, {u'TargetSize': 529295724, u'LogicalSize': 398371422, u'SizeRatio': 0.7526443232705957, u'NWriteBytes': 57806848, u'BloomFilterMemUsed': 0, u'NumTables': 183, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 7036928}, {u'TargetSize': 2110999008, u'LogicalSize': 1058591458, u'SizeRatio': 0.501464687566542, u'NWriteBytes': 3908960256L, u'BloomFilterMemUsed': 0, u'NumTables': 336, u'WastedSpace': 3091954, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 1058533376}], u'NItemsGarbageCollected': 2712415, u'NInternalFragmentationCompacts': 9, u'NTTLCompacts': 23, u'ActiveIndexBlocksSize': 7693441, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 54912, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 1112126, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 161304, u'CompactWriter': {u'NDataBlockWrites': 543839, u'NWriteBytes': 1619759104, u'NIndexBlockWrites': 6526, u'NBytesIncoming': 2544781164L, u'NSyncs': 0, u'NInserts': 6440277, u'NWriteIOs': 5238}, u'NonResidentBloomFilterSize': 0, u'IteratorMemUsed': 0, u'NWriteBytes': 4045846608L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 2712415, u'LogicalDiskSize': 1058591458, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.9498724578016738, u'BufferMemUsed': 296960, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 660220036, u'NTableFiles': 594, u'NFileCountCompacts': 0, u'TableObjectMemUsed': 137280, u'TreeSnapshotMemoryUsed': 177008, u'TotalDiskUsage': 1095016448, u'BytesIncoming': 4259357743L, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 1886, u'NItemsPut': 6050121, u'NSyncs': 2995, u'CompactReader': {u'NReadIOs': 1112126, u'NReadBytes': 2954280280L, u'NDataBlockReads': 1101285, u'NBytesOutgoing': 0, u'NIndexBlockReads': 10841}, u'ActiveDiskUsage': 1066774528, u'NTablesDeleted': 2933, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 923844, u'NWriteBytes': 2426087504L, u'NIndexBlockWrites': 8599, u'NBytesIncoming': 4345556092L, u'NSyncs': 2995, u'NInserts': 7728292, u'NWriteIOs': 8533}}, u'NReadBytes': 1249892630800L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 32.29999923706055, u'NInternalFragmentationCompacts': 9, u'NTTLCompacts': 212, u'ActiveIndexBlocksSize': 8613514, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 36506828, u'NTxns': 57628, u'NWriteBytes': 4508319744L, u'NSegments': 216, u'NSyncs': 57628, u'NLoggedBytes': 4508319744L, u'NActiveSegments': 4, u'NWrites': 84822, u'PreallocateSupported': True, u'NInserts': 4936558, u'DiskUsed': 65540096}, u'NGets': 548580, u'NReadBytesSet': 245398408, u'NReadBytesCompact': 3140796624L, u'IndexResidentRatio': 0.5859166383743286, u'CompactQueueSize': 0, u'NSyncsPerSec': 1.0, u'BlockCacheHitRatio': 97.30558776855469, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 1246751834176L, u'BloomFilterMemUsedRatio': 0.013242948800325394, u'NonResidentBloomFilterSize': 0, u'NWriteBytes': 9060080846L, u'BytesOutgoing': 0, u'RecentReadIOAmp': 0.0, u'WriteAmp': 2.1479451656341553, u'NBlocksCached': 10826, u'LogicalDataSize': 712234644, u'ActiveBloomFilterMemUsed': 1432192, u'NFileCountCompacts': 64, u'NTableFiles': 709, u'FlushQueueSize': 8, u'TableObjectMemUsed': 164880, u'BlockCacheQuota': 80785106, u'TotalDiskUsage': 1170264064, u'NPurgerRuns': 58, u'ReadIOAmpSet': 0.0, u'NSyncs': 8985, u'NReadIOsPerSec': 38686.83203125, u'NReadIOsSetPerSec': 38.13333511352539, u'NTablesDeleted': 5110, u'BloomFilterFPR': 0.016301730647683144, u'NBloomFilterHitsPerSec': 153.63333129882812, u'WriteCacheMemUsed': 6380920, u'RecentBloomFilterFPR': 0.0555327869951725, u'BytesPerRead': 2272241.75, u'BytesIncomingPerSec': 376904.0625, u'NReadIOsGetPerSec': 38369.265625, u'NumKVStores': 29, u'NGetStats': 20569, u'TotalIndexBlocksSize': 8819311, u'NGetStatsPerSec': 0.8333333134651184, u'NWriteBytesCompact': 1809494016, u'NReadIOsSet': 76816, u'ActiveDataSize': 712234644, u'NWriteBytesCompactPerSec': 489335.46875, u'NBloomFilterMissesPerSec': 9.033333778381348, u'NCommitBatches': 57628, u'NGetStatsComputed': 20569, u'NWriteIOs': 107454, u'NReadIOsGet': 415664998, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 20196276, u'LSMTreeObjectMemUsed': 500192, u'NSetsPerSec': 408.20001220703125, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.3333333432674408, u'ManagedFileObjectMemUsed': 65952, u'ReadAmpGet': 0.0, u'BlockCacheMemUsed': 34935112, u'NWriterCompacts': 0, u'NReadIOs': 416826792, u'WALBufferMemUsed': 5242880, u'ReadAmp': 0.0, u'keyStats': {u'NItemsTransformed': 4182699, u'NTablesCreated': 1147, u'ReadIOAmp': 81.74559333896967, u'MemtablesMemUsed': 532264, u'NBloomFilterHits': 19712248, u'TotalIndexBlocksSize': 917047, u'CheckpointOverhead': 0, u'NItemsGet': 5084859, u'NWriteIOs': 4721, u'NReadBytes': 1246937234137L, u'LevelStats': [{u'TargetSize': 121634816, u'LogicalSize': 22929785, u'SizeRatio': 0.18851333651049385, u'NWriteBytes': 305397760, u'BloomFilterMemUsed': 454912, u'NumTables': 28, u'WastedSpace': 16249, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 22929408}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 51725475, u'SizeRatio': 0.42525221561563425, u'NWriteBytes': 188678144, u'BloomFilterMemUsed': 975744, u'NumTables': 30, u'WastedSpace': 127081, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 51851264}], u'NItemsGarbageCollected': 3024846, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 92, u'ActiveIndexBlocksSize': 917047, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 5568, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 415712438, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 1686583, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 17707, u'CompactWriter': {u'NDataBlockWrites': 64556, u'NWriteBytes': 188678144, u'NIndexBlockWrites': 921, u'NBytesIncoming': 243385642, u'NSyncs': 0, u'NInserts': 4182699, u'NWriteIOs': 432}, u'NonResidentBloomFilterSize': 0, u'IteratorMemUsed': 0, u'NWriteBytes': 495507618, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 415664816, u'NReadBytes': 1246751741518L, u'NDataBlockReads': 9676023298L, u'NBytesOutgoing': 0, u'NIndexBlockReads': 18839828485L}, u'NBloomFilterMisses': 326669, u'NItemsMerged': 0, u'LogicalDiskSize': 51725475, u'TotalBloomFilterMemUsed': 1430656, u'WriteAmp': 2.184675722589565, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1430656, u'LogicalDataSize': 51725475, u'NTableFiles': 58, u'NFileCountCompacts': 32, u'TableObjectMemUsed': 13920, u'TreeSnapshotMemoryUsed': 132160, u'TotalDiskUsage': 74780672, u'BytesIncoming': 226810603, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 129, u'NItemsPut': 4536279, u'NSyncs': 2995, u'CompactReader': {u'NReadIOs': 47622, u'NReadBytes': 185492619, u'NDataBlockReads': 112303, u'NBytesOutgoing': 0, u'NIndexBlockReads': 2277}, u'ActiveDiskUsage': 74780672, u'NTablesDeleted': 1089, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.016301729280080356, u'Writer': {u'NDataBlockWrites': 71010, u'NWriteBytes': 306829474, u'NIndexBlockWrites': 1700, u'NBytesIncoming': 267306242, u'NSyncs': 2995, u'NInserts': 4530671, u'NWriteIOs': 4289}}, u'NReadBytesCompactPerSec': 910015.25, u'TableMetaMemUsed': 195633, u'NDeletes': 0, u'BlockCacheHitsPerSec': 429.1666564941406, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 326669, u'LogicalDiskSize': 1110606066, u'NSets': 4536279, u'TotalBloomFilterMemUsed': 1432192, u'BufferMemUsed': 296960, u'RecentReadAmp': 0.0, u'NGetsPerSec': 0.0, u'NInserts': 0, u'NCommitBatchesPerSec': 0.1666666716337204, u'FragmentationWithOverhead': 0.39138978719711304, u'NCompactsPerSec': 0.06666667014360428, u'WALMemUsed': 5308496, u'MemoryQuota': 108147512, u'NReadBytesPerSec': 108187688.0, u'TreeSnapshotMemoryUsed': 402256, u'BytesIncoming': 4218022292L, u'NCompacts': 2144, u'NumPersistentFlushes': 2995, u'NWriteIOsPerSec': 2.866666555404663, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 0.0, u'ActiveDiskUsage': 1142022144, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 0.8333333134651184, u'NReadBytesSetPerSec': 107003.5, u'BlockCacheHits': 5278027, u'NWriteBytesPerSec': 1090304.625, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 1145, u'ReadIOAmp': 0.2171837708830549, u'MemtablesMemUsed': 57128, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 3026, u'CheckpointOverhead': 0, u'NItemsGet': 838, u'NWriteIOs': 4140, u'NReadBytes': 1116383, u'LevelStats': [{u'TargetSize': 121634816, u'LogicalSize': 263472, u'SizeRatio': 0.002166090340449892, u'NWriteBytes': 8323072, u'BloomFilterMemUsed': 608, u'NumTables': 28, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 229376}, {u'TargetSize': 121634816, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 121634816, u'LogicalSize': 289133, u'SizeRatio': 0.0023770578976335197, u'NWriteBytes': 1056768, u'BloomFilterMemUsed': 928, u'NumTables': 29, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 237568}], u'NItemsGarbageCollected': 226369, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 97, u'ActiveIndexBlocksSize': 3026, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 5472, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 2228, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 16622, u'CompactWriter': {u'NDataBlockWrites': 129, u'NWriteBytes': 1056768, u'NIndexBlockWrites': 129, u'NBytesIncoming': 285782, u'NSyncs': 0, u'NInserts': 2967, u'NWriteIOs': 129}, u'NonResidentBloomFilterSize': 0, u'IteratorMemUsed': 0, u'NWriteBytes': 10406876, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 182, u'NReadBytes': 92658, u'NDataBlockReads': 91, u'NBytesOutgoing': 0, u'NIndexBlockReads': 91}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 289133, u'TotalBloomFilterMemUsed': 1536, u'WriteAmp': 0.3174779086037721, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1536, u'LogicalDataSize': 289133, u'NTableFiles': 57, u'NFileCountCompacts': 32, u'TableObjectMemUsed': 13680, u'TreeSnapshotMemoryUsed': 93088, u'TotalDiskUsage': 466944, u'BytesIncoming': 32779843, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 129, u'NItemsPut': 227437, u'NSyncs': 2995, u'CompactReader': {u'NReadIOs': 2046, u'NReadBytes': 1023725, u'NDataBlockReads': 1088, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1088}, u'ActiveDiskUsage': 466944, u'NTablesDeleted': 1088, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 1016, u'NWriteBytes': 9350108, u'NIndexBlockWrites': 1016, u'NBytesIncoming': 825715, u'NSyncs': 2995, u'NInserts': 12304, u'NWriteIOs': 4011}}}}} 2022-02-08 00:16:40,207 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:728] magma stats fragmentation result {'172.23.121.123': [0.35277706384658813, 0.38722315430641174, 0.33606359362602234, 0.3179052770137787, 0.3729638457298279, 0.34746742248535156, 0.3377484679222107, 0.3689146935939789], '172.23.121.124': [0.3304431736469269, 0.34964707493782043, 0.3541373610496521, 0.3266328275203705, 0.3297898471355438, 0.3394237756729126, 0.33931463956832886, 0.3248528838157654], '172.23.121.115': [0.3586973249912262, 0.3644739091396332, 0.37626245617866516, 0.324814110994339, 0.3367738127708435, 0.35779696702957153, 0.3561156392097473, 0.33937162160873413], '172.23.121.126': [0.347074955701828, 0.360504686832428, 0.3466273844242096, 0.35112884640693665, 0.3270756006240845, 0.3755476474761963, 0.3459625244140625, 0.35927170515060425], '172.23.121.116': [0.4022288918495178, 0.4192350208759308, 0.4064333140850067, 0.42009514570236206, 0.39414119720458984, 0.40736886858940125, 0.3902329206466675, 0.41056591272354126], '172.23.121.127': [0.33984363079071045, 0.3392837345600128, 0.34347736835479736, 0.36163756251335144, 0.36723822355270386, 0.3489506244659424, 0.3132169842720032, 0.33223140239715576], '172.23.121.128': [0.3409234583377838, 0.3492090702056885, 0.31929418444633484, 0.31405800580978394, 0.3377780318260193, 0.3128279149532318, 0.34381425380706787, 0.3481939435005188], '172.23.121.129': [0.359859436750412, 0.32445231080055237, 0.3210977017879486, 0.33663210272789, 0.32896819710731506, 0.34452998638153076, 0.35442569851875305, 0.35104772448539734], '172.23.120.170': [0.32905933260917664, 0.357782781124115, 0.3323001563549042, 0.3316788375377655, 0.3270244300365448, 0.34271663427352905, 0.35027995705604553, 0.3243715167045593]} 2022-02-08 00:16:40,450 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.115 is 37.0 2022-02-08 00:16:40,711 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.116 is 37.0 2022-02-08 00:16:40,960 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.123 is 37.0 2022-02-08 00:16:41,201 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.124 is 37.0 2022-02-08 00:16:41,470 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.126 is 37.0 2022-02-08 00:16:41,750 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.127 is 37.0 2022-02-08 00:16:42,006 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.128 is 37.0 2022-02-08 00:16:42,250 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.129 is 37.0 2022-02-08 00:16:42,509 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.120.170 is 37.0 2022-02-08 00:16:42,510 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:677] KV stats fragmentation values {'172.23.121.123': 37.0, '172.23.121.124': 37.0, '172.23.121.115': 37.0, '172.23.121.126': 37.0, '172.23.121.116': 37.0, '172.23.121.127': 37.0, '172.23.121.128': 37.0, '172.23.121.129': 37.0, '172.23.120.170': 37.0} 2022-02-08 00:16:42,513 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:16:42,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:42,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:16:42,865 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:42,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:16:42,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:16:42,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:16:42,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:16:42,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:16:42,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:16:42,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:16:42,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:16:42,969 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 00:16:43,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:16:43,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:43,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:16:43,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:43,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:43,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:16:43,404 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 00:16:43,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:16:43,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:43,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:16:43,447 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 00:16:43,447 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:16:43,450 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:16:43,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:43,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:16:43,805 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:43,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:16:43,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:16:43,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:16:43,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:16:43,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:16:43,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:16:43,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:16:43,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:16:43,908 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 00:16:43,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:16:43,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:44,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:16:44,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:44,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:44,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:16:44,617 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 00:16:44,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:16:44,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:44,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:16:44,658 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 00:16:44,660 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:16:44,663 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:16:44,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:44,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:16:45,009 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:45,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:16:45,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:16:45,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:16:45,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:16:45,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:16:45,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:16:45,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:16:45,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:16:45,124 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 00:16:45,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:16:45,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:45,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:16:45,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:45,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:45,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:16:45,377 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 00:16:45,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:16:45,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:45,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:16:45,418 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 00:16:45,420 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:16:45,423 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:16:45,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:45,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:16:45,776 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:45,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:16:45,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:16:45,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:16:45,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:16:45,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:16:45,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:16:45,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:16:45,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:16:45,882 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 00:16:45,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:16:45,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:46,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:16:46,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:46,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:46,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:16:46,451 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 00:16:46,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:16:46,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:46,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:16:46,493 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 00:16:46,493 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:16:46,496 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:16:46,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:46,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:16:46,832 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:46,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:16:46,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:16:46,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:16:46,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:16:46,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:16:46,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:16:46,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:16:46,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:16:46,961 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 00:16:46,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:16:47,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:47,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:16:47,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:47,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:47,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:16:47,483 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 00:16:47,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:16:47,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:47,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:16:47,529 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 00:16:47,529 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:16:47,532 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:16:47,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:47,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:16:47,861 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:47,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:16:47,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:16:47,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:16:47,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:16:47,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:16:47,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:16:47,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:16:47,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:16:47,969 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 00:16:47,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:16:48,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:48,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:16:48,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:48,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:48,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:16:48,515 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 00:16:48,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:16:48,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:48,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:16:48,555 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 00:16:48,556 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:16:48,559 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:16:48,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:48,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:16:48,888 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:48,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:16:48,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:16:48,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:16:48,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:16:48,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:16:48,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:16:48,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:16:48,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:16:48,997 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 00:16:49,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:16:49,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:49,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:16:49,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:49,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:49,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:16:49,588 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 00:16:49,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:16:49,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:49,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:16:49,628 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 00:16:49,628 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:16:49,631 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:16:49,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:49,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:16:49,970 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:49,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:16:49,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:16:49,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:16:50,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:16:50,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:16:50,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:16:50,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:16:50,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:16:50,073 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 00:16:50,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:16:50,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:50,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:16:50,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:50,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:50,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:16:50,530 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 00:16:50,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:16:50,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:50,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:16:50,569 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 00:16:50,569 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:16:50,572 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:16:50,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:50,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:16:50,903 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:50,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:16:50,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:16:50,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:16:50,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:16:50,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:16:50,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:16:50,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:16:50,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:16:51,009 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 00:16:51,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:16:51,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:51,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:16:51,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:51,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:51,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:16:51,490 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 00:16:51,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:16:51,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:51,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:16:51,529 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 00:16:51,529 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:16:51,532 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 00:16:51,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:51,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 00:16:51,871 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:51,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 00:16:51,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 00:16:51,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 00:16:51,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 00:16:51,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 00:16:51,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 00:16:51,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 00:16:51,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:16:51,983 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 00:16:52,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:16:52,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:52,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:16:52,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:52,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:52,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:16:52,095 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 00:16:52,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:16:52,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:52,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:16:52,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:16:52,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,269 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 00:16:52,272 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 00:16:52,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:52,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 00:16:52,617 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:52,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 00:16:52,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 00:16:52,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 00:16:52,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 00:16:52,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 00:16:52,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 00:16:52,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 00:16:52,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:16:52,726 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 00:16:52,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:16:52,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:52,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:16:52,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:52,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:52,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:16:52,832 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 00:16:52,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:16:52,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:52,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:16:52,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:16:52,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:52,997 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 00:16:53,000 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 00:16:53,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:53,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 00:16:53,342 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:53,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 00:16:53,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 00:16:53,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 00:16:53,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 00:16:53,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 00:16:53,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 00:16:53,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 00:16:53,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:16:53,447 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 00:16:53,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:16:53,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:53,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:16:53,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:53,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:53,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:16:53,551 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 00:16:53,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:16:53,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:53,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:16:53,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:53,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:16:53,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:53,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:53,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:53,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:53,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:53,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:53,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:53,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:53,711 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 00:16:53,714 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 00:16:53,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:53,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 00:16:54,051 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:54,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 00:16:54,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 00:16:54,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 00:16:54,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 00:16:54,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 00:16:54,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 00:16:54,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 00:16:54,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:16:54,151 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 00:16:54,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:16:54,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:54,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:16:54,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:54,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:54,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:16:54,257 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 00:16:54,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:16:54,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:54,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:16:54,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:54,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:16:54,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:54,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:54,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:54,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:54,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:54,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:54,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:54,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:54,428 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 00:16:54,431 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 00:16:54,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:16:54,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 00:16:54,769 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:16:54,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 00:16:54,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 00:16:54,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 00:16:54,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 00:16:54,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 00:16:54,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 00:16:54,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 00:16:54,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:16:54,874 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 00:16:54,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:16:54,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:54,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:16:54,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:54,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:16:54,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:16:54,976 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 00:16:54,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:16:54,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:16:55,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:16:55,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:55,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:16:55,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:55,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:55,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:55,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:55,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:55,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:55,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:55,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:16:55,154 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 00:17:08,707 | test | INFO | MainThread | [GsiHelper_Rest:polling_create_index_status:371] Starting polling for index:idx0 2022-02-08 00:17:08,734 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/gsiLib/GsiHelper_Rest.py -> polling_create_index_status():L378 2022-02-08 00:17:08,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:17:18,736 | test | INFO | MainThread | [GsiHelper_Rest:polling_create_index_status:379] Index idx0 not found with iteration 0 2022-02-08 00:17:18,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/gsiLib/GsiHelper_Rest.py -> polling_create_index_status():L378 2022-02-08 00:17:18,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:17:28,757 | test | INFO | MainThread | [GsiHelper_Rest:polling_create_index_status:379] Index idx0 not found with iteration 1 2022-02-08 00:17:28,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/gsiLib/GsiHelper_Rest.py -> polling_create_index_status():L378 2022-02-08 00:17:28,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:17:38,773 | test | INFO | MainThread | [GsiHelper_Rest:polling_create_index_status:379] Index idx0 not found with iteration 2 2022-02-08 00:17:38,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/gsiLib/GsiHelper_Rest.py -> polling_create_index_status():L378 2022-02-08 00:17:38,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:17:48,792 | test | INFO | MainThread | [GsiHelper_Rest:polling_create_index_status:379] Index idx0 not found with iteration 3 2022-02-08 00:17:48,818 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/gsiLib/GsiHelper_Rest.py -> polling_create_index_status():L378 2022-02-08 00:17:48,819 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:17:58,819 | test | INFO | MainThread | [GsiHelper_Rest:polling_create_index_status:379] Index idx0 not found with iteration 4 2022-02-08 00:17:58,834 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/gsiLib/GsiHelper_Rest.py -> polling_create_index_status():L378 2022-02-08 00:17:58,834 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:18:08,834 | test | INFO | MainThread | [GsiHelper_Rest:polling_create_index_status:379] Index idx0 not found with iteration 5 2022-02-08 00:18:08,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/gsiLib/GsiHelper_Rest.py -> polling_create_index_status():L378 2022-02-08 00:18:08,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:18:18,851 | test | INFO | MainThread | [GsiHelper_Rest:polling_create_index_status:379] Index idx0 not found with iteration 6 2022-02-08 00:18:18,894 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/gsiLib/GsiHelper_Rest.py -> polling_create_index_status():L378 2022-02-08 00:18:18,894 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:18:28,894 | test | INFO | MainThread | [GsiHelper_Rest:polling_create_index_status:379] Index idx0 not found with iteration 7 2022-02-08 00:18:28,911 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/gsiLib/GsiHelper_Rest.py -> polling_create_index_status():L378 2022-02-08 00:18:28,911 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:18:38,911 | test | INFO | MainThread | [GsiHelper_Rest:polling_create_index_status:379] Index idx0 not found with iteration 8 2022-02-08 00:18:38,927 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/gsiLib/GsiHelper_Rest.py -> polling_create_index_status():L378 2022-02-08 00:18:38,927 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:18:48,928 | test | INFO | MainThread | [GsiHelper_Rest:polling_create_index_status:379] Index idx0 not found with iteration 9 2022-02-08 00:18:48,940 | test | INFO | MainThread | [GsiHelper_Rest:polling_create_index_status:371] Starting polling for index:idx2 2022-02-08 00:18:48,950 | test | INFO | MainThread | [GsiHelper_Rest:polling_create_index_status:371] Starting polling for index:idx1 2022-02-08 00:18:48,960 | test | INFO | MainThread | [GsiHelper_Rest:polling_create_index_status:371] Starting polling for index:idx4 2022-02-08 00:18:48,970 | test | INFO | MainThread | [GsiHelper_Rest:polling_create_index_status:371] Starting polling for index:idx3 ############################################################ # # RAM FootPrint: 3840.71875 # ############################################################ Read Start: 0 Read End: 1000000 Update Start: 2000000 Update End: 20000000 Expiry Start: 0 Expiry End: 0 Delete Start: 1000000 Delete End: 2000000 Create Start: 2000000 Create End: 20000000 Final Start: 2000000 Final End: 20000000 2022-02-08 00:18:49,381 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:18:49,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection9_10_1644308330.38 2022-02-08 00:18:50,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:18:50,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection8_9_1644308331.39 2022-02-08 00:18:51,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:18:51,401 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection19_8_1644308332.4 2022-02-08 00:18:52,413 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:18:52,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection13_7_1644308333.41 2022-02-08 00:18:53,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:18:53,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection14_6_1644308334.42 2022-02-08 00:18:54,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:18:54,436 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection11_5_1644308335.44 2022-02-08 00:18:55,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:18:55,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection12_4_1644308336.45 2022-02-08 00:18:56,464 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:18:56,464 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection17_3_1644308337.46 2022-02-08 00:18:57,474 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:18:57,474 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection18_2_1644308338.48 2022-02-08 00:18:58,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:18:58,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection15_1_1644308339.49 2022-02-08 00:18:59,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:18:59,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-08 00:19:00,512 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:19:00,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-08 00:19:01,522 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:19:01,522 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-08 00:19:02,533 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:19:02,533 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-08 00:19:03,548 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:19:03,548 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-08 00:19:04,556 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:19:04,556 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-08 00:19:05,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:19:05,566 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-08 00:19:06,575 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:19:06,575 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-08 00:19:07,586 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:19:07,586 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2022-02-08 00:19:08,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L579 2022-02-08 00:19:08,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None ############################################################ # # Step 4: Rebalance in with Loading of docs # ############################################################ KV nodes in cluster: ['172.23.121.115', '172.23.121.116', '172.23.121.123', '172.23.121.124', '172.23.121.126', '172.23.121.127', '172.23.121.128', '172.23.121.129'] CBAS nodes in cluster: [] INDEX nodes in cluster: ['172.23.121.130', '172.23.121.131', '172.23.121.132', '172.23.121.133', '172.23.121.134'] FTS nodes in cluster: [] QUERY nodes in cluster: ['172.23.121.130', '172.23.121.131', '172.23.121.132', '172.23.121.133', '172.23.121.134'] EVENTING nodes in cluster: [] AVAILABLE nodes for cluster: ['172.23.121.135', '172.23.121.136'] Servers coming in : ['172.23.121.135'] with services: ['kv'] Servers going out : [] NEW AVAILABLE nodes for cluster: ['172.23.121.136'] 2022-02-08 00:19:09,678 | infra | DEBUG | MainThread | [task:__init__:249] Last known rebalance status_id: 21d7450d52d81c3d095bdb0f158fde38 2022-02-08 00:19:09,678 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[172.23.121.135]_OUT=[]_1644308349.6 2022-02-08 00:19:09,678 | infra | INFO | pool-3-thread-6 | [task:start_task:91] Thread 'Rebalance_task_IN=[172.23.121.135]_OUT=[]_1644308349.6' started 2022-02-08 00:19:09,684 | test | DEBUG | pool-3-thread-6 | [rest_client:add_node:1199] Adding remote node 172.23.121.135:8091 to cluster 172.23.120.170:8091 2022-02-08 00:19:09,693 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/Hospital.py -> SystemTestMagma():L492 2022-02-08 00:19:09,694 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep for 60s for rebalance to start 2022-02-08 00:19:13,622 | test | DEBUG | pool-3-thread-6 | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.128,ns_1@172.23.121.115,ns_1@172.23.121.132,ns_1@172.23.121.135,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.134,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123,ns_1@172.23.121.131,ns_1@172.23.121.130,ns_1@172.23.121.133', 'ejectedNodes': '', 'user': 'Administrator'} 2022-02-08 00:19:13,852 | test | DEBUG | pool-3-thread-6 | [rest_client:rebalance:1459] Rebalance operation started 2022-02-08 00:19:13,881 | test | INFO | pool-3-thread-6 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.121.128 | kv | 7.1.0-2255-enterprise | 75.8081199897 | Cluster node | | 172.23.121.115 | kv | 7.1.0-2255-enterprise | 81.2990353698 | Cluster node | | 172.23.121.132 | index, n1ql | 7.1.0-2255-enterprise | 11.0779220779 | Cluster node | | 172.23.120.170 | kv | 7.1.0-2255-enterprise | 77.6862694634 | Cluster node | | 172.23.121.129 | kv | 7.1.0-2255-enterprise | 78.7937491928 | Cluster node | | 172.23.121.124 | kv | 7.1.0-2255-enterprise | 78.5705053638 | Cluster node | | 172.23.121.126 | kv | 7.1.0-2255-enterprise | 79.221448827 | Cluster node | | 172.23.121.134 | index, n1ql | 7.1.0-2255-enterprise | 12.7298627299 | Cluster node | | 172.23.121.116 | kv | 7.1.0-2255-enterprise | 70.6302021403 | Cluster node | | 172.23.121.127 | kv | 7.1.0-2255-enterprise | 78.607220866 | Cluster node | | 172.23.121.123 | kv | 7.1.0-2255-enterprise | 80.056686421 | Cluster node | | 172.23.121.131 | index, n1ql | 7.1.0-2255-enterprise | 10.9507088048 | Cluster node | | 172.23.121.130 | index, n1ql | 7.1.0-2255-enterprise | 19.5726271631 | Cluster node | | 172.23.121.133 | index, n1ql | 7.1.0-2255-enterprise | 10.9639344262 | Cluster node | | 172.23.121.135 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2022-02-08 00:19:13,900 | infra | DEBUG | pool-3-thread-6 | [task:call:316] New rebalance status_id: bf7ce2718436b9ab3f4e698aa4eb0ab5 2022-02-08 00:19:13,924 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 00:19:13,924 | test | INFO | pool-3-thread-6 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 00:19:14,023 | infra | DEBUG | pool-3-thread-6 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:19:14,026 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:19:14,026 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:19:24,055 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 00:19:24,055 | test | INFO | pool-3-thread-6 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 00:19:24,101 | infra | DEBUG | pool-3-thread-6 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:19:24,102 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:19:24,102 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:19:34,130 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.43 % 2022-02-08 00:19:34,130 | test | INFO | pool-3-thread-6 | [task:check:414] Rebalance - status: running, progress: 1.43 2022-02-08 00:19:34,174 | infra | DEBUG | pool-3-thread-6 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:19:34,178 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:19:34,178 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:19:44,203 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.02 % 2022-02-08 00:19:44,203 | test | INFO | pool-3-thread-6 | [task:check:414] Rebalance - status: running, progress: 6.02 2022-02-08 00:19:44,246 | infra | DEBUG | pool-3-thread-6 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:19:44,250 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:19:44,250 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 121 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:19:54,275 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.43 % 2022-02-08 00:19:54,275 | test | INFO | pool-3-thread-6 | [task:check:414] Rebalance - status: running, progress: 9.43 2022-02-08 00:19:54,322 | infra | DEBUG | pool-3-thread-6 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:19:54,325 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:19:54,325 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:20:04,344 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.43 % 2022-02-08 00:20:04,344 | test | INFO | pool-3-thread-6 | [task:check:414] Rebalance - status: running, progress: 14.43 2022-02-08 00:20:04,394 | infra | DEBUG | pool-3-thread-6 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:20:04,397 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:20:04,397 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:20:09,703 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L766 2022-02-08 00:20:09,703 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 30 seconds. Reason: Let the rebalance begin! 2022-02-08 00:20:14,444 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.03 % 2022-02-08 00:20:14,444 | test | INFO | pool-3-thread-6 | [task:check:414] Rebalance - status: running, progress: 19.03 2022-02-08 00:20:14,493 | infra | DEBUG | pool-3-thread-6 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:20:14,496 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:20:14,496 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:20:24,513 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.44 % 2022-02-08 00:20:24,513 | test | INFO | pool-3-thread-6 | [task:check:414] Rebalance - status: running, progress: 22.44 2022-02-08 00:20:24,562 | infra | DEBUG | pool-3-thread-6 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:20:24,565 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:20:24,565 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:20:34,585 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.90 % 2022-02-08 00:20:34,585 | test | INFO | pool-3-thread-6 | [task:check:414] Rebalance - status: running, progress: 26.9 2022-02-08 00:20:34,625 | infra | DEBUG | pool-3-thread-6 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:20:34,628 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:20:34,628 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:20:39,732 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.75 % 2022-02-08 00:20:39,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 00:20:39,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:20:44,648 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.14 % 2022-02-08 00:20:44,648 | test | INFO | pool-3-thread-6 | [task:check:414] Rebalance - status: running, progress: 29.14 2022-02-08 00:20:44,677 | infra | DEBUG | pool-3-thread-6 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:20:44,680 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:20:44,680 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 252 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:20:49,740 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >28.75% in 10.0180001259 seconds 2022-02-08 00:20:49,759 | test | DEBUG | MainThread | [rest_client:diag_eval:1476] /diag/eval status on 172.23.120.170:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2022-02-08 00:20:49,759 | test | INFO | MainThread | [opd:abort_rebalance:780] Abort rebalance 2022-02-08 00:20:49,765 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:20:50,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:20:50,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:20:50,957 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:20:50,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:20:50,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:20:51,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:20:51,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:20:51,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:20:51,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:20:51,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:20:51,151 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:20:51,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:20:51,223 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['33184 memcached\n'], []) 2022-02-08 00:20:51,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 00:20:51,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:20:51,276 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['33184 memcached\n'], []) 2022-02-08 00:20:51,276 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:20:51,279 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:20:51,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:20:51,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:20:51,641 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:20:51,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:20:51,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:20:51,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:20:51,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:20:51,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:20:51,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:20:51,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:20:51,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:20:51,753 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 00:20:51,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:20:51,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:20:52,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:20:53,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:20:53,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:20:53,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:20:53,191 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 00:20:53,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:20:53,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:20:53,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:20:53,250 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 00:20:53,252 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:20:53,255 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:20:54,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:20:54,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:20:54,691 | test | ERROR | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'3d29ee070292cdbab2e1f645cb51b077', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=19cf9b305c73dbce0f56336422643e21', u'status': u'notRunning'} - rebalance failed 2022-02-08 00:20:54,698 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:20:54,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:20:54,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:20:54,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:20:54,733 | test | INFO | pool-3-thread-6 | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.120.170: 2022-02-08 00:20:54,733 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644308452203L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:20:52.203Z', u'text': u"Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {'EXIT',<0.8090.38>,\n {{{{socket_closed,\n {gen_server,call,\n [<0.14794.0>,get_partitions,infinity]}},\n {gen_server,call,\n [<0.14793.0>,get_partitions,infinity]}},\n {gen_server,call,\n ['dcp_replication_manager-GleamBookUsers0',\n {get_replicator_pid,127},\n infinity]}},\n {gen_server,call,\n [{'janitor_agent-GleamBookUsers0',\n 'ns_1@172.23.120.170'},\n {if_rebalance,<0.27385.37>,\n {wait_dcp_data_move,\n ['ns_1@172.23.121.116',\n 'ns_1@172.23.121.135'],\n 27}},\n infinity]}}}}}.\nRebalance Operation Id = 7312ac940b46242ac493c5624d9fcb55"} 2022-02-08 00:20:54,734 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644308452170L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:20:52.170Z', u'text': u"Worker <0.8375.38> (for action {move,{27,\n ['ns_1@172.23.120.170',\n 'ns_1@172.23.121.116'],\n ['ns_1@172.23.121.116',\n 'ns_1@172.23.121.135'],\n []}}) exited with reason {unexpected_exit,\n {'EXIT',\n <0.8090.38>,\n {{{{socket_closed,\n {gen_server,\n call,\n [<0.14794.0>,\n get_partitions,\n infinity]}},\n {gen_server,\n call,\n [<0.14793.0>,\n get_partitions,\n infinity]}},\n {gen_server,\n call,\n ['dcp_replication_manager-GleamBookUsers0',\n {get_replicator_pid,\n 127},\n infinity]}},\n {gen_server,\n call,\n [{'janitor_agent-GleamBookUsers0',\n 'ns_1@172.23.120.170'},\n {if_rebalance,\n <0.27385.37>,\n {wait_dcp_data_move,\n ['ns_1@172.23.121.116',\n 'ns_1@172.23.121.135'],\n 27}},\n infinity]}}}}"} 2022-02-08 00:20:54,736 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644308452019L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:20:52.019Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW0207 21:21:31.348912 33241 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2022-02-08 00:20:54,736 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644308452005L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:20:52.005Z', u'text': u"Control connection to memcached on 'ns_1@172.23.120.170' disconnected. Check logs for details."} 2022-02-08 00:20:54,736 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644308366875L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:19:26.875Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2022-02-08 00:20:54,736 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.135', u'tstamp': 1644308354344L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:19:14.344Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.121.135\' in 0 seconds.'} 2022-02-08 00:20:54,736 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644308354193L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:19:14.193Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2022-02-08 00:20:54,736 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.121.135', u'tstamp': 1644308353681L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:19:13.681Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_storage_threads">>,\n <<"num_writer_threads">>,\n <<"scramsha_fallback_salt">>]'} 2022-02-08 00:20:54,736 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644308353639L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:19:13.639Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.121.128','ns_1@172.23.121.115',\n 'ns_1@172.23.121.132','ns_1@172.23.121.135',\n 'ns_1@172.23.120.170','ns_1@172.23.121.129',\n 'ns_1@172.23.121.124','ns_1@172.23.121.126',\n 'ns_1@172.23.121.134','ns_1@172.23.121.116',\n 'ns_1@172.23.121.127','ns_1@172.23.121.123',\n 'ns_1@172.23.121.131','ns_1@172.23.121.130',\n 'ns_1@172.23.121.133'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 7312ac940b46242ac493c5624d9fcb55"} 2022-02-08 00:20:54,736 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2831] {u'code': 3, u'module': u'ns_cluster', u'type': u'info', u'node': u'ns_1@172.23.121.135', u'tstamp': 1644308353536L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:19:13.536Z', u'text': u'Node ns_1@172.23.121.135 joined cluster'} 2022-02-08 00:20:54,736 | test | ERROR | pool-3-thread-6 | [task:call:325] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'3d29ee070292cdbab2e1f645cb51b077', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=19cf9b305c73dbce0f56336422643e21', u'status': u'notRunning'} - rebalance failed 2022-02-08 00:20:54,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:20:54,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:20:54,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:20:54,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:20:54,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:20:54,828 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 00:20:54,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:20:54,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:20:56,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:20:56,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:20:56,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:20:56,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:20:56,506 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 00:20:56,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:20:56,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:20:56,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:20:56,561 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 00:20:56,561 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:20:56,563 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:20:57,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:20:57,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:20:57,970 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:20:57,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:20:57,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:20:58,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:20:58,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:20:58,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:20:58,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:20:58,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:20:58,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:20:58,119 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 00:20:58,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:20:58,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:20:59,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:20:59,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:20:59,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:20:59,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:20:59,556 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 00:20:59,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:20:59,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:20:59,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:20:59,638 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 00:20:59,638 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:20:59,641 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 00:20:59,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:20:59,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 00:20:59,983 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:20:59,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 00:20:59,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 00:21:00,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 00:21:00,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 00:21:00,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 00:21:00,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 00:21:00,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 00:21:00,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:21:00,092 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 00:21:00,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:21:00,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:00,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:21:00,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:00,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:00,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:21:00,220 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 00:21:00,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:21:00,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:00,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:21:00,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:00,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:21:00,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:00,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:00,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:00,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:00,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:00,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:00,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:00,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:00,456 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 00:21:00,459 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:21:00,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:21:00,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:21:00,821 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:21:00,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:21:00,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:21:00,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:21:00,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:21:00,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:21:00,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:21:00,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:21:00,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:21:00,923 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 00:21:00,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:21:00,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:00,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:21:00,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:01,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:01,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:21:01,032 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 00:21:01,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:21:01,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:01,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:21:01,075 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 00:21:01,075 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:21:01,078 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 00:21:01,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:21:01,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 00:21:01,417 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:21:01,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 00:21:01,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 00:21:01,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 00:21:01,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 00:21:01,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 00:21:01,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 00:21:01,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 00:21:01,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:21:01,540 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 00:21:01,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:21:01,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:01,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:21:01,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:01,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:01,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:21:01,653 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 00:21:01,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:21:01,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:01,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:21:01,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:01,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:21:01,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:01,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:01,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:01,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:01,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:01,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:01,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:01,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:01,831 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 00:21:01,832 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:21:02,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:21:03,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:21:03,236 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:21:03,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:21:03,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:21:03,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:21:03,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:21:03,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:21:03,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:21:03,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:21:03,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:21:03,380 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 00:21:03,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:21:03,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:04,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:21:04,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:04,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:04,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:21:04,950 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 00:21:04,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:21:04,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:04,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:21:05,007 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 00:21:05,007 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:21:05,012 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:21:05,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:21:06,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:21:06,226 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:21:06,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:21:06,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:21:06,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:21:06,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:21:06,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:21:06,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:21:06,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:21:06,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:21:06,403 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 00:21:06,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:21:06,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:07,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:21:07,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:07,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:08,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:21:08,076 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 00:21:08,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:21:08,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:08,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:21:08,145 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 00:21:08,145 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:21:08,148 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 00:21:08,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:21:08,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 00:21:08,529 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:21:08,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 00:21:08,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 00:21:08,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 00:21:08,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 00:21:08,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 00:21:08,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 00:21:08,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 00:21:08,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:21:08,637 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 00:21:08,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:21:08,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:08,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:21:08,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:08,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:08,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:21:08,756 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 00:21:08,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:21:08,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:08,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:21:08,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:08,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:21:08,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:08,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:08,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:08,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:08,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:08,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:08,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:08,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:08,937 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 00:21:08,940 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 00:21:09,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:21:09,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 00:21:09,288 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:21:09,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 00:21:09,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 00:21:09,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 00:21:09,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 00:21:09,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 00:21:09,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 00:21:09,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 00:21:09,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:21:09,385 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 00:21:09,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:21:09,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:09,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:21:09,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:09,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:09,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:21:09,496 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 00:21:09,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:21:09,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:09,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:21:09,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:09,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:21:09,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:09,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:09,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:09,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:09,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:09,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:09,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:09,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:09,661 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 00:21:09,664 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:21:10,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:21:10,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:21:11,002 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:21:11,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:21:11,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:21:11,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:21:11,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:21:11,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:21:11,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:21:11,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:21:11,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:21:11,122 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 00:21:11,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:21:11,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:12,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:21:12,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:13,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:13,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:21:13,145 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 00:21:13,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:21:13,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:13,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:21:13,197 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 00:21:13,197 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:21:13,200 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:21:14,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:21:14,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:21:14,565 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:21:14,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:21:14,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:21:14,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:21:14,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:21:14,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:21:14,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:21:14,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:21:14,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:21:14,710 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 00:21:14,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:21:14,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:15,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:21:16,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:16,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:16,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:21:16,326 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 00:21:16,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:21:16,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:16,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:21:16,378 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 00:21:16,378 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:21:16,381 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:21:17,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:21:17,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:21:17,572 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:21:17,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:21:17,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:21:17,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:21:17,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:21:17,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:21:17,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:21:17,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:21:17,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:21:17,698 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 00:21:17,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:21:17,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:18,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:21:18,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:18,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:18,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:21:18,415 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 00:21:18,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:21:18,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:18,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:21:18,463 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 00:21:18,463 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:21:18,467 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:21:19,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:21:19,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:21:19,838 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:21:19,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:21:19,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:21:19,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:21:19,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:21:19,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:21:19,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:21:19,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:21:19,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:21:19,963 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 00:21:19,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:21:20,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:20,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:21:21,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:21,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:21,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:21:21,211 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 00:21:21,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:21:21,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:21,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:21:21,269 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 00:21:21,269 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:21:21,272 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 00:21:21,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:21:21,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 00:21:21,617 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:21:21,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 00:21:21,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 00:21:21,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 00:21:21,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 00:21:21,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 00:21:21,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 00:21:21,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 00:21:21,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:21:21,726 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 00:21:21,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:21:21,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:21,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:21:21,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:21,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:21:21,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:21:21,852 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 00:21:21,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:21:21,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:21:21,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:21:21,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:21,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:21:21,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:21,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:21,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:21,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:21,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:21,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:21,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:22,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:21:22,025 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 00:21:22,032 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L789 2022-02-08 00:21:22,032 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 5 queries submitted, 0 failed, 320 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:22:22,055 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.120.170:8091 2022-02-08 00:22:22,055 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:22:22,069 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.120.170:8091 is running 2022-02-08 00:22:22,082 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.126:8091 2022-02-08 00:22:22,082 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:22:22,094 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.126:8091 is running 2022-02-08 00:22:22,105 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.128:8091 2022-02-08 00:22:22,107 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:22:22,118 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.128:8091 is running 2022-02-08 00:22:22,131 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.133:8091 2022-02-08 00:22:22,132 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:22:22,144 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.133:8091 is running 2022-02-08 00:22:22,160 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.135:8091 2022-02-08 00:22:22,161 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:22:22,171 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.135:8091 is running 2022-02-08 00:22:22,183 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.134:8091 2022-02-08 00:22:22,183 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:22:22,197 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.134:8091 is running 2022-02-08 00:22:22,213 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.127:8091 2022-02-08 00:22:22,213 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:22:22,226 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.127:8091 is running 2022-02-08 00:22:22,240 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.124:8091 2022-02-08 00:22:22,240 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:22:22,252 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.124:8091 is running 2022-02-08 00:22:22,265 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.130:8091 2022-02-08 00:22:22,265 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:22:22,278 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.130:8091 is running 2022-02-08 00:22:22,325 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.132:8091 2022-02-08 00:22:22,325 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:22:22,398 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.132:8091 is running 2022-02-08 00:22:22,421 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.115:8091 2022-02-08 00:22:22,421 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:22:22,437 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.115:8091 is running 2022-02-08 00:22:22,451 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.123:8091 2022-02-08 00:22:22,451 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:22:22,467 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.123:8091 is running 2022-02-08 00:22:22,493 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.116:8091 2022-02-08 00:22:22,493 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:22:22,515 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.116:8091 is running 2022-02-08 00:22:22,532 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.129:8091 2022-02-08 00:22:22,532 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:22:22,559 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.129:8091 is running 2022-02-08 00:22:22,575 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.131:8091 2022-02-08 00:22:22,575 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:22:22,596 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.131:8091 is running 2022-02-08 00:22:22,599 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:22:23,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:22:23,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:22:23,604 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:22:23,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:22:23,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:22:23,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:22:23,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:22:23,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:22:23,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:22:23,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:22:23,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:22:23,760 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 00:22:23,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:22:23,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:23,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:22:24,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:24,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:24,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:22:24,226 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 00:22:24,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:22:24,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:24,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:22:24,278 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 00:22:24,278 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:22:24,280 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:22:24,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:22:24,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:22:24,703 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:22:24,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:22:24,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:22:24,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:22:24,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:22:24,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:22:24,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:22:24,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:22:24,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:22:24,904 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 00:22:24,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:22:24,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:25,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:22:25,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:25,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:25,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:22:25,427 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 00:22:25,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:22:25,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:25,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:22:25,479 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 00:22:25,479 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:22:25,484 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:22:25,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:22:25,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:22:25,959 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:22:25,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:22:25,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:22:25,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:22:25,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:22:26,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:22:26,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:22:26,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:22:26,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:22:26,065 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 00:22:26,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:22:26,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:26,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:22:26,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:26,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:26,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:22:26,480 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 00:22:26,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:22:26,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:26,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:22:26,520 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 00:22:26,520 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:22:26,523 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 00:22:26,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:22:26,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 00:22:26,882 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:22:26,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 00:22:26,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 00:22:26,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 00:22:26,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 00:22:26,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 00:22:26,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 00:22:26,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 00:22:27,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:22:27,022 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 00:22:27,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:22:27,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:27,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:22:27,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:27,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:27,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:22:27,141 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 00:22:27,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:22:27,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:27,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:22:27,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:27,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:22:27,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:27,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:27,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:27,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:27,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:27,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:27,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:27,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:27,329 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 00:22:27,332 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:22:27,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:22:27,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:22:27,691 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:22:27,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:22:27,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:22:27,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:22:27,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:22:27,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:22:27,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:22:27,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:22:27,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:22:27,819 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 00:22:27,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:22:27,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:27,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:22:27,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:27,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:27,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:22:27,966 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 00:22:27,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:22:27,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:28,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:22:28,033 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 00:22:28,035 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:22:28,038 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 00:22:28,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:22:28,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 00:22:28,410 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:22:28,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 00:22:28,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 00:22:28,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 00:22:28,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 00:22:28,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 00:22:28,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 00:22:28,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 00:22:28,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:22:28,535 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 00:22:28,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:22:28,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:28,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:22:28,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:28,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:28,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:22:28,663 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 00:22:28,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:22:28,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:28,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:22:28,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:28,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:22:28,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:28,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:28,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:28,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:28,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:28,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:28,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:28,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:28,880 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 00:22:28,881 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:22:28,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:22:29,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:22:29,259 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:22:29,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:22:29,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:22:29,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:22:29,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:22:29,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:22:29,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:22:29,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:22:29,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:22:29,372 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 00:22:29,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:22:29,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:29,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:22:29,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:29,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:29,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:22:29,936 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 00:22:29,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:22:29,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:29,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:22:30,013 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 00:22:30,013 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:22:30,016 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:22:30,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:22:30,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:22:30,509 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:22:30,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:22:30,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:22:30,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:22:30,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:22:30,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:22:30,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:22:30,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:22:30,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:22:30,650 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 00:22:30,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:22:30,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:30,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:22:30,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:31,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:31,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:22:31,161 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 00:22:31,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:22:31,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:31,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:22:31,210 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 00:22:31,210 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:22:31,211 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 00:22:31,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:22:31,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 00:22:31,706 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:22:31,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 00:22:31,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 00:22:31,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 00:22:31,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 00:22:31,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 00:22:31,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 00:22:31,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 00:22:31,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:22:31,934 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 00:22:31,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:22:31,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:31,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:22:32,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:32,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:32,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:22:32,038 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 00:22:32,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:22:32,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:32,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:22:32,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:32,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:22:32,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:32,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:32,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:32,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:32,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:32,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:32,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:32,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:32,292 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 00:22:32,296 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 00:22:32,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:22:32,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 00:22:32,687 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:22:32,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 00:22:32,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 00:22:32,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 00:22:32,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 00:22:32,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 00:22:32,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 00:22:32,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 00:22:32,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:22:32,809 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 00:22:32,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:22:32,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:32,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:22:32,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:32,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:32,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:22:32,943 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 00:22:32,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:22:32,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:32,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:22:32,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:33,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:22:33,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:33,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:33,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:33,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:33,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:33,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:33,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:33,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:33,147 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 00:22:33,150 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:22:33,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:22:33,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:22:33,934 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:22:33,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:22:34,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:22:34,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:22:34,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:22:34,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:22:34,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:22:34,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:22:34,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:22:34,369 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 00:22:34,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:22:34,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:34,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:22:34,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:35,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:35,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:22:35,466 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 00:22:35,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:22:35,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:35,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:22:35,654 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 00:22:35,654 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:22:35,668 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:22:35,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:22:36,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:22:36,515 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:22:36,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:22:36,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:22:36,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:22:36,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:22:36,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:22:36,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:22:36,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:22:36,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:22:36,825 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 00:22:36,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:22:36,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:37,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:22:37,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:37,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:37,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:22:37,614 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 00:22:37,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:22:37,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:37,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:22:37,788 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 00:22:37,788 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:22:37,790 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:22:37,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:22:38,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:22:38,223 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:22:38,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:22:38,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:22:38,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:22:38,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:22:38,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:22:38,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:22:38,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:22:38,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:22:38,364 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 00:22:38,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:22:38,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:38,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:22:38,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:38,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:38,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:22:38,734 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 00:22:38,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:22:38,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:38,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:22:38,821 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 00:22:38,821 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:22:38,822 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:22:38,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:22:39,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:22:39,200 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:22:39,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:22:39,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:22:39,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:22:39,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:22:39,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:22:39,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:22:39,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:22:39,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:22:39,312 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 00:22:39,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:22:39,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:39,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:22:39,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:39,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:39,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:22:39,799 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 00:22:39,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:22:39,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:39,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:22:39,841 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 00:22:39,841 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:22:39,844 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 00:22:39,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:22:40,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 00:22:40,234 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:22:40,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 00:22:40,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 00:22:40,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 00:22:40,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 00:22:40,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 00:22:40,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 00:22:40,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 00:22:40,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:22:40,365 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 00:22:40,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:22:40,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:40,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:22:40,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:40,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:22:40,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:22:40,529 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 00:22:40,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:22:40,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:22:40,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:22:40,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:40,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:22:40,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:40,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:40,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:40,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:40,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:40,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:40,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:40,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:22:40,755 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 00:22:40,755 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[172.23.121.135]_OUT=[]_1644308349.6 2022-02-08 00:22:40,756 | test | INFO | MainThread | [opd:abort_rebalance:808] Restarting Rebalance after killing at 20 2022-02-08 00:22:40,865 | infra | DEBUG | MainThread | [task:__init__:249] Last known rebalance status_id: 3d29ee070292cdbab2e1f645cb51b077 2022-02-08 00:22:40,865 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[]_1644308560.76 2022-02-08 00:22:40,865 | infra | INFO | pool-3-thread-7 | [task:start_task:91] Thread 'Rebalance_task_IN=[]_OUT=[]_1644308560.76' started 2022-02-08 00:22:40,894 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L813 2022-02-08 00:22:40,894 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2022-02-08 00:22:40,924 | test | DEBUG | pool-3-thread-7 | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.128,ns_1@172.23.121.115,ns_1@172.23.121.132,ns_1@172.23.121.135,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.134,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123,ns_1@172.23.121.131,ns_1@172.23.121.130,ns_1@172.23.121.133', 'ejectedNodes': '', 'user': 'Administrator'} 2022-02-08 00:22:40,993 | test | DEBUG | pool-3-thread-7 | [rest_client:rebalance:1459] Rebalance operation started 2022-02-08 00:22:40,993 | test | INFO | pool-3-thread-7 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.121.128 | kv | 7.1.0-2255-enterprise | 33.5685352149 | Cluster node | | 172.23.121.115 | kv | 7.1.0-2255-enterprise | 40.6546304715 | Cluster node | | 172.23.121.132 | index, n1ql | 7.1.0-2255-enterprise | 6.03647572566 | Cluster node | | 172.23.121.135 | kv | 7.1.0-2255-enterprise | 18.0201698514 | Cluster node | | 172.23.120.170 | kv | 7.1.0-2255-enterprise | 26.3872255489 | Cluster node | | 172.23.121.129 | kv | 7.1.0-2255-enterprise | 39.981749446 | Cluster node | | 172.23.121.124 | kv | 7.1.0-2255-enterprise | 45.2467532468 | Cluster node | | 172.23.121.126 | kv | 7.1.0-2255-enterprise | 41.6851152494 | Cluster node | | 172.23.121.134 | index, n1ql | 7.1.0-2255-enterprise | 8.64878737328 | Cluster node | | 172.23.121.116 | kv | 7.1.0-2255-enterprise | 63.5277308741 | Cluster node | | 172.23.121.127 | kv | 7.1.0-2255-enterprise | 44.9249093734 | Cluster node | | 172.23.121.123 | kv | 7.1.0-2255-enterprise | 41.8650019508 | Cluster node | | 172.23.121.131 | index, n1ql | 7.1.0-2255-enterprise | 5.10911424904 | Cluster node | | 172.23.121.130 | index, n1ql | 7.1.0-2255-enterprise | 8.90885750963 | Cluster node | | 172.23.121.133 | index, n1ql | 7.1.0-2255-enterprise | 8.12788906009 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2022-02-08 00:22:41,007 | infra | DEBUG | pool-3-thread-7 | [task:call:316] New rebalance status_id: 1b0e5bf79654fa3b8aa3b72b57b0bb75 2022-02-08 00:22:41,023 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 00:22:41,023 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 00:22:41,068 | infra | DEBUG | pool-3-thread-7 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:22:41,071 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:22:41,072 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 399 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:22:51,105 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.98 % 2022-02-08 00:22:51,105 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance - status: running, progress: 1.98 2022-02-08 00:22:51,138 | infra | DEBUG | pool-3-thread-7 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:22:51,141 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:22:51,142 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:23:01,173 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.43 % 2022-02-08 00:23:01,174 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance - status: running, progress: 5.43 2022-02-08 00:23:01,246 | infra | DEBUG | pool-3-thread-7 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:23:01,247 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:23:01,249 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:23:11,290 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.08 % 2022-02-08 00:23:11,292 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance - status: running, progress: 14.08 2022-02-08 00:23:11,365 | infra | DEBUG | pool-3-thread-7 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:23:11,368 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:23:11,368 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:23:21,382 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.20 % 2022-02-08 00:23:21,382 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance - status: running, progress: 17.2 2022-02-08 00:23:21,420 | infra | DEBUG | pool-3-thread-7 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:23:21,423 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:23:21,423 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:23:31,437 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.75 % 2022-02-08 00:23:31,437 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance - status: running, progress: 25.75 2022-02-08 00:23:31,479 | infra | DEBUG | pool-3-thread-7 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:23:31,480 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:23:31,480 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:23:41,500 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.95 % 2022-02-08 00:23:41,500 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance - status: running, progress: 28.95 2022-02-08 00:23:41,529 | infra | DEBUG | pool-3-thread-7 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:23:41,530 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:23:41,530 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 473 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:23:51,566 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.06 % 2022-02-08 00:23:51,566 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance - status: running, progress: 35.06 2022-02-08 00:23:51,608 | infra | DEBUG | pool-3-thread-7 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:23:51,609 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:23:51,611 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:24:01,647 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.25 % 2022-02-08 00:24:01,647 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance - status: running, progress: 36.25 2022-02-08 00:24:01,687 | infra | DEBUG | pool-3-thread-7 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:24:01,690 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:24:01,690 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:24:11,730 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.13 % 2022-02-08 00:24:11,732 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance - status: running, progress: 40.13 2022-02-08 00:24:11,780 | infra | DEBUG | pool-3-thread-7 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:24:11,783 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:24:11,783 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:24:21,808 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.66 % 2022-02-08 00:24:21,808 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance - status: running, progress: 44.66 2022-02-08 00:24:21,868 | infra | DEBUG | pool-3-thread-7 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:24:21,871 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:24:21,871 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:24:31,894 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.75 % 2022-02-08 00:24:31,895 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance - status: running, progress: 47.75 2022-02-08 00:24:31,924 | infra | DEBUG | pool-3-thread-7 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:24:31,927 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:24:31,927 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:24:40,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.39 % 2022-02-08 00:24:40,917 | test | INFO | MainThread | [opd:abort_rebalance:814] Rebalance % = 51.39 2022-02-08 00:24:40,931 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.39 % 2022-02-08 00:24:40,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 00:24:40,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:24:41,941 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.80 % 2022-02-08 00:24:41,941 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance - status: running, progress: 51.8 2022-02-08 00:24:41,970 | infra | DEBUG | pool-3-thread-7 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:24:41,973 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:24:41,973 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 546 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:24:50,940 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >51.39% in 10.0230000019 seconds 2022-02-08 00:24:50,960 | test | DEBUG | MainThread | [rest_client:diag_eval:1476] /diag/eval status on 172.23.120.170:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2022-02-08 00:24:50,960 | test | INFO | MainThread | [opd:abort_rebalance:780] Abort rebalance 2022-02-08 00:24:50,963 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:24:51,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:24:51,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:24:51,424 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:24:51,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:24:51,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:24:51,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:24:51,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:24:51,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:24:51,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:24:51,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:24:51,589 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:24:51,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:24:51,663 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['57009 memcached\n'], []) 2022-02-08 00:24:51,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 00:24:51,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:24:51,711 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['57009 memcached\n'], []) 2022-02-08 00:24:51,711 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:24:51,719 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:24:51,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:24:51,983 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 53.56 % 2022-02-08 00:24:51,984 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance - status: running, progress: 53.56 2022-02-08 00:24:52,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:24:52,010 | infra | DEBUG | pool-3-thread-7 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:24:52,013 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:24:52,013 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:24:52,069 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:24:52,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:24:52,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:24:52,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:24:52,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:24:52,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:24:52,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:24:52,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:24:52,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:24:52,174 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 00:24:52,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:24:52,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:52,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:24:52,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:52,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:52,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:24:52,621 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 00:24:52,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:24:52,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:52,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:24:52,674 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 00:24:52,674 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:24:52,677 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:24:52,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:24:52,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:24:53,045 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:24:53,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:24:53,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:24:53,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:24:53,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:24:53,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:24:53,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:24:53,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:24:53,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:24:53,151 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 00:24:53,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:24:53,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:53,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:24:53,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:53,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:53,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:24:53,661 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 00:24:53,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:24:53,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:53,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:24:53,703 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 00:24:53,703 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:24:53,707 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:24:53,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:24:53,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:24:54,049 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:24:54,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:24:54,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:24:54,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:24:54,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:24:54,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:24:54,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:24:54,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:24:54,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:24:54,180 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 00:24:54,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:24:54,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:54,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:24:54,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:54,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:54,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:24:54,627 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 00:24:54,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:24:54,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:54,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:24:54,670 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 00:24:54,671 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:24:54,674 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 00:24:54,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:24:55,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 00:24:55,069 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:24:55,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 00:24:55,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 00:24:55,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 00:24:55,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 00:24:55,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 00:24:55,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 00:24:55,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 00:24:55,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:24:55,180 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 00:24:55,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:24:55,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:55,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:24:55,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:55,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:55,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:24:55,311 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 00:24:55,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:24:55,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:55,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:24:55,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:55,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:24:55,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:55,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:55,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:55,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:55,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:55,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:55,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:55,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:55,519 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 00:24:55,523 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:24:55,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:24:55,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:24:55,891 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:24:55,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:24:55,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:24:55,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:24:55,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:24:55,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:24:55,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:24:55,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:24:55,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:24:56,000 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 00:24:56,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:24:56,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:56,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:24:56,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:56,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:56,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:24:56,119 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 00:24:56,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:24:56,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:56,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:24:56,160 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 00:24:56,161 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:24:56,163 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 00:24:56,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:24:56,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 00:24:56,500 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:24:56,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 00:24:56,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 00:24:56,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 00:24:56,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 00:24:56,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 00:24:56,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 00:24:56,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 00:24:56,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:24:56,602 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 00:24:56,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:24:56,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:56,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:24:56,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:56,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:56,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:24:56,720 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 00:24:56,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:24:56,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:56,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:24:56,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:56,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:24:56,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:56,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:56,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:56,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:56,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:56,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:56,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:56,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:56,934 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 00:24:56,937 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:24:57,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:24:57,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:24:57,267 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:24:57,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:24:57,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:24:57,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:24:57,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:24:57,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:24:57,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:24:57,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:24:57,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:24:57,398 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 00:24:57,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:24:57,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:57,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:24:57,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:57,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:57,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:24:57,924 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 00:24:57,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:24:57,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:57,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:24:57,967 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 00:24:57,967 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:24:57,970 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:24:58,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:24:58,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:24:58,378 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:24:58,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:24:58,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:24:58,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:24:58,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:24:58,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:24:58,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:24:58,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:24:58,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:24:58,523 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 00:24:58,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:24:58,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:58,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:24:58,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:58,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:58,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:24:58,996 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 00:24:58,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:24:59,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:59,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:24:59,039 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 00:24:59,040 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:24:59,042 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 00:24:59,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:24:59,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 00:24:59,391 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:24:59,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 00:24:59,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 00:24:59,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 00:24:59,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 00:24:59,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 00:24:59,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 00:24:59,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 00:24:59,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:24:59,506 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 00:24:59,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:24:59,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:59,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:24:59,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:59,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:24:59,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:24:59,625 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 00:24:59,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:24:59,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:24:59,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:24:59,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:59,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:24:59,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:59,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:59,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:59,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:59,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:59,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:59,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:59,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:24:59,813 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 00:24:59,816 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 00:24:59,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:25:00,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 00:25:00,167 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:25:00,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 00:25:00,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 00:25:00,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 00:25:00,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 00:25:00,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 00:25:00,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 00:25:00,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 00:25:00,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:25:00,279 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 00:25:00,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:25:00,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:25:00,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:25:00,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:25:00,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:25:00,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:25:00,394 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 00:25:00,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:25:00,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:25:00,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:25:00,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:00,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:25:00,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:00,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:00,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:00,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:00,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:00,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:00,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:00,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:00,589 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 00:25:00,592 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:25:00,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:25:00,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:25:00,989 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:25:00,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:25:01,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:25:01,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:25:01,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:25:01,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:25:01,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:25:01,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:25:01,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:25:01,108 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 00:25:01,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:25:01,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:25:01,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:25:01,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:25:01,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:25:01,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:25:01,740 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 00:25:01,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:25:01,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:25:01,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:25:01,789 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 00:25:01,789 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:25:01,792 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:25:01,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:25:02,025 | test | ERROR | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'1896465d088ea797077b7aa29ce93ff7', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=167897248c4840fdbcdbedaa49f9d2c8', u'status': u'notRunning'} - rebalance failed 2022-02-08 00:25:02,049 | test | INFO | pool-3-thread-7 | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.120.170: 2022-02-08 00:25:02,049 | test | ERROR | pool-3-thread-7 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644308693978L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:24:53.978Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.6192.39>,\n {{dcp_wait_for_data_move_failed,\n "GleamBookUsers0",925,\'ns_1@172.23.121.129\',\n [\'ns_1@172.23.121.135\',\n \'ns_1@172.23.120.170\'],\n {error,no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,451}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,211}]}]}}}}.\nRebalance Operation Id = c0534d90226e1795ab10195a3f010857'} 2022-02-08 00:25:02,051 | test | ERROR | pool-3-thread-7 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644308693959L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:24:53.959Z', u'text': u'Worker <0.5949.39> (for action {move,{925,\n [\'ns_1@172.23.121.129\',\n \'ns_1@172.23.120.170\'],\n [\'ns_1@172.23.121.135\',\n \'ns_1@172.23.120.170\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.6192.39>,\n {{dcp_wait_for_data_move_failed,\n "GleamBookUsers0",\n 925,\n \'ns_1@172.23.121.129\',\n [\'ns_1@172.23.121.135\',\n \'ns_1@172.23.120.170\'],\n {error,\n no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 451}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 211}]}]}}}'} 2022-02-08 00:25:02,051 | test | ERROR | pool-3-thread-7 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644308692919L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:24:52.919Z', u'text': u"Control connection to memcached on 'ns_1@172.23.120.170' disconnected. Check logs for details."} 2022-02-08 00:25:02,051 | test | ERROR | pool-3-thread-7 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644308692410L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:24:52.410Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW0208 00:21:23.556671 57073 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2022-02-08 00:25:02,051 | test | ERROR | pool-3-thread-7 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644308561703L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:22:41.703Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2022-02-08 00:25:02,052 | test | ERROR | pool-3-thread-7 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644308561356L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:22:41.356Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2022-02-08 00:25:02,052 | test | ERROR | pool-3-thread-7 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644308560932L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:22:40.932Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.121.128','ns_1@172.23.121.115',\n 'ns_1@172.23.121.132','ns_1@172.23.121.135',\n 'ns_1@172.23.120.170','ns_1@172.23.121.129',\n 'ns_1@172.23.121.124','ns_1@172.23.121.126',\n 'ns_1@172.23.121.134','ns_1@172.23.121.116',\n 'ns_1@172.23.121.127','ns_1@172.23.121.123',\n 'ns_1@172.23.121.131','ns_1@172.23.121.130',\n 'ns_1@172.23.121.133'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = c0534d90226e1795ab10195a3f010857"} 2022-02-08 00:25:02,052 | test | ERROR | pool-3-thread-7 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644308478757L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:21:18.757Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.120.170\' in 18 seconds.'} 2022-02-08 00:25:02,052 | test | ERROR | pool-3-thread-7 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644308452203L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:20:52.203Z', u'text': u"Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {'EXIT',<0.8090.38>,\n {{{{socket_closed,\n {gen_server,call,\n [<0.14794.0>,get_partitions,infinity]}},\n {gen_server,call,\n [<0.14793.0>,get_partitions,infinity]}},\n {gen_server,call,\n ['dcp_replication_manager-GleamBookUsers0',\n {get_replicator_pid,127},\n infinity]}},\n {gen_server,call,\n [{'janitor_agent-GleamBookUsers0',\n 'ns_1@172.23.120.170'},\n {if_rebalance,<0.27385.37>,\n {wait_dcp_data_move,\n ['ns_1@172.23.121.116',\n 'ns_1@172.23.121.135'],\n 27}},\n infinity]}}}}}.\nRebalance Operation Id = 7312ac940b46242ac493c5624d9fcb55"} 2022-02-08 00:25:02,052 | test | ERROR | pool-3-thread-7 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644308452170L, u'shortText': u'message', u'serverTime': u'2022-02-08T00:20:52.170Z', u'text': u"Worker <0.8375.38> (for action {move,{27,\n ['ns_1@172.23.120.170',\n 'ns_1@172.23.121.116'],\n ['ns_1@172.23.121.116',\n 'ns_1@172.23.121.135'],\n []}}) exited with reason {unexpected_exit,\n {'EXIT',\n <0.8090.38>,\n {{{{socket_closed,\n {gen_server,\n call,\n [<0.14794.0>,\n get_partitions,\n infinity]}},\n {gen_server,\n call,\n [<0.14793.0>,\n get_partitions,\n infinity]}},\n {gen_server,\n call,\n ['dcp_replication_manager-GleamBookUsers0',\n {get_replicator_pid,\n 127},\n infinity]}},\n {gen_server,\n call,\n [{'janitor_agent-GleamBookUsers0',\n 'ns_1@172.23.120.170'},\n {if_rebalance,\n <0.27385.37>,\n {wait_dcp_data_move,\n ['ns_1@172.23.121.116',\n 'ns_1@172.23.121.135'],\n 27}},\n infinity]}}}}"} 2022-02-08 00:25:02,052 | test | ERROR | pool-3-thread-7 | [task:call:325] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'1896465d088ea797077b7aa29ce93ff7', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=167897248c4840fdbcdbedaa49f9d2c8', u'status': u'notRunning'} - rebalance failed 2022-02-08 00:25:02,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:25:02,167 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:25:02,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:25:02,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:25:02,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:25:02,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:25:02,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:25:02,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:25:02,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:25:02,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:25:02,276 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 00:25:02,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:25:02,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:25:02,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:25:02,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:25:02,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:25:02,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:25:02,816 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 00:25:02,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:25:02,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:25:02,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:25:02,858 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 00:25:02,858 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:25:02,861 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:25:02,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:25:03,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:25:03,392 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:25:03,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:25:03,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:25:03,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:25:03,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:25:03,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:25:03,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:25:03,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:25:03,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:25:03,709 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 00:25:03,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:25:03,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:25:03,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:25:04,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:25:04,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:25:04,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:25:04,427 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 00:25:04,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:25:04,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:25:04,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:25:04,614 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 00:25:04,614 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:25:04,615 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:25:04,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:25:04,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:25:04,997 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:25:04,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:25:05,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:25:05,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:25:05,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:25:05,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:25:05,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:25:05,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:25:05,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:25:05,125 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 00:25:05,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:25:05,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:25:05,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:25:05,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:25:05,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:25:05,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:25:05,630 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 00:25:05,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:25:05,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:25:05,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:25:05,700 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 00:25:05,700 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:25:05,701 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 00:25:05,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:25:06,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 00:25:06,085 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:25:06,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 00:25:06,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 00:25:06,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 00:25:06,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 00:25:06,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 00:25:06,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 00:25:06,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 00:25:06,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:25:06,220 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 00:25:06,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:25:06,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:25:06,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:25:06,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:25:06,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:25:06,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:25:06,357 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 00:25:06,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:25:06,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:25:06,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:25:06,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:06,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:25:06,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:06,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:06,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:06,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:06,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:06,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:06,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:06,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:25:06,586 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 00:25:06,596 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L789 2022-02-08 00:25:06,596 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 5 queries submitted, 0 failed, 590 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:26:06,631 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.120.170:8091 2022-02-08 00:26:06,631 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:26:06,645 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.120.170:8091 is running 2022-02-08 00:26:06,660 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.126:8091 2022-02-08 00:26:06,661 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:26:06,674 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.126:8091 is running 2022-02-08 00:26:06,688 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.128:8091 2022-02-08 00:26:06,688 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:26:06,703 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.128:8091 is running 2022-02-08 00:26:06,720 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.133:8091 2022-02-08 00:26:06,720 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:26:06,734 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.133:8091 is running 2022-02-08 00:26:06,752 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.135:8091 2022-02-08 00:26:06,753 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:26:06,765 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.135:8091 is running 2022-02-08 00:26:06,779 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.134:8091 2022-02-08 00:26:06,779 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:26:06,795 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.134:8091 is running 2022-02-08 00:26:06,811 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.127:8091 2022-02-08 00:26:06,812 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:26:06,825 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.127:8091 is running 2022-02-08 00:26:06,851 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.124:8091 2022-02-08 00:26:06,851 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:26:06,865 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.124:8091 is running 2022-02-08 00:26:06,891 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.130:8091 2022-02-08 00:26:06,891 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:26:06,905 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.130:8091 is running 2022-02-08 00:26:06,920 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.132:8091 2022-02-08 00:26:06,921 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:26:06,938 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.132:8091 is running 2022-02-08 00:26:06,957 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.115:8091 2022-02-08 00:26:06,957 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:26:06,970 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.115:8091 is running 2022-02-08 00:26:06,986 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.123:8091 2022-02-08 00:26:06,986 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:26:07,000 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.123:8091 is running 2022-02-08 00:26:07,022 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.116:8091 2022-02-08 00:26:07,023 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:26:07,043 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.116:8091 is running 2022-02-08 00:26:07,072 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.129:8091 2022-02-08 00:26:07,072 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:26:07,094 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.129:8091 is running 2022-02-08 00:26:07,118 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.131:8091 2022-02-08 00:26:07,118 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 00:26:07,134 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.131:8091 is running 2022-02-08 00:26:07,137 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:26:07,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:26:08,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:26:08,441 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:26:08,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:26:08,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:26:08,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:26:08,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:26:08,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:26:08,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:26:08,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:26:08,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:26:08,582 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 00:26:08,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:26:08,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:08,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:26:08,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:09,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:09,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:26:09,184 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 00:26:09,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:26:09,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:09,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:26:09,309 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 00:26:09,309 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:26:09,312 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:26:09,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:26:09,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:26:10,015 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:26:10,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:26:10,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:26:10,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:26:10,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:26:10,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:26:10,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:26:10,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:26:10,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:26:10,121 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 00:26:10,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:26:10,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:10,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:26:10,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:10,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:10,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:26:10,752 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 00:26:10,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:26:10,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:10,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:26:10,859 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 00:26:10,859 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:26:10,861 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:26:11,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:26:11,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:26:11,480 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:26:11,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:26:11,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:26:11,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:26:11,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:26:11,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:26:11,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:26:11,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:26:11,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:26:11,698 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 00:26:11,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:26:11,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:11,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:26:11,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:12,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:12,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:26:12,201 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 00:26:12,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:26:12,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:12,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:26:12,295 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 00:26:12,295 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:26:12,298 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 00:26:12,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:26:12,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 00:26:12,700 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:26:12,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 00:26:12,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 00:26:12,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 00:26:12,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 00:26:12,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 00:26:12,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 00:26:12,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 00:26:12,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:26:12,841 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 00:26:12,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:26:12,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:12,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:26:12,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:12,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:12,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:26:12,964 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 00:26:12,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:26:12,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:12,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:26:13,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:13,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:26:13,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:13,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:13,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:13,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:13,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:13,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:13,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:13,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:13,164 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 00:26:13,167 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:26:13,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:26:13,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:26:13,585 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:26:13,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:26:13,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:26:13,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:26:13,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:26:13,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:26:13,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:26:13,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:26:13,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:26:13,709 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 00:26:13,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:26:13,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:13,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:26:13,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:13,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:13,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:26:13,862 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 00:26:13,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:26:13,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:13,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:26:13,908 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 00:26:13,910 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:26:13,911 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 00:26:13,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:26:14,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 00:26:14,265 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:26:14,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 00:26:14,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 00:26:14,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 00:26:14,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 00:26:14,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 00:26:14,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 00:26:14,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 00:26:14,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:26:14,369 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 00:26:14,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:26:14,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:14,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:26:14,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:14,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:14,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:26:14,493 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 00:26:14,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:26:14,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:14,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:26:14,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:14,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:26:14,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:14,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:14,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:14,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:14,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:14,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:14,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:14,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:14,701 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 00:26:14,704 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:26:14,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:26:15,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:26:15,071 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:26:15,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:26:15,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:26:15,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:26:15,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:26:15,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:26:15,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:26:15,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:26:15,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:26:15,243 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 00:26:15,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:26:15,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:15,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:26:15,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:15,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:15,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:26:16,005 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 00:26:16,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:26:16,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:16,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:26:16,112 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 00:26:16,112 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:26:16,115 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:26:16,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:26:16,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:26:16,642 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:26:16,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:26:16,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:26:16,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:26:16,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:26:16,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:26:16,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:26:16,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:26:16,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:26:16,831 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 00:26:16,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:26:16,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:17,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:26:17,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:17,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:17,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:26:17,526 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 00:26:17,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:26:17,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:17,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:26:17,605 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 00:26:17,605 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:26:17,608 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 00:26:17,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:26:17,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 00:26:18,062 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:26:18,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 00:26:18,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 00:26:18,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 00:26:18,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 00:26:18,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 00:26:18,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 00:26:18,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 00:26:18,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:26:18,252 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 00:26:18,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:26:18,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:18,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:26:18,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:18,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:18,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:26:18,420 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 00:26:18,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:26:18,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:18,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:26:18,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:18,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:26:18,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:18,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:18,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:18,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:18,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:18,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:18,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:18,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:18,726 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 00:26:18,730 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 00:26:18,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:26:19,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 00:26:19,132 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:26:19,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 00:26:19,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 00:26:19,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 00:26:19,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 00:26:19,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 00:26:19,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 00:26:19,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 00:26:19,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:26:19,279 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 00:26:19,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:26:19,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:19,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:26:19,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:19,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:19,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:26:19,397 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 00:26:19,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:26:19,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:19,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:26:19,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:19,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:26:19,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:19,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:19,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:19,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:19,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:19,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:19,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:19,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:19,611 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 00:26:19,614 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:26:19,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:26:20,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:26:20,338 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:26:20,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:26:20,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:26:20,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:26:20,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:26:20,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:26:20,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:26:20,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:26:20,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:26:20,578 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 00:26:20,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:26:20,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:20,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:26:20,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:21,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:21,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:26:21,197 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 00:26:21,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:26:21,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:21,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:26:21,236 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 00:26:21,237 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:26:21,240 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:26:21,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:26:21,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:26:21,726 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:26:21,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:26:21,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:26:21,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:26:21,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:26:21,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:26:21,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:26:21,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:26:21,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:26:21,980 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 00:26:22,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:26:22,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:22,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:26:22,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:22,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:22,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:26:22,709 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 00:26:22,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:26:22,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:22,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:26:22,828 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 00:26:22,828 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:26:22,839 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:26:22,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:26:23,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:26:23,233 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:26:23,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:26:23,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:26:23,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:26:23,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:26:23,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:26:23,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:26:23,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:26:23,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:26:23,338 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 00:26:23,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:26:23,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:23,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:26:23,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:23,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:23,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:26:23,630 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 00:26:23,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:26:23,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:23,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:26:23,668 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 00:26:23,668 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:26:23,671 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:26:23,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:26:23,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:26:24,048 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:26:24,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:26:24,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:26:24,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:26:24,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:26:24,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:26:24,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:26:24,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:26:24,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:26:24,155 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 00:26:24,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:26:24,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:24,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:26:24,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:24,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:24,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:26:24,678 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 00:26:24,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:26:24,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:24,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:26:24,809 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 00:26:24,809 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:26:24,812 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 00:26:24,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:26:25,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 00:26:25,227 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:26:25,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 00:26:25,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 00:26:25,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 00:26:25,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 00:26:25,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 00:26:25,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 00:26:25,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 00:26:25,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:26:25,358 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 00:26:25,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:26:25,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:25,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:26:25,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:25,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:26:25,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:26:25,487 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 00:26:25,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:26:25,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:26:25,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:26:25,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:25,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:26:25,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:25,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:25,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:25,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:25,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:25,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:25,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:25,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:26:25,724 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 00:26:25,730 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[]_1644308560.76 2022-02-08 00:26:25,730 | test | INFO | MainThread | [opd:abort_rebalance:808] Restarting Rebalance after killing at 40 2022-02-08 00:26:25,796 | infra | DEBUG | MainThread | [task:__init__:249] Last known rebalance status_id: 1896465d088ea797077b7aa29ce93ff7 2022-02-08 00:26:25,796 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[]_1644308785.73 2022-02-08 00:26:25,796 | infra | INFO | pool-3-thread-8 | [task:start_task:91] Thread 'Rebalance_task_IN=[]_OUT=[]_1644308785.73' started 2022-02-08 00:26:25,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L813 2022-02-08 00:26:25,805 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2022-02-08 00:26:25,831 | test | DEBUG | pool-3-thread-8 | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.128,ns_1@172.23.121.115,ns_1@172.23.121.132,ns_1@172.23.121.135,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.134,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123,ns_1@172.23.121.131,ns_1@172.23.121.130,ns_1@172.23.121.133', 'ejectedNodes': '', 'user': 'Administrator'} 2022-02-08 00:26:25,891 | test | DEBUG | pool-3-thread-8 | [rest_client:rebalance:1459] Rebalance operation started 2022-02-08 00:26:25,900 | test | INFO | pool-3-thread-8 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.121.128 | kv | 7.1.0-2255-enterprise | 58.574754268 | Cluster node | | 172.23.121.115 | kv | 7.1.0-2255-enterprise | 53.3899395343 | Cluster node | | 172.23.121.132 | index, n1ql | 7.1.0-2255-enterprise | 9.64751958225 | Cluster node | | 172.23.121.135 | kv | 7.1.0-2255-enterprise | 43.6036512766 | Cluster node | | 172.23.120.170 | kv | 7.1.0-2255-enterprise | 52.3100815323 | Cluster node | | 172.23.121.129 | kv | 7.1.0-2255-enterprise | 52.9373789542 | Cluster node | | 172.23.121.124 | kv | 7.1.0-2255-enterprise | 53.8174327282 | Cluster node | | 172.23.121.126 | kv | 7.1.0-2255-enterprise | 54.8573631457 | Cluster node | | 172.23.121.134 | index, n1ql | 7.1.0-2255-enterprise | 11.879021879 | Cluster node | | 172.23.121.116 | kv | 7.1.0-2255-enterprise | 50.5931430061 | Cluster node | | 172.23.121.127 | kv | 7.1.0-2255-enterprise | 55.6210742213 | Cluster node | | 172.23.121.123 | kv | 7.1.0-2255-enterprise | 53.0798969072 | Cluster node | | 172.23.121.131 | index, n1ql | 7.1.0-2255-enterprise | 10.7902539642 | Cluster node | | 172.23.121.130 | index, n1ql | 7.1.0-2255-enterprise | 16.8229166667 | Cluster node | | 172.23.121.133 | index, n1ql | 7.1.0-2255-enterprise | 11.5120051914 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2022-02-08 00:26:25,913 | infra | DEBUG | pool-3-thread-8 | [task:call:316] New rebalance status_id: e094f17581210abdc47528250bf6d6a1 2022-02-08 00:26:25,924 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 00:26:25,924 | test | INFO | pool-3-thread-8 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 00:26:25,970 | infra | DEBUG | pool-3-thread-8 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:26:25,973 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:26:25,973 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:26:35,990 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.98 % 2022-02-08 00:26:35,990 | test | INFO | pool-3-thread-8 | [task:check:414] Rebalance - status: running, progress: 4.98 2022-02-08 00:26:36,045 | infra | DEBUG | pool-3-thread-8 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:26:36,048 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:26:36,048 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:26:46,062 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.24 % 2022-02-08 00:26:46,063 | test | INFO | pool-3-thread-8 | [task:check:414] Rebalance - status: running, progress: 6.24 2022-02-08 00:26:46,091 | infra | DEBUG | pool-3-thread-8 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:26:46,092 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:26:46,092 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 661 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:26:56,108 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.91 % 2022-02-08 00:26:56,108 | test | INFO | pool-3-thread-8 | [task:check:414] Rebalance - status: running, progress: 8.91 2022-02-08 00:26:56,138 | infra | DEBUG | pool-3-thread-8 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:26:56,140 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:26:56,140 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:27:06,158 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.19 % 2022-02-08 00:27:06,158 | test | INFO | pool-3-thread-8 | [task:check:414] Rebalance - status: running, progress: 13.19 2022-02-08 00:27:06,207 | infra | DEBUG | pool-3-thread-8 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:27:06,210 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:27:06,210 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:27:16,255 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.72 % 2022-02-08 00:27:16,255 | test | INFO | pool-3-thread-8 | [task:check:414] Rebalance - status: running, progress: 15.72 2022-02-08 00:27:16,299 | infra | DEBUG | pool-3-thread-8 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:27:16,302 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:27:16,302 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:27:26,323 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.84 % 2022-02-08 00:27:26,325 | test | INFO | pool-3-thread-8 | [task:check:414] Rebalance - status: running, progress: 16.84 2022-02-08 00:27:26,361 | infra | DEBUG | pool-3-thread-8 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:27:26,365 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:27:26,365 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:27:36,380 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.37 % 2022-02-08 00:27:36,381 | test | INFO | pool-3-thread-8 | [task:check:414] Rebalance - status: running, progress: 19.37 2022-02-08 00:27:36,431 | infra | DEBUG | pool-3-thread-8 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:27:36,434 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:27:36,434 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:27:46,461 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.49 % 2022-02-08 00:27:46,463 | test | INFO | pool-3-thread-8 | [task:check:414] Rebalance - status: running, progress: 20.49 2022-02-08 00:27:46,509 | infra | DEBUG | pool-3-thread-8 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:27:46,513 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:27:46,513 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 734 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:27:56,529 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.89 % 2022-02-08 00:27:56,529 | test | INFO | pool-3-thread-8 | [task:check:414] Rebalance - status: running, progress: 21.89 2022-02-08 00:27:56,558 | infra | DEBUG | pool-3-thread-8 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:27:56,559 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:27:56,559 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:28:06,575 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.02 % 2022-02-08 00:28:06,575 | test | INFO | pool-3-thread-8 | [task:check:414] Rebalance - status: running, progress: 23.02 2022-02-08 00:28:06,602 | infra | DEBUG | pool-3-thread-8 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:28:06,605 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:28:06,605 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:28:16,622 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.02 % 2022-02-08 00:28:16,622 | test | INFO | pool-3-thread-8 | [task:check:414] Rebalance - status: running, progress: 23.02 2022-02-08 00:28:16,654 | infra | DEBUG | pool-3-thread-8 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:28:16,657 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:28:16,657 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:28:25,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.54 % 2022-02-08 00:28:25,825 | test | INFO | MainThread | [opd:abort_rebalance:814] Rebalance % = 25.54 2022-02-08 00:28:25,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.54 % 2022-02-08 00:28:25,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 00:28:25,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:28:26,670 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.54 % 2022-02-08 00:28:26,670 | test | INFO | pool-3-thread-8 | [task:check:414] Rebalance - status: running, progress: 25.54 2022-02-08 00:28:26,714 | infra | DEBUG | pool-3-thread-8 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:28:26,717 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:28:26,717 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:28:35,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.54 % 2022-02-08 00:28:35,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 00:28:35,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:28:36,733 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.54 % 2022-02-08 00:28:36,733 | test | INFO | pool-3-thread-8 | [task:check:414] Rebalance - status: running, progress: 25.54 2022-02-08 00:28:36,783 | infra | DEBUG | pool-3-thread-8 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:28:36,786 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:28:36,786 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:28:45,885 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.54 % 2022-02-08 00:28:45,892 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 00:28:45,892 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:28:46,799 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.54 % 2022-02-08 00:28:46,799 | test | INFO | pool-3-thread-8 | [task:check:414] Rebalance - status: running, progress: 25.54 2022-02-08 00:28:46,829 | infra | DEBUG | pool-3-thread-8 | [task:check:441] Wait before next rebalance progress check 2022-02-08 00:28:46,832 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 00:28:46,832 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 805 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:28:55,914 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-08 00:28:55,914 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:28:56,848 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-08 00:28:56,848 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:29:00,933 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 00:29:00,933 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:29:01,865 | test | INFO | pool-3-thread-8 | [task:check:414] Rebalance - status: none, progress: 100 2022-02-08 00:29:01,921 | test | INFO | pool-3-thread-8 | [task:check:473] Rebalance completed with progress: 100% in 156.030999899 sec 2022-02-08 00:29:01,923 | infra | INFO | pool-3-thread-8 | [task:complete_task:106] Thread 'Rebalance_task_IN=[]_OUT=[]_1644308785.73' completed 2022-02-08 00:29:10,937 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >100% in 45.111000061 seconds 2022-02-08 00:29:10,970 | test | DEBUG | MainThread | [rest_client:diag_eval:1476] /diag/eval status on 172.23.120.170:8091: 1 content: false command: ns_orchestrator:needs_rebalance() 2022-02-08 00:29:10,996 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-08 00:29:10,996 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 00:29:16,030 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 00:29:16,032 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 00:29:26,032 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >100% in 15.0620000362 seconds 2022-02-08 00:29:26,052 | test | DEBUG | MainThread | [rest_client:diag_eval:1476] /diag/eval status on 172.23.120.170:8091: 1 content: false command: ns_orchestrator:needs_rebalance() 2022-02-08 00:29:26,052 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[]_1644308785.73 2022-02-08 00:29:26,164 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3970] Stats: {u'opsPerSec': 22236, u'diskFetches': 394258, u'dataUsed': 52189290587L, u'itemCount': 40005370, u'quotaPercentUsed': 69.11357611024066, u'storageTotals': {u'hdd': {u'usedByData': 92190498657L, u'used': 197056022193L, u'quotaTotal': 1298666618880L, u'total': 1298666618880L, u'free': 952816709230L}, u'ram': {u'usedByData': 59795687760L, u'used': 115936788480L, u'quotaTotal': 87566581760L, u'total': 122941186048L, u'quotaUsed': 86518005760L, u'quotaUsedPerNode': 8651800576L, u'quotaTotalPerNode': 8756658176L}}, u'diskUsed': 92190498657L, u'vbActiveNumNonResident': 2158778, u'memUsed': 59795687760L} 2022-02-08 00:29:26,690 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 40005370 | 80.58 GiB | 55.69 GiB | 85.86 GiB | 94.6086081036 | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+---------------+ 2022-02-08 00:29:26,690 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+------------------+------------------+-----------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+------------------+------------------+-----------------+---------------+ | 40000000 | 380000000 | 2000000-20000000 | 2000000-20000000 | 1000000-2000000 | 0-0 | +---------------+---------------+------------------+------------------+-----------------+---------------+ 2022-02-08 00:29:27,020 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 94.6086081036 2022-02-08 00:29:27,023 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:29:27,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:27,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:29:27,401 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:27,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:29:27,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:29:27,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:29:27,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:29:27,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:29:27,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:29:27,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:29:27,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:29:27,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:29:27,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:29:27,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:29:27,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:29:28,026 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:29:28,028 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:29:28,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:28,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:29:28,364 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:28,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:29:28,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:29:28,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:29:28,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:29:28,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:29:28,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:29:28,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:29:28,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:29:28,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:29:28,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:29:28,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:29:28,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:29:28,986 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:29:28,989 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:29:29,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:29,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:29:29,331 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:29,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:29:29,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:29:29,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:29:29,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:29:29,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:29:29,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:29:29,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:29:29,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:29:29,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:29:29,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:29:29,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:29:29,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:29:31,710 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:29:31,714 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 00:29:31,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:32,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 00:29:32,108 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:32,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 00:29:32,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 00:29:32,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 00:29:32,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 00:29:32,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 00:29:32,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 00:29:32,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 00:29:32,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:29:32,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:29:32,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:29:32,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:29:32,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:29:32,286 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 00:29:32,288 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:29:32,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:32,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:29:32,655 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:32,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:29:32,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:29:32,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:29:32,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:29:32,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:29:32,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:29:32,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:29:32,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:29:32,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:29:32,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:29:32,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:29:32,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:29:33,200 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:29:33,203 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 00:29:33,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:33,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 00:29:33,562 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:33,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 00:29:33,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 00:29:33,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 00:29:33,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 00:29:33,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 00:29:33,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 00:29:33,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 00:29:33,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:29:33,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:29:33,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:29:33,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:29:33,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:29:33,809 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 00:29:33,812 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:29:33,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:34,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:29:34,151 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:34,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:29:34,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:29:34,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:29:34,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:29:34,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:29:34,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:29:34,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:29:34,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:29:34,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:29:34,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:29:34,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:29:34,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:29:34,997 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:29:35,000 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:29:35,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:35,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:29:35,483 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:35,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:29:35,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:29:35,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:29:35,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:29:35,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:29:35,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:29:35,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:29:35,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:29:35,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:29:35,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:29:35,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:29:36,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:29:36,319 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:29:36,322 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 00:29:36,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:36,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 00:29:36,693 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:36,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 00:29:36,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 00:29:36,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 00:29:36,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 00:29:36,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 00:29:36,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 00:29:36,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 00:29:36,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:29:36,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:29:36,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:29:36,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:29:36,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:29:36,894 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 00:29:36,895 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 00:29:36,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:37,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 00:29:37,276 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:37,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 00:29:37,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 00:29:37,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 00:29:37,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 00:29:37,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 00:29:37,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 00:29:37,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 00:29:37,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:29:37,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:29:37,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:29:37,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:29:37,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:29:37,500 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 00:29:37,503 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:29:37,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:37,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:29:38,039 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:38,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:29:38,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:29:38,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:29:38,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:29:38,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:29:38,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:29:38,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:29:38,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:29:38,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:29:38,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:29:38,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:29:38,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:29:38,934 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:29:38,937 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:29:39,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:39,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:29:39,306 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:39,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:29:39,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:29:39,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:29:39,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:29:39,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:29:39,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:29:39,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:29:39,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:29:39,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:29:39,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:29:39,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:29:39,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:29:40,033 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:29:40,036 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:29:40,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:40,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:29:40,407 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:40,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:29:40,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:29:40,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:29:40,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:29:40,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:29:40,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:29:40,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:29:40,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:29:40,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:29:40,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:29:40,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:29:40,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:29:41,220 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:29:41,223 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:29:41,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:41,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:29:41,766 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:41,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:29:41,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:29:41,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:29:41,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:29:41,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:29:41,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:29:41,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:29:41,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:29:42,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:29:42,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:29:42,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:29:42,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:29:42,503 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:29:42,506 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 00:29:42,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:42,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 00:29:42,858 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:42,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 00:29:42,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 00:29:42,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 00:29:42,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 00:29:42,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 00:29:42,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 00:29:42,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 00:29:43,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 00:29:43,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 00:29:43,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 00:29:43,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 00:29:43,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 00:29:43,121 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 00:29:43,121 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for kvstore is 83424MB 2022-02-08 00:29:43,121 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for wal is 4958MB 2022-02-08 00:29:43,121 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:642] Total Disk usage for keyTree is 7046MB 2022-02-08 00:29:43,121 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:643] Total Disk usage for seqTree is 76068MB 2022-02-08 00:29:43,124 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:29:43,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:43,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:29:43,489 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:43,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:29:43,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:29:43,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:29:43,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:29:43,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:29:43,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:29:43,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:29:43,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:29:43,640 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.115 - core(s): 8 2022-02-08 00:29:43,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:29:44,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:29:44,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:29:44,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:29:44,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:29:44,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:29:44,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:29:44,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:29:45,049 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:29:45,051 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:29:45,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:45,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:29:45,453 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:45,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:29:45,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:29:45,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:29:45,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:29:45,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:29:45,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:29:45,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:29:45,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:29:45,647 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.116 - core(s): 8 2022-02-08 00:29:45,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:29:46,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:29:46,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:29:46,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:29:46,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:29:46,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:29:46,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:29:47,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:29:47,177 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:29:47,181 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:29:47,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:47,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:29:47,582 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:47,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:29:47,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:29:47,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:29:47,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:29:47,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:29:47,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:29:47,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:29:47,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:29:47,730 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.123 - core(s): 8 2022-02-08 00:29:47,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:29:48,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:29:48,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:29:48,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:29:48,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:29:48,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 5 queries submitted, 0 failed, 858 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:29:49,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:29:49,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:29:49,285 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:29:49,288 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:29:49,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:49,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:29:49,647 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:49,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:29:49,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:29:49,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:29:49,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:29:49,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:29:49,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:29:49,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:29:49,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:29:49,805 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.124 - core(s): 8 2022-02-08 00:29:49,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:29:50,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:29:50,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:29:50,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:29:50,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:29:50,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:29:51,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:29:51,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:29:51,825 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:29:51,826 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:29:51,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:52,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:29:52,338 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:52,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:29:52,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:29:52,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:29:52,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:29:52,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:29:52,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:29:52,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:29:52,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:29:52,503 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.126 - core(s): 8 2022-02-08 00:29:52,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:29:53,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:29:53,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:29:53,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:29:53,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:29:53,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:29:53,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:29:53,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:29:54,043 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:29:54,055 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:29:54,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:54,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:29:54,403 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:54,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:29:54,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:29:54,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:29:54,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:29:54,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:29:54,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:29:54,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:29:54,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:29:54,556 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.127 - core(s): 8 2022-02-08 00:29:54,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:29:55,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:29:55,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:29:55,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:29:55,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:29:55,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:29:56,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:29:56,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:29:56,755 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:29:56,757 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:29:56,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:57,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:29:57,226 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:57,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:29:57,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:29:57,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:29:57,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:29:57,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:29:57,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:29:57,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:29:57,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:29:57,414 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.128 - core(s): 8 2022-02-08 00:29:57,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:29:58,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:29:58,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:29:58,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:29:58,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:29:58,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:29:59,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:29:59,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:29:59,377 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:29:59,380 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:29:59,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:29:59,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:29:59,819 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:29:59,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:29:59,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:29:59,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:29:59,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:29:59,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:29:59,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:29:59,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:29:59,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:29:59,983 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.129 - core(s): 8 2022-02-08 00:29:59,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:30:00,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:30:00,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:30:00,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:30:01,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:30:01,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:30:01,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:30:01,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:30:01,677 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:30:01,680 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:30:01,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:30:02,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:30:02,138 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:30:02,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:30:02,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:30:02,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:30:02,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:30:02,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:30:02,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:30:02,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:30:02,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:30:02,377 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.135 - core(s): 8 2022-02-08 00:30:02,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:30:02,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:30:02,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:30:03,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:30:03,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:30:03,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:30:03,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:30:03,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:30:03,834 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:30:03,835 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:30:03,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:30:04,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:30:04,272 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:30:04,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:30:04,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:30:04,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:30:04,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:30:04,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:30:04,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:30:04,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:30:04,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 00:30:04,460 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.120.170 - core(s): 8 2022-02-08 00:30:04,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 00:30:05,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 00:30:05,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 00:30:05,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 00:30:05,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 00:30:05,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 00:30:05,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 00:30:06,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 00:30:06,250 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:30:06,252 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:723] {'172.23.121.115': {'rw_0:magma': {u'NReadBytesGetPerSec': 18363706.0, u'ReplayWALOffset': u'227:12824576', u'NTablesCreated': 8033, u'RecentReadIOAmpSet': 0.06795273721218109, u'ReadIOAmp': 747.8624877929688, u'BlockCacheMisses': 179970, u'NFlushes': 3894, u'RecentWriteAmp': 4.523878574371338, u'NumSyncFlushes': 109, u'TotalMemUsed': 73442055, u'NBloomFilterHits': 23043002, u'Fragmentation': 0.3334708511829376, u'NumNormalFlushes': 667, u'RecentBlockCacheHitRatio': 91.65525817871094, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 4907, u'ReadIOAmp': 1.0032938880078077, u'MemtablesMemUsed': 9083872, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 9045761, u'CheckpointOverhead': 53633024, u'NItemsGet': 8197, u'NWriteIOs': 14626, u'NReadBytes': 3022520997L, u'LevelStats': [{u'TargetSize': 113246208, u'LogicalSize': 36713439, u'SizeRatio': 0.3241913318634033, u'NWriteBytes': 77225984, u'BloomFilterMemUsed': 0, u'NumTables': 184, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2207744}, {u'TargetSize': 113246208, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 113246208, u'LogicalSize': 46822447, u'SizeRatio': 0.41345708458511915, u'NWriteBytes': 65503232, u'BloomFilterMemUsed': 0, u'NumTables': 34, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 950272}, {u'TargetSize': 480544037, u'LogicalSize': 337670588, u'SizeRatio': 0.7026839623441212, u'NWriteBytes': 58138624, u'BloomFilterMemUsed': 0, u'NumTables': 163, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 5640192}, {u'TargetSize': 1910151266, u'LogicalSize': 961088082, u'SizeRatio': 0.5031476297752012, u'NWriteBytes': 3845910528L, u'BloomFilterMemUsed': 0, u'NumTables': 333, u'WastedSpace': 6012449, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 961032192}], u'NItemsGarbageCollected': 2771872, u'NInternalFragmentationCompacts': 9, u'NTTLCompacts': 21, u'ActiveIndexBlocksSize': 7004010, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 93984, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 1132611, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 276078, u'CompactWriter': {u'NDataBlockWrites': 546975, u'NWriteBytes': 1630208000, u'NIndexBlockWrites': 6599, u'NBytesIncoming': 2558865791L, u'NSyncs': 0, u'NInserts': 6581872, u'NWriteIOs': 5314}, u'NonResidentBloomFilterSize': 0, u'IteratorMemUsed': 0, u'NWriteBytes': 3988025932L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 8224, u'NReadBytes': 26582818, u'NDataBlockReads': 8197, u'NBytesOutgoing': 0, u'NIndexBlockReads': 16394}, u'NBloomFilterMisses': 0, u'NItemsMerged': 2739444, u'LogicalDiskSize': 961088082, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.9581587478878085, u'BufferMemUsed': 1037312, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 623417494, u'NTableFiles': 878, u'NFileCountCompacts': 2, u'TableObjectMemUsed': 269040, u'TreeSnapshotMemoryUsed': 188736, u'TotalDiskUsage': 1229168640, u'BytesIncoming': 4162176613L, u'NItemsDel': 0, u'PendingCheckpointBytes': 380928, u'NCompacts': 1882, u'NItemsPut': 6120868, u'NSyncs': 3227, u'CompactReader': {u'NReadIOs': 1121307, u'NReadBytes': 2991301642L, u'NDataBlockReads': 1110097, u'NBytesOutgoing': 0, u'NIndexBlockReads': 11216}, u'ActiveDiskUsage': 969830400, u'NTablesDeleted': 3252, u'IterateReader': {u'NReadIOs': 3080, u'NReadBytes': 4636537, u'NDataBlockReads': 1815, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1265}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 897180, u'NWriteBytes': 2357817932L, u'NIndexBlockWrites': 9567, u'NBytesIncoming': 4215310804L, u'NSyncs': 3227, u'NInserts': 7739801, u'NWriteIOs': 9312}}, u'NReadBytes': 1169385335742L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 63.0, u'NInternalFragmentationCompacts': 9, u'NTTLCompacts': 198, u'ActiveIndexBlocksSize': 7898319, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 209920, u'NTxns': 66131, u'NWriteBytes': 4741808128L, u'NSegments': 229, u'NSyncs': 66131, u'NLoggedBytes': 4741808128L, u'NActiveSegments': 3, u'NWrites': 93718, u'PreallocateSupported': True, u'NInserts': 5376317, u'DiskUsed': 38821888}, u'NGets': 519860, u'NReadBytesSet': 273435052, u'NReadBytesCompact': 3180691873L, u'IndexResidentRatio': 0.9524017572402954, u'CompactQueueSize': 0, u'NSyncsPerSec': 2.0999999046325684, u'BlockCacheHitRatio': 97.44470977783203, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 1166200007332L, u'BloomFilterMemUsedRatio': 0.013009120710194111, u'NonResidentBloomFilterSize': 281807, u'NWriteBytes': 9263367062L, u'BytesOutgoing': 7358217, u'RecentReadIOAmp': 84.89690399169922, u'WriteAmp': 2.2470288276672363, u'NBlocksCached': 16772, u'LogicalDataSize': 674923431, u'ActiveBloomFilterMemUsed': 1406904, u'NFileCountCompacts': 90, u'NTableFiles': 1613, u'FlushQueueSize': 8, u'TableObjectMemUsed': 445440, u'BlockCacheQuota': 79268924, u'TotalDiskUsage': 1320800256, u'NPurgerRuns': 85, u'ReadIOAmpSet': 0.0, u'NSyncs': 9681, u'NReadIOsPerSec': 6651.43310546875, u'NReadIOsSetPerSec': 33.93333435058594, u'NTablesDeleted': 5839, u'BloomFilterFPR': 0.01572263613343239, u'NBloomFilterHitsPerSec': 5723.13330078125, u'WriteCacheMemUsed': 10965920, u'RecentBloomFilterFPR': 0.01405183132737875, u'BytesPerRead': 2242770.25, u'BytesIncomingPerSec': 246370.859375, u'NReadIOsGetPerSec': 6313.5, u'NumKVStores': 27, u'NGetStats': 21295, u'TotalIndexBlocksSize': 10136253, u'NGetStatsPerSec': 0.9666666388511658, u'NWriteBytesCompact': 1840390144, u'NReadIOsSet': 86294, u'ActiveDataSize': 674923431, u'NWriteBytesCompactPerSec': 700689.0625, u'NBloomFilterMissesPerSec': 81.56666564941406, u'NCommitBatches': 61877, u'NGetStatsComputed': 21295, u'NWriteIOs': 118511, u'NReadIOsGet': 388870115, u'RecentReadAmpGet': 1353.051513671875, u'WriteCacheQuota': 24937072, u'LSMTreeObjectMemUsed': 465696, u'NSetsPerSec': 499.3666687011719, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.8666666746139526, u'ManagedFileObjectMemUsed': 164544, u'ReadAmpGet': 158452.328125, u'BlockCacheMemUsed': 57412600, u'NWriterCompacts': 0, u'NReadIOs': 390045237, u'WALBufferMemUsed': 327680, u'ReadAmp': 158885.21875, u'keyStats': {u'NItemsTransformed': 4633734, u'NTablesCreated': 1589, u'ReadIOAmp': 68.69776726981456, u'MemtablesMemUsed': 1580000, u'NBloomFilterHits': 23042984, u'TotalIndexBlocksSize': 1073084, u'CheckpointOverhead': 16285696, u'NItemsGet': 5660469, u'NWriteIOs': 5403, u'NReadBytes': 1166361511978L, u'LevelStats': [{u'TargetSize': 113246208, u'LogicalSize': 21523700, u'SizeRatio': 0.19006111003734447, u'NWriteBytes': 312164352, u'BloomFilterMemUsed': 393656, u'NumTables': 242, u'WastedSpace': 7857, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 21073920}, {u'TargetSize': 113246208, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 113246208, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 0}, {u'TargetSize': 113246208, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 0}, {u'TargetSize': 113246208, u'LogicalSize': 51236149, u'SizeRatio': 0.4524314756746645, u'NWriteBytes': 206036992, u'BloomFilterMemUsed': 1008512, u'NumTables': 28, u'WastedSpace': 120791, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 51355648}], u'NItemsGarbageCollected': 3118552, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 86, u'ActiveIndexBlocksSize': 881127, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 36384, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 388909997, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 1704317, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 147849, u'CompactWriter': {u'NDataBlockWrites': 71613, u'NWriteBytes': 209076224, u'NIndexBlockWrites': 1049, u'NBytesIncoming': 270081759, u'NSyncs': 0, u'NInserts': 4636269, u'NWriteIOs': 515}, u'NonResidentBloomFilterSize': 280580, u'IteratorMemUsed': 304, u'NWriteBytes': 519790953, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 388861582, u'NReadBytes': 1166173267364L, u'NDataBlockReads': 9020183194L, u'NBytesOutgoing': 0, u'NIndexBlockReads': 17576886741L}, u'NBloomFilterMisses': 368084, u'NItemsMerged': 0, u'LogicalDiskSize': 51236149, u'TotalBloomFilterMemUsed': 1402168, u'WriteAmp': 2.261364835131714, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1402168, u'LogicalDataSize': 51236149, u'NTableFiles': 379, u'NFileCountCompacts': 44, u'TableObjectMemUsed': 90960, u'TreeSnapshotMemoryUsed': 143936, u'TotalDiskUsage': 88715264, u'BytesIncoming': 229857184, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 185, u'NItemsPut': 4584188, u'NSyncs': 3227, u'CompactReader': {u'NReadIOs': 48415, u'NReadBytes': 188244614, u'NDataBlockReads': 120475, u'NBytesOutgoing': 0, u'NIndexBlockReads': 2576}, u'ActiveDiskUsage': 72429568, u'NTablesDeleted': 1319, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.01572264879158866, u'Writer': {u'NDataBlockWrites': 71453, u'NWriteBytes': 310714729, u'NIndexBlockWrites': 2039, u'NBytesIncoming': 268231834, u'NSyncs': 3227, u'NInserts': 4535863, u'NWriteIOs': 4888}}, u'NReadBytesCompactPerSec': 930675.5625, u'TableMetaMemUsed': 526111, u'NDeletes': 0, u'BlockCacheHitsPerSec': 691.9666748046875, u'BytesOutgoingPerSec': 13572.06640625, u'NBloomFilterMisses': 368084, u'LogicalDiskSize': 1012594019, u'NSets': 4584188, u'TotalBloomFilterMemUsed': 1406904, u'BufferMemUsed': 1037312, u'RecentReadAmp': 1421.6243896484375, u'NGetsPerSec': 74.36666870117188, u'NInserts': 0, u'NCommitBatchesPerSec': 1.4666666984558105, u'FragmentationWithOverhead': 0.48900413513183594, u'NCompactsPerSec': 0.5333333611488342, u'WALMemUsed': 393296, u'MemoryQuota': 108147512, u'NReadBytesPerSec': 19294380.0, u'TreeSnapshotMemoryUsed': 440224, u'BytesIncoming': 4122496009L, u'NCompacts': 2202, u'NumPersistentFlushes': 3118, u'NWriteIOsPerSec': 8.333333015441895, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 246934.640625, u'ActiveDiskUsage': 1044463616, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 0.9666666388511658, u'NReadBytesSetPerSec': 98715.8671875, u'BlockCacheHits': 6863068, u'NWriteBytesPerSec': 1114551.875, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 1537, u'ReadIOAmp': 0.2125171939477304, u'MemtablesMemUsed': 302048, u'NBloomFilterHits': 18, u'TotalIndexBlocksSize': 17408, u'CheckpointOverhead': 712704, u'NItemsGet': 1454, u'NWriteIOs': 4764, u'NReadBytes': 1302767, u'LevelStats': [{u'TargetSize': 113246208, u'LogicalSize': 2258100, u'SizeRatio': 0.019939740498860676, u'NWriteBytes': 11485184, u'BloomFilterMemUsed': 3872, u'NumTables': 242, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 1982464}, {u'TargetSize': 113246208, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 113246208, u'LogicalSize': 269788, u'SizeRatio': 0.0023823137636537904, u'NWriteBytes': 1105920, u'BloomFilterMemUsed': 864, u'NumTables': 27, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 221184}], u'NItemsGarbageCollected': 252692, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 91, u'ActiveIndexBlocksSize': 13182, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 34176, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 2629, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 102184, u'CompactWriter': {u'NDataBlockWrites': 135, u'NWriteBytes': 1105920, u'NIndexBlockWrites': 135, u'NBytesIncoming': 299067, u'NSyncs': 0, u'NInserts': 3105, u'NWriteIOs': 135}, u'NonResidentBloomFilterSize': 1227, u'IteratorMemUsed': 0, u'NWriteBytes': 13742049, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 309, u'NReadBytes': 157150, u'NDataBlockReads': 390, u'NBytesOutgoing': 0, u'NIndexBlockReads': 390}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 269788, u'TotalBloomFilterMemUsed': 4736, u'WriteAmp': 0.38922159131527423, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 4736, u'LogicalDataSize': 269788, u'NTableFiles': 356, u'NFileCountCompacts': 44, u'TableObjectMemUsed': 85440, u'TreeSnapshotMemoryUsed': 107552, u'TotalDiskUsage': 2916352, u'BytesIncoming': 35306492, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 135, u'NItemsPut': 256554, u'NSyncs': 3227, u'CompactReader': {u'NReadIOs': 2320, u'NReadBytes': 1145617, u'NDataBlockReads': 1268, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1268}, u'ActiveDiskUsage': 2203648, u'NTablesDeleted': 1268, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 1402, u'NWriteBytes': 12636129, u'NIndexBlockWrites': 1402, u'NBytesIncoming': 1117081, u'NSyncs': 3227, u'NInserts': 16324, u'NWriteIOs': 4629}}}}} 2022-02-08 00:30:06,253 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:728] magma stats fragmentation result {'172.23.121.123': [0.3598615825176239, 0.38387423753738403, 0.362295001745224, 0.3337768018245697, 0.3923378884792328, 0.3345831036567688, 0.3372827172279358, 0.36963096261024475], '172.23.121.124': [0.35166409611701965, 0.3665745258331299, 0.3549863398075104, 0.328989714384079, 0.34095990657806396, 0.3525520861148834, 0.34236159920692444, 0.33356985449790955], '172.23.121.135': [0.025510050356388092, 0.05982702597975731, 0.0, 0.03066325932741165, 0.006257697939872742, 0.06720182299613953, 0.049282852560281754, 0.071332648396492], '172.23.121.115': [0.3334708511829376, 0.36592116951942444, 0.3541037142276764, 0.3480992019176483, 0.3270602822303772, 0.34696048498153687, 0.33600282669067383, 0.34024739265441895], '172.23.121.126': [0.3693874478340149, 0.36834466457366943, 0.35025766491889954, 0.3541148602962494, 0.3541406989097595, 0.35169512033462524, 0.34964972734451294, 0.3566073477268219], '172.23.121.116': [0.37162908911705017, 0.40180492401123047, 0.38663774728775024, 0.40892916917800903, 0.41784748435020447, 0.3993678092956543, 0.4013037383556366, 0.3946928381919861], '172.23.121.127': [0.35707733035087585, 0.36358603835105896, 0.37501588463783264, 0.3529120683670044, 0.39972198009490967, 0.349418044090271, 0.336150199174881, 0.3253394365310669], '172.23.121.128': [0.3672178089618683, 0.34408578276634216, 0.3291299045085907, 0.341987669467926, 0.34729281067848206, 0.3537975251674652, 0.3420118987560272, 0.3508259952068329], '172.23.121.129': [0.3699209988117218, 0.35860586166381836, 0.3406347334384918, 0.36307889223098755, 0.33566388487815857, 0.3645162284374237, 0.3568940758705139, 0.3534667491912842], '172.23.120.170': [0.35037854313850403, 0.3607839047908783, 0.35410380363464355, 0.3462834060192108, 0.3540555536746979, 0.34574222564697266, 0.3555380702018738, 0.3458726704120636]} 2022-02-08 00:30:06,539 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.115 is 35.0 2022-02-08 00:30:06,821 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.116 is 35.0 2022-02-08 00:30:07,107 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.123 is 35.0 2022-02-08 00:30:07,408 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.124 is 35.0 2022-02-08 00:30:07,836 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.126 is 35.0 2022-02-08 00:30:08,332 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.127 is 35.0 2022-02-08 00:30:08,846 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.128 is 35.0 2022-02-08 00:30:09,155 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.129 is 35.0 2022-02-08 00:30:09,490 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.135 is 35.0 2022-02-08 00:30:09,788 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.120.170 is 35.0 2022-02-08 00:30:09,789 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:677] KV stats fragmentation values {'172.23.121.123': 35.0, '172.23.121.124': 35.0, '172.23.121.135': 35.0, '172.23.121.115': 35.0, '172.23.121.126': 35.0, '172.23.121.116': 35.0, '172.23.121.127': 35.0, '172.23.121.128': 35.0, '172.23.121.129': 35.0, '172.23.120.170': 35.0} ############################################################ # # Step 5: Crash Magma/memc with Loading of docs # ############################################################ ############################################################ # # RAM FootPrint: 3886.3515625 # ############################################################ 2022-02-08 00:30:09,941 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 00:30:09,941 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 90 seconds. Reason: Iteration:0 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 905 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:31:39,951 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 00:31:39,951 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 00:31:44,957 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:31:45,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:45,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:31:45,470 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:45,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:31:45,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:31:45,528 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:31:45,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:31:45,571 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:31:45,609 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:31:45,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:31:45,653 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:31:45,653 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 00:31:45,740 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['103538 memcached\n'], []) 2022-02-08 00:31:45,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 00:31:45,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 00:31:45,829 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['103538 memcached\n'], []) 2022-02-08 00:31:45,829 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:31:45,839 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:31:45,931 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:46,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:31:46,232 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:46,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:31:46,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:31:46,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:31:46,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:31:46,285 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:31:46,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:31:46,313 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:31:46,325 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:31:46,326 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 00:31:46,380 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['58004 memcached\n'], []) 2022-02-08 00:31:46,380 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 00:31:46,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 00:31:46,426 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['58004 memcached\n'], []) 2022-02-08 00:31:46,426 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:31:46,428 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:31:46,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:46,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:31:46,864 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:46,864 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:31:46,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:31:46,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:31:46,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:31:46,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:31:46,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:31:46,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:31:46,963 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:31:46,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 00:31:47,022 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['96787 memcached\n'], []) 2022-02-08 00:31:47,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 00:31:47,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 00:31:47,085 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['96787 memcached\n'], []) 2022-02-08 00:31:47,085 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:31:47,088 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:31:47,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:47,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:31:47,479 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:47,479 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:31:47,497 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:31:47,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:31:47,545 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:31:47,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:31:47,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:31:47,599 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:31:47,618 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:31:47,619 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 00:31:47,663 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['81998 memcached\n'], []) 2022-02-08 00:31:47,663 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 00:31:47,688 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 00:31:47,710 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['81998 memcached\n'], []) 2022-02-08 00:31:47,711 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:31:47,713 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:31:47,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:48,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:31:48,063 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:48,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:31:48,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:31:48,091 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:31:48,102 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:31:48,119 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:31:48,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:31:48,147 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:31:48,161 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:31:48,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 00:31:48,207 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['123939 memcached\n'], []) 2022-02-08 00:31:48,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 00:31:48,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 00:31:48,263 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['123939 memcached\n'], []) 2022-02-08 00:31:48,265 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:31:48,266 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:31:48,346 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:48,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:31:48,607 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:48,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:31:48,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:31:48,632 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:31:48,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:31:48,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:31:48,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:31:48,688 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:31:48,703 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:31:48,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 00:31:48,756 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['117556 memcached\n'], []) 2022-02-08 00:31:48,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 00:31:48,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 00:31:48,808 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['117556 memcached\n'], []) 2022-02-08 00:31:48,808 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:31:48,811 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:31:48,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 950 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:31:49,089 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:31:49,145 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:49,145 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:31:49,158 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:31:49,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:31:49,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:31:49,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:31:49,211 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:31:49,224 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:31:49,237 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:31:49,237 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 00:31:49,292 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['27831 memcached\n'], []) 2022-02-08 00:31:49,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 00:31:49,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 00:31:49,336 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['27831 memcached\n'], []) 2022-02-08 00:31:49,336 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:31:49,338 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:31:49,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:49,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:31:49,710 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:49,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:31:49,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:31:49,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:31:49,760 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:31:49,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:31:49,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:31:49,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:31:49,832 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:31:49,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 00:31:49,904 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['22007 memcached\n'], []) 2022-02-08 00:31:49,905 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 00:31:49,931 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 00:31:49,954 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['22007 memcached\n'], []) 2022-02-08 00:31:49,954 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:31:49,957 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:31:50,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:50,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:31:50,319 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:50,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:31:50,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:31:50,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:31:50,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:31:50,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:31:50,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:31:50,405 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:31:50,420 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:31:50,421 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 00:31:50,463 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['92453 memcached\n'], []) 2022-02-08 00:31:50,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 00:31:50,489 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 00:31:50,510 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['92453 memcached\n'], []) 2022-02-08 00:31:50,510 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:31:50,513 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:31:50,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:50,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:31:50,918 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:50,920 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:31:50,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:31:50,946 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:31:50,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:31:50,974 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:31:50,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:31:51,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:31:51,013 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:31:51,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:31:51,081 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['57800 memcached\n'], []) 2022-02-08 00:31:51,081 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 00:31:51,119 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:31:51,142 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['57800 memcached\n'], []) 2022-02-08 00:31:51,144 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:31:51,145 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:31:51,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:51,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:31:51,509 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:51,510 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:31:51,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:31:51,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:31:51,553 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:31:51,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:31:51,586 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:31:51,601 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:31:51,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:31:51,630 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 00:31:51,660 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:31:51,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:51,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:31:51,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:51,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:52,061 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:31:52,075 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 00:31:52,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:31:52,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:52,102 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:31:52,117 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 00:31:52,117 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:31:52,118 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:31:52,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:52,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:31:52,538 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:52,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:31:52,553 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:31:52,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:31:52,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:31:52,599 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:31:52,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:31:52,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:31:52,650 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:31:52,665 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 00:31:52,691 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:31:52,706 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:52,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:31:52,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:53,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:53,178 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:31:53,194 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 00:31:53,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:31:53,209 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:53,224 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:31:53,239 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 00:31:53,239 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:31:53,240 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:31:53,326 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:53,532 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:31:53,592 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:53,594 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:31:53,612 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:31:53,640 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:31:53,655 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:31:53,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:31:53,688 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:31:53,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:31:53,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:31:53,736 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 00:31:53,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:31:53,790 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:53,903 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:31:54,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:54,119 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:54,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:31:54,210 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 00:31:54,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:31:54,224 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:54,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:31:54,262 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 00:31:54,262 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:31:54,263 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 00:31:54,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:54,553 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 00:31:54,609 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:54,609 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 00:31:54,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 00:31:54,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 00:31:54,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 00:31:54,668 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 00:31:54,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 00:31:54,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 00:31:54,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:31:54,740 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 00:31:54,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:31:54,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:54,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:31:54,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:54,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:54,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:31:54,855 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 00:31:54,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:31:54,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:54,882 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:31:54,914 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:54,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:31:54,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:54,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:54,989 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:55,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:55,020 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:55,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:55,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:55,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:55,089 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 00:31:55,092 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:31:55,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:55,385 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:31:55,444 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:55,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:31:55,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:31:55,469 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:31:55,482 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:31:55,497 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:31:55,510 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:31:55,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:31:55,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:31:55,552 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 00:31:55,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:31:55,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:55,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:31:55,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:55,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:55,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:31:55,667 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 00:31:55,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:31:55,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:55,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:31:55,706 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 00:31:55,707 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:31:55,709 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 00:31:55,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:56,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 00:31:56,066 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:56,066 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 00:31:56,081 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 00:31:56,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 00:31:56,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 00:31:56,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 00:31:56,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 00:31:56,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 00:31:56,165 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:31:56,181 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 00:31:56,213 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:31:56,227 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:56,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:31:56,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:56,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:56,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:31:56,308 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 00:31:56,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:31:56,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:56,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:31:56,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:56,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:31:56,400 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:56,414 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:56,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:56,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:56,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:56,474 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:56,487 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:56,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:56,529 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 00:31:56,532 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:31:56,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:56,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:31:56,878 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:56,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:31:56,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:31:56,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:31:56,920 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:31:56,940 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:31:56,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:31:56,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:31:56,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:31:56,994 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 00:31:57,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:31:57,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:57,173 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:31:57,323 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:57,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:57,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:31:57,533 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 00:31:57,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:31:57,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:57,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:31:57,576 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 00:31:57,576 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:31:57,588 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:31:57,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:57,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:31:57,967 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:57,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:31:57,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:31:58,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:31:58,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:31:58,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:31:58,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:31:58,068 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:31:58,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:31:58,104 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 00:31:58,130 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:31:58,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:58,276 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:31:58,413 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:58,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:58,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:31:58,628 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 00:31:58,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:31:58,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:58,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:31:58,676 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 00:31:58,677 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:31:58,678 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 00:31:58,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:59,007 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 00:31:59,069 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:59,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 00:31:59,084 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 00:31:59,096 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 00:31:59,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 00:31:59,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 00:31:59,135 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 00:31:59,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 00:31:59,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:31:59,181 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 00:31:59,209 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:31:59,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:59,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:31:59,255 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:59,270 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:31:59,285 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:31:59,308 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 00:31:59,308 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:31:59,321 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:31:59,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:31:59,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:59,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:31:59,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:59,395 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:59,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:59,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:59,441 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:59,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:59,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:59,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:31:59,506 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 00:31:59,507 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 00:31:59,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:31:59,821 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 00:31:59,878 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:31:59,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 00:31:59,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 00:31:59,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 00:31:59,915 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 00:31:59,931 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 00:31:59,946 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 00:31:59,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 00:31:59,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:31:59,984 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 00:32:00,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:32:00,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:32:00,051 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:32:00,063 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:32:00,081 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:32:00,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:32:00,105 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 00:32:00,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:32:00,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:32:00,132 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:32:00,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:00,183 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:32:00,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:00,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:00,242 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:00,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:00,270 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:00,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:00,303 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:00,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:00,336 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 00:32:00,339 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:32:00,414 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:32:00,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:32:00,677 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:32:00,678 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:32:00,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:32:00,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:32:00,714 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:32:00,732 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:32:00,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:32:00,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:32:00,770 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:32:00,789 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 00:32:00,816 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:32:00,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:32:01,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:32:01,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:32:01,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:32:01,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:32:01,441 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 00:32:01,443 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:32:01,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:32:01,474 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:32:01,493 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 00:32:01,493 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:32:01,506 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:32:01,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:32:01,811 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:32:01,872 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:32:01,872 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:32:01,885 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:32:01,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:32:01,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:32:01,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:32:01,943 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:32:01,956 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:32:01,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:32:01,983 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 00:32:02,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:32:02,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:32:02,193 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:32:02,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:32:02,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:32:02,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:32:02,562 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 00:32:02,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:32:02,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:32:02,589 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:32:02,602 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 00:32:02,604 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:32:02,607 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:32:02,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:32:02,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:32:02,957 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:32:02,959 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:32:02,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:32:02,984 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:32:02,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:32:03,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:32:03,028 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:32:03,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:32:03,053 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:32:03,068 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 00:32:03,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:32:03,109 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:32:03,176 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:32:03,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:32:03,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:32:03,354 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:32:03,368 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 00:32:03,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:32:03,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:32:03,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:32:03,408 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 00:32:03,408 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:32:03,411 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:32:03,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:32:03,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:32:03,773 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:32:03,775 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:32:03,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:32:03,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:32:03,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:32:03,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:32:03,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:32:03,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:32:03,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:32:03,907 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 00:32:03,938 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:32:03,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:32:04,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:32:04,204 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:32:04,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:32:04,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:32:04,382 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 00:32:04,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:32:04,397 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:32:04,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:32:04,423 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 00:32:04,423 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:32:04,426 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 00:32:04,525 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:32:04,755 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 00:32:04,812 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:32:04,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 00:32:04,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 00:32:04,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 00:32:04,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 00:32:04,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 00:32:04,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 00:32:04,892 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 00:32:04,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:32:04,924 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 00:32:04,951 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:32:04,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:32:04,982 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:32:04,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:32:05,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:32:05,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:32:05,036 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 00:32:05,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:32:05,053 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:32:05,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:32:05,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:05,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:32:05,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:05,137 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:05,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:05,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:05,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:05,206 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:05,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:05,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:32:05,256 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 00:32:05,256 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:32:05,259 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:32:05,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:32:05,576 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:32:05,634 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:32:05,635 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:32:05,650 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:32:05,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:32:05,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:32:05,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:32:05,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:32:05,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:32:05,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:05,875 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:32:05,881 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:05,881 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:07,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:07,996 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:32:08,003 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:08,003 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:10,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:10,142 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:10,142 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:12,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:12,293 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:12,293 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:14,295 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:14,440 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:14,441 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:16,443 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:16,588 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:16,588 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:18,588 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:18,723 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:18,724 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:20,724 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:20,872 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:20,872 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:22,872 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:23,026 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:23,026 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:25,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:25,176 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:25,176 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:27,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:27,323 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:27,323 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:29,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:29,480 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:29,482 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:31,482 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:31,637 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:31,638 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:33,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:33,793 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:33,795 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:35,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:35,951 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:35,953 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:37,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:38,099 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:38,101 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:40,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:40,388 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:40,398 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:42,400 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:43,081 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:43,082 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:45,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:45,533 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:45,533 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:47,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:48,012 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:48,012 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 977 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:32:50,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:50,270 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:50,270 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:52,272 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:52,703 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:52,704 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:54,704 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:55,520 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:55,520 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:57,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:57,776 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:57,776 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:32:59,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:32:59,971 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:32:59,973 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:33:01,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:02,243 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:33:02,243 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:33:04,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:04,568 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:33:04,568 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:33:06,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:06,732 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:33:06,733 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:33:08,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:08,895 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:33:08,897 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:33:10,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:11,098 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:33:11,098 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:33:13,099 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:13,318 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:33:13,319 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:33:15,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:15,497 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:33:15,499 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:33:17,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:17,691 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:33:17,691 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:33:19,693 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:19,878 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:33:19,878 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:33:21,880 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:22,040 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:33:22,042 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:33:24,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:24,207 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:33:24,207 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:33:26,209 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:26,362 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:33:26,362 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 81.1059999466. 2022-02-08 00:33:26,364 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:33:26,365 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:33:26,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:33:27,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:33:27,233 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:33:27,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:33:27,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:33:27,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:33:27,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:33:27,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:33:27,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:33:27,339 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:33:27,352 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:27,601 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:33:27,601 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.23900008202. 2022-02-08 00:33:27,601 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:33:27,604 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:33:28,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:33:28,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:33:28,720 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:33:28,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:33:28,737 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:33:28,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:33:28,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:33:28,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:33:28,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:33:28,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:33:28,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:29,204 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:33:29,206 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.60400009155. 2022-02-08 00:33:29,206 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:33:29,207 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:33:29,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:33:30,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:33:30,198 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:33:30,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:33:30,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:33:30,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:33:30,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:33:30,283 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:33:30,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:33:30,415 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:33:30,446 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:30,711 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:33:30,711 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.50599980354. 2022-02-08 00:33:30,713 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:33:30,714 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:33:31,602 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:33:31,999 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:33:32,246 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:33:32,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:33:32,263 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:33:32,321 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:33:32,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:33:32,413 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:33:32,599 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:33:32,691 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:33:32,785 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:33,334 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:33:33,335 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.62199997902. 2022-02-08 00:33:33,335 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:33:33,336 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:33:34,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:33:34,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:33:34,430 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:33:34,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:33:34,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:33:34,474 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:33:34,489 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:33:34,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:33:34,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:33:34,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:33:34,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:34,907 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:33:34,907 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 00:33:36,910 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:37,072 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:33:37,072 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 00:33:39,073 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:39,463 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:33:39,463 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.12800002098. 2022-02-08 00:33:39,463 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:33:39,464 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:33:40,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:33:40,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:33:40,622 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:33:40,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:33:40,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:33:40,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:33:40,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:33:40,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:33:40,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:33:40,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:33:40,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:41,359 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:33:41,361 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.89800000191. 2022-02-08 00:33:41,361 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:33:41,362 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:33:42,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:33:42,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:33:43,492 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:33:43,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:33:43,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:33:43,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:33:43,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:33:43,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:33:44,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:33:44,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:33:44,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:44,940 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:33:44,940 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.57999992371. 2022-02-08 00:33:44,940 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:33:44,943 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:33:45,216 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:33:45,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:33:45,543 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:33:45,543 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:33:45,556 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:33:45,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:33:45,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:33:45,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:33:45,612 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:33:45,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:33:45,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:45,773 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:33:45,773 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.832000017166. 2022-02-08 00:33:45,773 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:33:45,775 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:33:46,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:33:46,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:33:46,769 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:33:46,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:33:46,785 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:33:46,803 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:33:46,818 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:33:46,839 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:33:46,858 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:33:46,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:33:46,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:33:47,308 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:33:47,309 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.53600001335. ############################################################ # # RAM FootPrint: 3890.94140625 # ############################################################ 2022-02-08 00:33:47,351 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 00:33:47,351 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 70 seconds. Reason: Iteration:1 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 987 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 1030 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:34:57,358 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 00:34:57,358 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 00:35:02,365 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:35:02,903 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:03,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:35:03,266 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:03,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:35:03,280 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:35:03,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:35:03,311 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:35:03,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:35:03,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:35:03,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:35:03,381 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:35:03,382 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 00:35:03,444 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['128401 memcached\n'], []) 2022-02-08 00:35:03,446 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 00:35:03,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 00:35:03,490 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['128401 memcached\n'], []) 2022-02-08 00:35:03,492 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:35:03,493 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:35:03,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:03,780 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:35:03,845 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:03,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:35:03,858 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:35:03,869 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:35:03,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:35:03,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:35:03,910 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:35:03,923 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:35:03,936 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:35:03,936 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 00:35:03,996 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['79739 memcached\n'], []) 2022-02-08 00:35:03,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 00:35:04,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 00:35:04,048 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['79739 memcached\n'], []) 2022-02-08 00:35:04,049 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:35:04,051 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:35:04,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:04,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:35:04,427 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:04,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:35:04,441 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:35:04,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:35:04,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:35:04,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:35:04,510 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:35:04,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:35:04,539 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:35:04,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 00:35:04,640 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['122082 memcached\n'], []) 2022-02-08 00:35:04,640 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 00:35:04,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 00:35:04,690 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['122082 memcached\n'], []) 2022-02-08 00:35:04,690 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:35:04,693 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:35:04,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:05,206 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:35:05,319 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:05,321 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:35:05,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:35:05,357 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:35:05,479 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:35:05,545 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:35:05,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:35:05,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:35:05,595 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:35:05,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 00:35:05,640 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['109169 memcached\n'], []) 2022-02-08 00:35:05,640 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 00:35:05,665 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 00:35:05,687 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['109169 memcached\n'], []) 2022-02-08 00:35:05,687 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:35:05,690 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:35:05,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:05,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:35:06,051 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:06,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:35:06,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:35:06,076 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:35:06,091 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:35:06,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:35:06,122 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:35:06,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:35:06,148 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:35:06,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 00:35:06,196 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['18273 memcached\n'], []) 2022-02-08 00:35:06,196 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 00:35:06,221 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 00:35:06,244 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['18273 memcached\n'], []) 2022-02-08 00:35:06,246 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:35:06,247 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:35:06,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:06,545 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:35:06,637 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:06,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:35:06,653 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:35:06,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:35:06,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:35:06,701 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:35:06,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:35:06,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:35:06,753 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:35:06,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 00:35:06,796 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['12037 memcached\n'], []) 2022-02-08 00:35:06,798 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 00:35:06,821 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 00:35:06,842 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['12037 memcached\n'], []) 2022-02-08 00:35:06,844 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:35:06,845 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:35:06,931 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:07,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:35:07,191 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:07,191 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:35:07,204 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:35:07,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:35:07,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:35:07,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:35:07,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:35:07,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:35:07,293 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:35:07,295 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 00:35:07,342 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['53085 memcached\n'], []) 2022-02-08 00:35:07,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 00:35:07,377 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 00:35:07,401 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['53085 memcached\n'], []) 2022-02-08 00:35:07,401 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:35:07,404 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:35:07,489 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:07,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:35:07,759 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:07,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:35:07,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:35:07,785 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:35:07,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:35:07,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:35:07,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:35:07,839 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:35:07,854 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:35:07,854 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 00:35:07,911 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['47018 memcached\n'], []) 2022-02-08 00:35:07,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 00:35:07,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 00:35:07,960 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['47018 memcached\n'], []) 2022-02-08 00:35:07,960 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:35:07,963 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:35:08,038 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:08,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:35:08,315 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:08,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:35:08,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:35:08,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:35:08,359 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:35:08,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:35:08,405 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:35:08,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:35:08,430 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:35:08,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 00:35:08,482 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['111095 memcached\n'], []) 2022-02-08 00:35:08,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 00:35:08,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 00:35:08,533 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['111095 memcached\n'], []) 2022-02-08 00:35:08,535 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:35:08,536 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:35:08,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:08,816 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:35:08,875 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:08,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:35:08,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:35:08,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:35:08,914 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:35:08,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:35:08,941 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:35:08,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:35:08,967 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:35:08,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:35:09,039 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['59147 memcached\n'], []) 2022-02-08 00:35:09,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 00:35:09,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:35:09,082 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['59147 memcached\n'], []) 2022-02-08 00:35:09,082 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:35:09,085 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:35:09,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:09,410 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:35:09,479 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:09,479 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:35:09,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:35:09,507 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:35:09,520 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:35:09,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:35:09,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:35:09,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:35:09,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:35:09,601 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 00:35:09,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:35:09,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:09,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:35:09,905 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:09,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:10,073 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:35:10,088 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 00:35:10,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:35:10,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:10,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:35:10,130 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 00:35:10,130 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:35:10,131 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:35:10,249 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:10,456 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:35:10,515 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:10,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:35:10,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:35:10,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:35:10,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:35:10,571 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:35:10,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:35:10,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:35:10,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:35:10,622 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 00:35:10,650 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:35:10,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:10,841 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:35:10,994 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:11,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:11,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:35:11,200 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 00:35:11,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:35:11,214 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:11,229 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:35:11,242 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 00:35:11,242 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:35:11,244 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:35:11,339 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:11,538 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:35:11,607 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:11,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:35:11,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:35:11,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:35:11,647 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:35:11,663 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:35:11,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:35:11,688 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:35:11,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:35:11,720 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 00:35:11,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:35:11,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:11,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:35:12,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:12,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:12,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:35:12,204 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 00:35:12,206 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:35:12,221 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:12,237 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:35:12,252 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 00:35:12,255 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:35:12,256 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 00:35:12,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:12,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 00:35:12,609 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:12,609 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 00:35:12,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 00:35:12,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 00:35:12,650 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 00:35:12,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 00:35:12,678 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 00:35:12,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 00:35:12,704 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:35:12,720 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 00:35:12,749 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:35:12,762 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:12,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:35:12,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:12,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:12,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:35:12,828 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 00:35:12,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:35:12,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:12,857 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:35:12,869 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:12,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:35:12,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:12,920 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:12,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:12,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:12,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:12,984 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:13,002 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:13,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:13,032 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 00:35:13,035 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:35:13,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:13,321 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:35:13,377 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:13,377 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:35:13,390 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:35:13,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:35:13,413 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:35:13,428 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:35:13,443 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:35:13,454 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:35:13,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:35:13,483 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 00:35:13,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:35:13,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:13,563 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:35:13,586 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:13,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:13,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:35:13,634 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 00:35:13,635 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:35:13,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:13,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:35:13,671 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 00:35:13,673 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:35:13,674 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 00:35:13,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:13,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 00:35:14,042 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:14,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 00:35:14,056 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 00:35:14,073 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 00:35:14,086 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 00:35:14,104 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 00:35:14,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 00:35:14,132 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 00:35:14,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:35:14,163 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 00:35:14,188 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:35:14,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:14,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:35:14,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:14,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:14,257 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:35:14,269 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 00:35:14,270 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:35:14,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:14,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:35:14,308 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:14,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:35:14,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:14,357 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:14,372 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:14,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:14,403 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:14,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:14,434 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:14,451 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:14,467 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 00:35:14,469 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:35:14,556 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:14,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:35:14,834 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:14,834 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:35:14,852 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:35:14,867 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:35:14,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:35:14,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:35:14,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:35:14,931 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:35:14,946 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:35:14,961 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 00:35:14,987 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:35:15,005 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:15,168 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:35:15,316 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:15,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:15,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:35:15,536 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 00:35:15,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:35:15,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:15,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:35:15,578 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 00:35:15,578 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:35:15,579 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:35:15,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:15,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:35:15,934 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:15,936 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:35:15,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:35:15,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:35:15,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:35:15,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:35:16,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:35:16,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:35:16,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:35:16,071 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 00:35:16,096 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:35:16,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:16,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:35:16,706 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:16,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:16,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:35:16,911 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 00:35:16,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:35:16,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:16,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:35:16,960 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 00:35:16,960 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:35:16,963 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 00:35:17,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:17,252 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 00:35:17,323 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:17,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 00:35:17,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 00:35:17,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 00:35:17,364 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 00:35:17,382 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 00:35:17,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 00:35:17,408 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 00:35:17,421 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:35:17,436 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 00:35:17,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:35:17,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:17,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:35:17,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:17,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:17,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:35:17,563 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 00:35:17,563 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:35:17,576 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:17,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:35:17,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:17,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:35:17,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:17,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:17,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:17,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:17,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:17,724 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:17,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:17,760 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:17,779 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 00:35:17,780 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 00:35:17,867 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:18,084 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 00:35:18,151 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:18,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 00:35:18,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 00:35:18,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 00:35:18,188 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 00:35:18,204 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 00:35:18,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 00:35:18,229 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 00:35:18,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:35:18,265 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 00:35:18,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:35:18,311 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:18,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:35:18,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:18,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:18,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:35:18,375 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 00:35:18,377 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:35:18,390 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:18,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:35:18,415 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:18,433 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:35:18,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:18,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:18,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:18,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:18,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:18,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:18,556 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:18,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:18,591 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 00:35:18,592 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:35:18,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:18,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:35:18,990 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:18,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:35:19,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:35:19,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:35:19,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:35:19,068 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:35:19,089 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:35:19,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:35:19,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:35:19,144 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 00:35:19,173 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:35:19,198 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:19,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:35:19,586 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:19,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:19,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:35:19,855 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 00:35:19,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:35:19,869 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:19,885 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:35:19,900 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 00:35:19,901 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:35:19,907 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:35:19,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:20,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:35:20,269 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:20,270 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:35:20,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:35:20,298 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:35:20,311 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:35:20,326 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:35:20,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:35:20,354 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:35:20,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:35:20,387 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 00:35:20,413 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:35:20,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:20,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:35:20,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:20,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:20,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:35:20,990 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 00:35:20,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:35:21,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:21,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:35:21,032 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 00:35:21,032 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:35:21,033 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:35:21,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:21,349 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:35:21,423 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:21,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:35:21,441 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:35:21,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:35:21,471 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:35:21,487 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:35:21,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:35:21,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:35:21,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:35:21,542 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 00:35:21,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:35:21,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:21,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:35:21,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:21,859 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:21,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:35:21,979 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 00:35:21,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:35:22,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:22,025 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:35:22,039 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 00:35:22,039 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:35:22,042 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:35:22,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:22,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:35:22,397 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:22,397 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:35:22,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:35:22,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:35:22,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:35:22,464 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:35:22,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:35:22,489 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:35:22,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:35:22,530 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 00:35:22,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:35:22,599 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:22,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:35:22,844 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:22,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:23,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:35:23,026 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 00:35:23,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:35:23,040 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:23,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:35:23,069 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 00:35:23,069 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:35:23,072 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 00:35:23,158 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:23,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 00:35:23,459 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:23,459 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 00:35:23,474 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 00:35:23,489 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 00:35:23,502 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 00:35:23,520 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 00:35:23,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 00:35:23,553 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 00:35:23,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:35:23,585 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 00:35:23,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:35:23,627 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:23,642 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:35:23,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:23,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:35:23,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:35:23,698 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 00:35:23,698 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:35:23,711 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:35:23,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:35:23,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:23,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:35:23,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:23,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:23,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:23,831 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:23,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:23,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:23,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:23,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:35:23,936 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 00:35:23,937 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:35:23,938 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:35:24,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:35:24,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:35:24,302 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:35:24,303 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:35:24,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:35:24,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:35:24,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:35:24,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:35:24,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:35:24,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:35:24,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:35:24,573 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:35:24,573 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:35:26,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:35:26,721 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:35:26,723 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:35:28,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:35:28,874 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:35:28,875 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:35:30,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:35:31,020 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:35:31,022 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:35:33,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:35:33,168 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:35:33,168 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:35:35,168 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:35:35,322 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:35:35,322 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:35:37,323 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:35:37,471 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:35:37,471 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:35:39,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:35:39,631 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:35:39,631 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:35:41,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:35:41,790 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:35:41,790 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:35:43,790 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:35:43,940 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:35:43,940 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:35:45,941 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:35:46,085 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:35:46,085 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:35:48,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:35:48,230 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:35:48,232 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 1045 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:35:50,232 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:35:50,378 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:35:50,380 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:35:52,380 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:35:52,536 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:35:52,536 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:35:54,538 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:35:54,694 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:35:54,694 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:35:56,696 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:35:56,859 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:35:56,861 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:35:58,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:35:59,012 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:35:59,013 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:36:01,015 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:01,161 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:36:01,161 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:36:03,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:03,321 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:36:03,321 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:36:05,321 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:05,470 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:36:05,470 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:36:07,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:07,627 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:36:07,628 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:36:09,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:09,770 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:36:09,770 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:36:11,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:11,914 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:36:11,914 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:36:13,915 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:14,062 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:36:14,062 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:36:16,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:16,213 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:36:16,213 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:36:18,214 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:18,365 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:36:18,369 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 54.4339997768. 2022-02-08 00:36:18,369 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:36:18,372 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:36:18,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:36:19,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:36:19,259 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:36:19,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:36:19,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:36:19,288 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:36:19,313 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:36:19,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:36:19,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:36:19,377 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:36:19,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:19,644 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:36:19,644 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.27400016785. 2022-02-08 00:36:19,644 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:36:19,647 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:36:20,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:36:20,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:36:20,624 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:36:20,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:36:20,640 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:36:20,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:36:20,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:36:20,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:36:20,709 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:36:20,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:36:20,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:21,236 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:36:21,236 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 00:36:23,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:23,388 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:36:23,388 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.74300003052. 2022-02-08 00:36:23,388 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:36:23,390 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:36:23,989 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:36:24,278 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:36:24,394 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:36:24,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:36:24,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:36:24,436 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:36:24,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:36:24,479 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:36:24,497 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:36:24,528 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:36:24,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:24,904 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:36:24,904 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 00:36:26,905 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:27,039 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:36:27,040 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.65299987793. 2022-02-08 00:36:27,040 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:36:27,042 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:36:27,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:36:28,020 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:36:28,121 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:36:28,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:36:28,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:36:28,155 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:36:28,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:36:28,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:36:28,214 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:36:28,242 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:36:28,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:28,730 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:36:28,730 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.6890001297. 2022-02-08 00:36:28,730 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:36:28,732 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:36:29,380 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:36:29,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:36:29,709 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:36:29,709 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:36:29,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:36:29,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:36:29,755 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:36:29,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:36:29,795 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:36:29,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:36:29,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:30,105 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:36:30,107 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.37700009346. 2022-02-08 00:36:30,107 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:36:30,108 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:36:30,971 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:36:31,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:36:31,641 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:36:31,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:36:31,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:36:31,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:36:31,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:36:31,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:36:31,716 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:36:31,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:36:31,760 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:32,125 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:36:32,125 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.01799988747. 2022-02-08 00:36:32,125 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:36:32,128 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:36:32,655 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:36:32,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:36:32,994 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:36:32,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:36:33,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:36:33,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:36:33,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:36:33,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:36:33,091 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:36:33,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:36:33,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:33,388 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:36:33,388 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.26300001144. 2022-02-08 00:36:33,388 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:36:33,391 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:36:33,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:36:34,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:36:34,187 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:36:34,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:36:34,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:36:34,213 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:36:34,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:36:34,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:36:34,257 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:36:34,270 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:36:34,283 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:34,415 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:36:34,417 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.02800011635. 2022-02-08 00:36:34,417 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:36:34,418 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:36:34,515 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:36:34,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:36:34,836 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:36:34,836 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:36:34,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:36:34,872 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:36:34,892 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:36:34,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:36:34,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:36:34,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:36:34,989 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:35,124 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:36:35,124 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 00:36:37,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:37,279 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:36:37,279 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 00:36:39,280 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:39,766 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:36:39,766 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 00:36:41,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:41,908 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:36:41,908 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 00:36:43,910 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:36:44,045 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:36:44,045 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 9.62800002098. ############################################################ # # RAM FootPrint: 3891.18359375 # ############################################################ 2022-02-08 00:36:44,114 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 00:36:44,114 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 99 seconds. Reason: Iteration:2 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 1058 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 1089 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:38:23,119 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 00:38:23,121 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 00:38:28,122 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:38:28,668 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:29,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:38:29,361 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:29,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:38:29,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:38:29,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:38:29,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:38:29,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:38:29,507 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:38:29,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:38:29,575 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:38:29,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 00:38:29,650 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['129520 memcached\n'], []) 2022-02-08 00:38:29,650 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 00:38:29,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 00:38:29,700 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['129520 memcached\n'], []) 2022-02-08 00:38:29,700 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:38:29,703 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:38:29,785 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:30,002 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:38:30,091 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:30,091 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:38:30,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:38:30,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:38:30,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:38:30,150 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:38:30,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:38:30,176 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:38:30,194 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:38:30,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 00:38:30,266 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['80396 memcached\n'], []) 2022-02-08 00:38:30,267 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 00:38:30,295 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 00:38:30,316 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['80396 memcached\n'], []) 2022-02-08 00:38:30,316 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:38:30,322 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:38:30,415 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:30,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:38:30,710 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:30,711 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:38:30,724 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:38:30,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:38:30,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:38:30,778 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:38:30,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:38:30,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:38:30,815 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:38:30,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 00:38:30,861 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['122776 memcached\n'], []) 2022-02-08 00:38:30,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 00:38:30,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 00:38:30,904 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['122776 memcached\n'], []) 2022-02-08 00:38:30,904 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:38:30,907 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:38:30,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:31,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:38:31,282 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:31,283 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:38:31,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:38:31,313 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:38:31,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:38:31,346 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:38:31,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:38:31,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:38:31,400 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:38:31,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 00:38:31,446 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['109921 memcached\n'], []) 2022-02-08 00:38:31,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 00:38:31,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 00:38:31,506 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['109921 memcached\n'], []) 2022-02-08 00:38:31,506 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:38:31,509 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:38:31,589 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:31,808 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:38:31,875 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:31,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:38:31,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:38:31,903 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:38:31,915 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:38:31,931 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:38:31,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:38:31,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:38:31,970 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:38:31,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 00:38:32,013 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['18917 memcached\n'], []) 2022-02-08 00:38:32,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 00:38:32,040 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 00:38:32,062 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['18917 memcached\n'], []) 2022-02-08 00:38:32,062 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:38:32,065 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:38:32,145 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:32,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:38:32,420 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:32,420 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:38:32,434 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:38:32,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:38:32,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:38:32,479 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:38:32,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:38:32,505 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:38:32,519 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:38:32,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 00:38:32,573 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['12718 memcached\n'], []) 2022-02-08 00:38:32,573 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 00:38:32,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 00:38:32,628 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['12718 memcached\n'], []) 2022-02-08 00:38:32,628 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:38:32,631 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:38:32,709 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:32,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:38:33,003 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:33,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:38:33,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:38:33,028 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:38:33,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:38:33,053 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:38:33,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:38:33,079 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:38:33,095 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:38:33,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 00:38:33,144 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['53725 memcached\n'], []) 2022-02-08 00:38:33,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 00:38:33,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 00:38:33,196 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['53725 memcached\n'], []) 2022-02-08 00:38:33,196 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:38:33,200 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:38:33,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:33,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:38:33,630 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:33,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:38:33,647 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:38:33,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:38:33,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:38:33,696 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:38:33,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:38:33,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:38:33,739 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:38:33,739 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 00:38:33,788 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['47675 memcached\n'], []) 2022-02-08 00:38:33,788 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 00:38:33,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 00:38:33,835 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['47675 memcached\n'], []) 2022-02-08 00:38:33,835 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:38:33,838 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:38:33,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:34,135 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:38:34,194 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:34,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:38:34,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:38:34,221 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:38:34,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:38:34,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:38:34,262 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:38:34,275 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:38:34,286 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:38:34,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 00:38:34,332 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['111752 memcached\n'], []) 2022-02-08 00:38:34,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 00:38:34,354 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 00:38:34,375 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['111752 memcached\n'], []) 2022-02-08 00:38:34,375 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:38:34,378 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:38:34,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:35,227 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:38:35,319 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:35,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:38:35,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:38:35,349 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:38:35,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:38:35,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:38:35,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:38:35,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:38:35,437 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:38:35,438 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:38:35,493 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['59783 memcached\n'], []) 2022-02-08 00:38:35,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 00:38:35,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:38:35,539 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['59783 memcached\n'], []) 2022-02-08 00:38:35,539 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:38:35,542 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:38:35,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:35,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:38:35,907 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:35,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:38:35,923 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:38:35,936 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:38:35,948 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:38:35,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:38:35,977 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:38:35,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:38:36,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:38:36,020 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 00:38:36,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:38:36,071 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:36,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:38:36,369 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:36,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:36,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:38:36,556 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 00:38:36,558 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:38:36,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:36,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:38:36,611 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 00:38:36,612 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:38:36,615 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:38:36,696 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:36,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:38:36,993 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:36,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:38:37,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:38:37,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:38:37,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:38:37,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:38:37,066 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:38:37,079 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:38:37,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:38:37,107 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 00:38:37,135 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:38:37,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:37,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:38:37,532 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:37,635 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:37,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:38:37,750 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 00:38:37,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:38:37,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:37,798 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:38:37,815 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 00:38:37,815 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:38:37,818 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:38:37,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:38,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:38:38,180 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:38,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:38:38,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:38:38,206 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:38:38,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:38:38,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:38:38,247 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:38:38,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:38:38,276 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:38:38,289 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 00:38:38,318 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:38:38,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:39,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:38:39,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:39,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:40,015 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:38:40,028 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 00:38:40,028 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:38:40,040 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:40,091 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:38:40,102 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 00:38:40,102 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:38:40,105 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 00:38:40,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:40,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 00:38:40,480 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:40,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 00:38:40,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 00:38:40,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 00:38:40,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 00:38:40,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 00:38:40,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 00:38:40,561 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 00:38:40,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:38:40,589 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 00:38:40,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:38:40,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:40,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:38:40,668 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:40,686 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:40,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:38:40,710 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 00:38:40,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:38:40,724 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:40,739 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:38:40,752 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:40,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:38:40,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:40,816 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:40,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:40,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:40,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:40,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:40,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:40,941 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:40,959 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 00:38:40,960 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:38:41,053 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:41,280 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:38:41,336 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:41,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:38:41,349 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:38:41,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:38:41,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:38:41,397 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:38:41,420 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:38:41,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:38:41,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:38:41,471 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 00:38:41,497 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:38:41,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:41,553 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:38:41,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:41,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:41,619 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:38:41,632 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 00:38:41,632 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:38:41,647 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:41,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:38:41,674 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 00:38:41,674 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:38:41,677 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 00:38:41,755 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:41,971 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 00:38:42,026 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:42,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 00:38:42,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 00:38:42,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 00:38:42,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 00:38:42,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 00:38:42,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 00:38:42,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 00:38:42,119 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:38:42,132 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 00:38:42,158 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:38:42,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:42,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:38:42,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:42,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:42,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:38:42,259 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 00:38:42,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:38:42,272 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:42,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:38:42,298 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:42,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:38:42,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:42,349 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:42,367 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:42,382 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:42,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:42,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:42,436 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:42,456 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:42,473 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 00:38:42,474 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:38:42,563 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:42,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:38:42,835 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:42,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:38:42,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:38:42,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:38:42,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:38:42,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:38:42,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:38:42,920 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:38:42,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:38:42,950 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 00:38:42,979 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:38:42,992 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:44,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:38:44,405 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:44,512 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:44,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:38:44,632 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 00:38:44,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:38:44,645 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:44,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:38:44,686 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 00:38:44,687 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:38:44,688 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:38:44,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:45,002 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:38:45,071 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:45,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:38:45,089 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:38:45,107 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:38:45,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:38:45,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:38:45,160 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:38:45,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:38:45,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:38:45,209 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 00:38:45,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:38:45,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:45,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:38:45,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:45,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:45,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:38:45,727 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 00:38:45,727 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:38:45,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:45,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:38:45,789 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 00:38:45,790 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:38:45,792 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 00:38:45,869 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:46,086 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 00:38:46,147 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:46,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 00:38:46,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 00:38:46,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 00:38:46,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 00:38:46,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 00:38:46,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 00:38:46,229 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 00:38:46,242 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:38:46,256 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 00:38:46,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:38:46,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:46,313 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:38:46,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:46,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:46,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:38:46,368 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 00:38:46,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:38:46,382 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:46,397 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:38:46,410 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:46,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:38:46,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:46,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:46,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:46,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:46,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:46,551 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:46,566 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:46,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:46,599 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 00:38:46,601 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 00:38:46,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:46,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 00:38:46,983 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:46,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 00:38:46,999 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 00:38:47,012 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 00:38:47,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 00:38:47,043 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 00:38:47,058 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 00:38:47,073 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 00:38:47,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:38:47,104 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 00:38:47,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:38:47,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:47,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:38:47,181 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:47,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:47,209 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:38:47,223 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 00:38:47,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:38:47,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:47,252 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:38:47,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:47,288 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:38:47,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:47,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:47,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:47,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:47,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:47,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:47,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:47,438 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:47,463 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 00:38:47,467 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:38:47,551 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:47,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:38:47,861 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:47,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:38:47,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:38:47,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:38:47,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:38:47,918 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:38:47,931 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:38:47,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:38:47,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:38:47,976 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 00:38:48,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:38:48,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 1135 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:38:49,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:38:49,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:50,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:50,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:38:50,259 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 00:38:50,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:38:50,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:50,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:38:50,309 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 00:38:50,309 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:38:50,312 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:38:50,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:50,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:38:50,676 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:50,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:38:50,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:38:50,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:38:50,716 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:38:50,732 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:38:50,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:38:50,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:38:50,767 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:38:50,782 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 00:38:50,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:38:50,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:51,079 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:38:51,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:51,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:51,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:38:51,456 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 00:38:51,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:38:51,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:51,492 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:38:51,506 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 00:38:51,506 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:38:51,507 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:38:51,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:51,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:38:52,183 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:52,183 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:38:52,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:38:52,265 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:38:52,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:38:52,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:38:52,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:38:52,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:38:52,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:38:52,473 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 00:38:52,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:38:52,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:52,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:38:52,821 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:52,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:52,948 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:38:52,963 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 00:38:52,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:38:52,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:53,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:38:53,035 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 00:38:53,035 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:38:53,038 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:38:53,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:53,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:38:53,375 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:53,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:38:53,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:38:53,400 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:38:53,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:38:53,426 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:38:53,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:38:53,448 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:38:53,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:38:53,480 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 00:38:53,507 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:38:53,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:54,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:38:54,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:54,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:54,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:38:54,999 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 00:38:54,999 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:38:55,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:55,040 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:38:55,055 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 00:38:55,055 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:38:55,059 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 00:38:55,140 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:55,374 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 00:38:55,437 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:55,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 00:38:55,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 00:38:55,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 00:38:55,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 00:38:55,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 00:38:55,525 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 00:38:55,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 00:38:55,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:38:55,576 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 00:38:55,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:38:55,624 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:55,640 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:38:55,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:55,668 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:38:55,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:38:55,700 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 00:38:55,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:38:55,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:38:55,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:38:55,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:55,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:38:55,783 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:55,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:55,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:55,839 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:55,858 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:55,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:55,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:55,913 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:38:55,930 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 00:38:55,930 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:38:55,933 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:38:56,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:38:56,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:38:56,286 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:38:56,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:38:56,301 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:38:56,313 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:38:56,326 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:38:56,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:38:56,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:38:56,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:38:56,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:38:56,546 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:38:56,551 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:38:56,551 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:38:58,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:38:58,668 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:38:58,674 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:38:58,674 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:00,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:00,802 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:39:00,808 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:00,808 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:02,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:02,923 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:39:02,928 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:02,928 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:04,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:05,042 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:39:05,048 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:05,048 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:07,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:07,164 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:39:07,170 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:07,170 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:09,170 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:09,282 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:39:09,289 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:09,289 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:11,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:11,413 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:39:11,418 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:11,418 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:13,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:13,529 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:39:13,536 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:13,536 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:15,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:15,657 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:39:15,661 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:15,661 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:17,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:17,793 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:39:17,799 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:17,799 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:19,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:19,924 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:39:19,930 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:19,930 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:21,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:22,096 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:22,096 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:24,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:24,249 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:24,249 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:26,249 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:26,418 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:26,418 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:28,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:28,585 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:28,585 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:30,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:30,744 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:30,746 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:32,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:32,894 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:32,894 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:34,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:35,052 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:35,052 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:37,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:37,239 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:37,239 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:39,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:39,413 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:39,413 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:41,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:41,579 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:41,581 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:43,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:43,757 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:43,757 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:45,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:45,947 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:45,947 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:47,948 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:48,545 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:48,545 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 1144 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:39:50,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:50,812 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:50,812 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:52,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:53,282 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:53,282 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:55,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:55,464 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:55,464 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:39:57,464 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:39:58,128 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:39:58,128 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:40:00,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:00,769 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:40:00,770 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:40:02,770 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:03,631 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:40:03,631 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:40:05,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:06,460 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:40:06,461 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:40:08,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:08,763 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:40:08,763 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:40:10,765 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:11,260 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:40:11,260 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:40:13,262 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:13,861 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:40:13,861 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:40:15,862 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:16,334 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:40:16,334 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:40:18,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:18,788 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:40:18,789 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:40:20,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:21,071 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:40:21,072 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 85.1410000324. 2022-02-08 00:40:21,072 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:40:21,073 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:40:21,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:40:22,071 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:40:22,161 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:40:22,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:40:22,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:40:22,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:40:22,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:40:22,219 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:40:22,232 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:40:22,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:40:22,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:22,727 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:40:22,729 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.65700006485. 2022-02-08 00:40:22,729 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:40:22,730 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:40:24,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:40:24,757 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:40:24,934 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:40:24,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:40:24,951 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:40:24,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:40:25,005 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:40:25,033 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:40:25,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:40:25,079 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:40:25,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:26,051 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:40:26,051 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.3220000267. 2022-02-08 00:40:26,051 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:40:26,052 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:40:26,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:40:26,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:40:26,782 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:40:26,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:40:26,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:40:26,831 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:40:26,858 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:40:26,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:40:26,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:40:26,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:40:26,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:27,443 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:40:27,444 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.39299988747. 2022-02-08 00:40:27,444 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:40:27,446 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:40:28,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:40:28,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:40:28,473 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:40:28,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:40:28,489 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:40:28,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:40:28,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:40:28,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:40:28,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:40:28,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:40:28,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:29,163 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:40:29,163 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.71900010109. 2022-02-08 00:40:29,164 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:40:29,165 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:40:29,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:40:30,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:40:30,196 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:40:30,196 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:40:30,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:40:30,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:40:30,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:40:30,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:40:30,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:40:30,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:40:30,344 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:30,599 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:40:30,601 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.4359998703. 2022-02-08 00:40:30,601 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:40:30,602 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:40:31,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:40:31,974 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:40:32,096 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:40:32,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:40:32,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:40:32,140 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:40:32,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:40:32,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:40:32,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:40:32,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:40:32,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:32,756 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:40:32,757 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.15599989891. 2022-02-08 00:40:32,757 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:40:32,759 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:40:33,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:40:33,576 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:40:33,661 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:40:33,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:40:33,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:40:33,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:40:33,706 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:40:33,721 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:40:33,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:40:33,798 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:40:33,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:34,033 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:40:34,033 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.27600002289. 2022-02-08 00:40:34,033 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:40:34,036 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:40:34,688 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:40:35,063 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:40:35,187 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:40:35,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:40:35,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:40:35,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:40:35,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:40:35,255 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:40:35,267 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:40:35,293 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:40:35,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:35,811 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:40:35,812 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.77800011635. 2022-02-08 00:40:35,812 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:40:35,813 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:40:37,183 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:40:38,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:40:38,367 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:40:38,367 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:40:38,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:40:38,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:40:38,428 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:40:38,454 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:40:38,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:40:38,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:40:38,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:40:40,874 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:40:40,875 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.06399989128. ############################################################ # # RAM FootPrint: 3892.4375 # ############################################################ 2022-02-08 00:40:40,911 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 00:40:40,911 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 103 seconds. Reason: Iteration:3 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 1163 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 1210 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:42:23,980 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 00:42:23,984 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 00:42:29,007 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:42:29,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:30,150 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:42:30,242 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:30,242 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:42:30,270 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:42:30,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:42:30,369 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:42:30,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:42:30,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:42:30,479 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:42:30,505 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:42:30,505 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 00:42:30,595 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['130489 memcached\n'], []) 2022-02-08 00:42:30,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 00:42:30,645 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 00:42:30,674 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['130489 memcached\n'], []) 2022-02-08 00:42:30,674 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:42:30,676 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:42:30,770 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:31,015 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:42:31,088 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:31,089 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:42:31,104 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:42:31,117 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:42:31,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:42:31,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:42:31,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:42:31,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:42:31,198 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:42:31,198 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 00:42:31,260 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['81096 memcached\n'], []) 2022-02-08 00:42:31,262 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 00:42:31,285 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 00:42:31,306 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['81096 memcached\n'], []) 2022-02-08 00:42:31,306 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:42:31,309 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:42:31,385 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:31,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:42:31,734 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:31,734 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:42:31,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:42:31,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:42:31,778 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:42:31,795 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:42:31,811 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:42:31,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:42:31,836 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:42:31,836 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 00:42:31,885 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['123453 memcached\n'], []) 2022-02-08 00:42:31,885 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 00:42:31,910 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 00:42:31,931 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['123453 memcached\n'], []) 2022-02-08 00:42:31,931 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:42:31,934 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:42:32,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:32,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:42:32,306 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:32,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:42:32,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:42:32,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:42:32,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:42:32,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:42:32,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:42:32,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:42:32,428 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:42:32,428 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 00:42:32,530 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['110728 memcached\n'], []) 2022-02-08 00:42:32,532 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 00:42:32,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 00:42:32,591 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['110728 memcached\n'], []) 2022-02-08 00:42:32,591 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:42:32,594 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:42:32,681 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:32,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:42:32,951 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:32,951 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:42:32,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:42:32,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:42:32,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:42:33,005 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:42:33,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:42:33,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:42:33,048 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:42:33,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 00:42:33,094 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['19582 memcached\n'], []) 2022-02-08 00:42:33,094 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 00:42:33,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 00:42:33,138 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['19582 memcached\n'], []) 2022-02-08 00:42:33,138 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:42:33,141 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:42:33,224 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:33,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:42:33,503 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:33,505 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:42:33,517 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:42:33,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:42:33,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:42:33,558 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:42:33,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:42:33,584 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:42:33,596 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:42:33,596 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 00:42:33,644 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['13416 memcached\n'], []) 2022-02-08 00:42:33,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 00:42:33,681 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 00:42:33,703 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['13416 memcached\n'], []) 2022-02-08 00:42:33,703 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:42:33,704 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:42:33,788 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:34,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:42:34,088 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:34,089 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:42:34,104 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:42:34,117 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:42:34,130 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:42:34,147 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:42:34,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:42:34,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:42:34,197 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:42:34,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 00:42:34,246 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['54393 memcached\n'], []) 2022-02-08 00:42:34,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 00:42:34,272 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 00:42:34,293 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['54393 memcached\n'], []) 2022-02-08 00:42:34,293 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:42:34,296 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:42:34,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:34,612 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:42:34,676 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:34,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:42:34,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:42:34,704 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:42:34,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:42:34,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:42:34,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:42:34,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:42:34,779 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:42:34,780 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 00:42:34,828 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['48375 memcached\n'], []) 2022-02-08 00:42:34,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 00:42:34,862 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 00:42:34,884 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['48375 memcached\n'], []) 2022-02-08 00:42:34,884 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:42:34,887 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:42:34,966 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:35,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:42:35,240 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:35,242 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:42:35,255 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:42:35,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:42:35,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:42:35,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:42:35,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:42:35,323 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:42:35,336 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:42:35,336 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 00:42:35,390 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['112450 memcached\n'], []) 2022-02-08 00:42:35,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 00:42:35,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 00:42:35,437 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['112450 memcached\n'], []) 2022-02-08 00:42:35,437 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:42:35,438 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:42:35,545 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:35,775 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:42:35,835 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:35,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:42:35,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:42:35,864 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:42:35,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:42:35,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:42:35,908 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:42:35,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:42:35,934 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:42:35,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:42:35,996 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['60538 memcached\n'], []) 2022-02-08 00:42:35,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 00:42:36,020 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:42:36,049 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['60538 memcached\n'], []) 2022-02-08 00:42:36,049 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:42:36,052 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:42:36,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:36,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:42:36,427 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:36,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:42:36,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:42:36,454 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:42:36,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:42:36,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:42:36,499 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:42:36,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:42:36,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:42:36,545 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 00:42:36,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:42:36,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:37,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:42:37,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:37,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:38,030 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:42:38,045 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 00:42:38,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:42:38,059 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:38,094 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:42:38,108 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 00:42:38,108 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:42:38,111 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:42:38,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:38,410 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:42:38,483 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:38,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:42:38,497 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:42:38,512 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:42:38,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:42:38,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:42:38,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:42:38,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:42:38,589 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:42:38,605 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 00:42:38,642 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:42:38,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:38,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:42:38,989 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:39,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:39,191 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:42:39,207 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 00:42:39,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:42:39,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:39,255 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:42:39,269 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 00:42:39,269 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:42:39,273 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:42:39,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:39,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:42:39,622 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:39,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:42:39,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:42:39,647 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:42:39,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:42:39,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:42:39,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:42:39,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:42:39,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:42:39,726 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 00:42:39,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:42:39,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:39,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:42:40,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:40,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:40,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:42:40,237 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 00:42:40,237 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:42:40,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:40,275 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:42:40,289 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 00:42:40,289 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:42:40,290 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 00:42:40,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:40,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 00:42:40,641 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:40,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 00:42:40,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 00:42:40,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 00:42:40,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 00:42:40,696 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 00:42:40,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 00:42:40,719 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 00:42:40,732 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:42:40,744 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 00:42:40,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:42:40,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:40,801 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:42:40,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:40,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:40,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:42:40,862 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 00:42:40,862 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:42:40,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:40,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:42:40,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:40,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:42:40,941 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:40,959 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:40,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:40,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:41,012 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:41,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:41,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:41,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:41,084 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 00:42:41,085 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:42:41,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:41,385 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:42:41,469 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:41,469 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:42:41,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:42:41,497 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:42:41,515 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:42:41,533 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:42:41,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:42:41,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:42:41,573 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:42:41,588 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 00:42:41,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:42:41,637 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:41,816 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:42:41,852 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:41,874 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:41,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:42:41,908 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 00:42:41,908 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:42:41,923 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:41,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:42:41,963 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 00:42:41,963 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:42:41,966 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 00:42:42,056 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:42,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 00:42:42,323 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:42,323 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 00:42:42,336 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 00:42:42,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 00:42:42,359 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 00:42:42,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 00:42:42,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 00:42:42,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 00:42:42,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:42:42,426 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 00:42:42,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:42:42,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:42,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:42:42,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:42,505 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:42,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:42:42,529 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 00:42:42,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:42:42,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:42,553 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:42:42,566 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:42,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:42:42,604 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:42,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:42,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:42,655 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:42,673 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:42,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:42,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:42,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:42,747 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 00:42:42,750 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:42:42,849 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:43,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:42:43,111 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:43,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:42:43,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:42:43,135 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:42:43,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:42:43,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:42:43,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:42:43,186 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:42:43,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:42:43,217 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 00:42:43,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:42:43,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:43,434 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:42:43,589 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:43,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:43,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:42:43,828 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 00:42:43,829 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:42:43,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:43,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:42:43,878 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 00:42:43,880 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:42:43,884 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:42:43,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:44,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:42:44,273 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:44,275 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:42:44,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:42:44,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:42:44,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:42:44,344 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:42:44,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:42:44,374 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:42:44,390 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:42:44,405 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 00:42:44,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:42:44,446 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:45,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:42:45,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:46,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:46,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:42:46,180 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 00:42:46,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:42:46,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:46,229 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:42:46,243 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 00:42:46,243 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:42:46,246 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 00:42:46,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:46,538 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 00:42:46,618 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:46,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 00:42:46,632 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 00:42:46,645 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 00:42:46,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 00:42:46,686 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 00:42:46,698 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 00:42:46,716 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 00:42:46,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:42:46,747 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 00:42:46,775 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:42:46,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:46,803 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:42:46,818 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:46,829 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:46,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:42:46,855 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 00:42:46,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:42:46,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:46,882 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:42:46,895 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:46,914 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:42:46,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:46,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:46,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:46,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:47,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:47,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:47,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:47,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:47,092 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 00:42:47,095 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 00:42:47,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:47,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 00:42:47,446 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:47,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 00:42:47,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 00:42:47,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 00:42:47,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 00:42:47,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 00:42:47,520 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 00:42:47,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 00:42:47,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:42:47,566 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 00:42:47,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:42:47,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:47,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:42:47,637 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:47,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:47,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:42:47,680 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 00:42:47,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:42:47,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:47,711 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:42:47,724 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:47,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:42:47,765 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:47,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:47,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:47,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:47,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:47,867 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:47,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:47,920 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:47,937 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 00:42:47,940 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:42:48,038 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:48,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:42:48,316 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:48,316 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:42:48,331 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:42:48,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:42:48,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:42:48,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:42:48,395 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:42:48,408 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:42:48,423 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:42:48,437 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 00:42:48,466 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:42:48,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:48,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:42:48,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:49,012 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 1258 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:42:49,130 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:42:49,151 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 00:42:49,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:42:49,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:49,191 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:42:49,206 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 00:42:49,206 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:42:49,210 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:42:49,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:49,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:42:49,565 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:49,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:42:49,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:42:49,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:42:49,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:42:49,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:42:49,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:42:49,647 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:42:49,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:42:49,673 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 00:42:49,698 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:42:49,711 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:49,992 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:42:50,150 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:50,257 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:50,357 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:42:50,371 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 00:42:50,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:42:50,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:50,405 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:42:50,421 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 00:42:50,421 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:42:50,423 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:42:50,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:50,780 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:42:50,907 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:50,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:42:50,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:42:50,979 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:42:50,992 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:42:51,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:42:51,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:42:51,056 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:42:51,079 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:42:51,117 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 00:42:51,150 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:42:51,168 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:51,808 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:42:51,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:52,066 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:52,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:42:52,302 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 00:42:52,303 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:42:52,346 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:52,471 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:42:52,486 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 00:42:52,486 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:42:52,489 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:42:52,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:52,795 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:42:52,887 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:52,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:42:52,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:42:52,914 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:42:52,926 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:42:52,943 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:42:52,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:42:52,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:42:52,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:42:52,994 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 00:42:53,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:42:53,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:53,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:42:53,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:53,413 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:53,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:42:53,522 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 00:42:53,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:42:53,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:53,566 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:42:53,579 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 00:42:53,581 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:42:53,582 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 00:42:53,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:53,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 00:42:53,990 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:53,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 00:42:54,005 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 00:42:54,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 00:42:54,035 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 00:42:54,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 00:42:54,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 00:42:54,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 00:42:54,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:42:54,130 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 00:42:54,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:42:54,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:54,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:42:54,211 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:54,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:42:54,242 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:42:54,256 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 00:42:54,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:42:54,272 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:42:54,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:42:54,301 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:54,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:42:54,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:54,364 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:54,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:54,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:54,423 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:54,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:54,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:54,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:42:54,526 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 00:42:54,528 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:42:54,529 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:42:54,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:42:54,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:42:54,904 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:42:54,905 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:42:54,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:42:54,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:42:54,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:42:54,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:42:54,984 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:42:54,999 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:42:55,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:42:55,127 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:42:55,132 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:42:55,132 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:42:57,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:42:57,244 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:42:57,250 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:42:57,250 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:42:59,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:42:59,398 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:42:59,398 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:01,400 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:01,545 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:01,545 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:03,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:03,687 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:03,687 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:05,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:05,839 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:05,839 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:07,841 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:07,986 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:07,986 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:09,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:10,131 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:10,131 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:12,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:12,282 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:12,282 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:14,283 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:14,443 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:14,444 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:16,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:16,594 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:16,594 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:18,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:18,953 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:18,953 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:20,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:21,096 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:21,096 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:23,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:23,292 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:23,292 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:25,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:25,895 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:25,895 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:27,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:28,371 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:28,371 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:30,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:30,681 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:30,681 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:32,683 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:32,878 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:32,878 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:34,880 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:35,216 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:35,216 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:37,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:37,450 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:37,450 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:39,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:39,700 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:39,700 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:41,709 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:42,046 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:42,048 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:44,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:44,273 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:44,273 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:46,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:46,630 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:46,631 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:48,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 1268 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:43:49,252 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:49,253 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:51,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:51,796 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:51,796 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:53,798 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:54,141 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:54,142 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:56,142 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:56,503 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:43:56,503 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:43:58,505 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:43:58,821 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:43:58,822 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 64.2939999104. 2022-02-08 00:43:58,822 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:43:58,823 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:43:59,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:43:59,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:43:59,694 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:43:59,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:43:59,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:43:59,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:43:59,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:43:59,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:43:59,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:43:59,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:43:59,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:44:00,198 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:44:00,200 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.37800002098. 2022-02-08 00:44:00,200 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:44:00,203 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:44:01,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:44:02,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:44:02,607 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:44:02,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:44:02,780 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:44:03,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:44:03,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:44:03,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:44:03,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:44:03,719 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:44:03,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:44:04,799 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:44:04,801 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 00:44:06,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:44:07,562 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:44:07,562 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 00:44:09,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:44:10,267 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:44:10,267 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 00:44:12,267 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:44:12,681 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:44:12,683 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 12.4830000401. 2022-02-08 00:44:12,683 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:44:12,686 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:44:12,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:44:13,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:44:14,861 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:44:14,862 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:44:15,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:44:15,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:44:15,693 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:44:15,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:44:15,933 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:44:16,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:44:16,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:44:17,834 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:44:17,834 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 00:44:19,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:44:20,428 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:44:20,428 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 00:44:22,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:44:22,953 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:44:22,953 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 00:44:24,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:44:25,635 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:44:25,635 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 00:44:27,635 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:44:28,181 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:44:28,183 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 15.5. 2022-02-08 00:44:28,183 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:44:28,184 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:44:29,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:44:29,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:44:29,858 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:44:29,859 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:44:29,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:44:29,895 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:44:29,918 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:44:29,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:44:29,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:44:30,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:44:30,045 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:44:30,655 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:44:30,657 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.47399997711. 2022-02-08 00:44:30,657 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:44:30,658 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:44:31,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:44:31,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:44:31,618 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:44:31,619 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:44:31,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:44:31,660 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:44:31,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:44:31,696 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:44:31,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:44:31,752 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:44:31,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:44:32,326 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:44:32,328 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.66999983788. 2022-02-08 00:44:32,328 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:44:32,329 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:44:32,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:44:33,096 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:44:33,203 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:44:33,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:44:33,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:44:33,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:44:33,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:44:33,278 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:44:33,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:44:33,318 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:44:33,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:44:33,723 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:44:33,729 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.40100002289. 2022-02-08 00:44:33,730 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:44:33,732 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:44:34,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:44:34,836 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:44:34,957 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:44:34,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:44:34,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:44:35,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:44:35,025 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:44:35,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:44:35,071 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:44:35,104 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:44:35,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:44:36,279 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:44:36,280 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.55099987984. 2022-02-08 00:44:36,280 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:44:36,282 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:44:36,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:44:37,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:44:37,492 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:44:37,492 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:44:37,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:44:37,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:44:37,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:44:37,586 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:44:37,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:44:37,663 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:44:37,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:44:38,132 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:44:38,134 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.85300016403. 2022-02-08 00:44:38,134 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:44:38,135 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:44:38,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:44:39,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:44:39,427 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:44:39,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:44:39,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:44:39,512 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:44:39,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:44:39,668 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:44:39,755 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:44:39,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:44:39,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:44:40,211 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:44:40,223 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.08899998665. ############################################################ # # RAM FootPrint: 3887.8984375 # ############################################################ 2022-02-08 00:44:40,540 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 00:44:40,540 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 114 seconds. Reason: Iteration:4 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 1301 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 1349 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:46:34,628 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 00:46:34,637 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 00:46:39,648 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:46:39,987 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:40,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:46:40,309 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:40,311 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:46:40,323 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:46:40,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:46:40,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:46:40,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:46:40,385 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:46:40,421 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:46:40,434 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:46:40,434 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 00:46:40,496 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['949 memcached\n'], []) 2022-02-08 00:46:40,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 00:46:40,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 00:46:40,552 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['949 memcached\n'], []) 2022-02-08 00:46:40,552 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:46:40,555 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:46:40,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:40,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:46:40,914 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:40,914 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:46:40,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:46:40,938 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:46:40,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:46:40,966 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:46:40,977 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:46:40,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:46:41,005 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:46:41,005 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 00:46:41,063 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['81821 memcached\n'], []) 2022-02-08 00:46:41,063 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 00:46:41,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 00:46:41,108 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['81821 memcached\n'], []) 2022-02-08 00:46:41,108 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:46:41,111 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:46:41,188 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:41,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:46:41,505 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:41,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:46:41,530 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:46:41,543 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:46:41,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:46:41,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:46:41,586 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:46:41,601 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:46:41,615 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:46:41,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 00:46:41,660 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['124178 memcached\n'], []) 2022-02-08 00:46:41,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 00:46:41,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 00:46:41,710 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['124178 memcached\n'], []) 2022-02-08 00:46:41,710 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:46:41,713 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:46:42,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:42,767 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:46:42,884 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:42,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:46:42,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:46:42,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:46:42,936 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:46:42,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:46:42,971 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:46:43,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:46:43,016 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:46:43,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 00:46:43,075 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['111553 memcached\n'], []) 2022-02-08 00:46:43,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 00:46:43,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 00:46:43,125 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['111553 memcached\n'], []) 2022-02-08 00:46:43,125 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:46:43,127 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:46:43,211 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:43,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:46:43,507 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:43,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:46:43,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:46:43,533 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:46:43,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:46:43,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:46:43,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:46:43,594 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:46:43,611 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:46:43,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 00:46:43,658 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['20308 memcached\n'], []) 2022-02-08 00:46:43,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 00:46:43,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 00:46:43,714 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['20308 memcached\n'], []) 2022-02-08 00:46:43,714 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:46:43,726 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:46:43,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:44,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:46:44,082 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:44,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:46:44,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:46:44,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:46:44,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:46:44,145 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:46:44,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:46:44,173 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:46:44,186 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:46:44,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 00:46:44,232 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['14141 memcached\n'], []) 2022-02-08 00:46:44,232 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 00:46:44,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 00:46:44,285 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['14141 memcached\n'], []) 2022-02-08 00:46:44,285 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:46:44,288 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:46:44,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:44,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:46:44,644 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:44,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:46:44,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:46:44,670 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:46:44,681 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:46:44,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:46:44,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:46:44,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:46:44,740 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:46:44,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 00:46:44,790 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['55127 memcached\n'], []) 2022-02-08 00:46:44,790 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 00:46:44,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 00:46:44,838 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['55127 memcached\n'], []) 2022-02-08 00:46:44,838 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:46:44,841 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:46:44,951 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:45,158 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:46:45,217 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:45,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:46:45,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:46:45,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:46:45,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:46:45,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:46:45,303 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:46:45,316 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:46:45,329 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:46:45,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 00:46:45,380 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['49100 memcached\n'], []) 2022-02-08 00:46:45,380 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 00:46:45,405 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 00:46:45,428 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['49100 memcached\n'], []) 2022-02-08 00:46:45,428 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:46:45,430 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:46:45,530 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:45,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:46:45,802 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:45,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:46:45,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:46:45,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:46:45,839 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:46:45,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:46:45,867 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:46:45,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:46:45,891 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:46:45,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 00:46:45,938 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['113180 memcached\n'], []) 2022-02-08 00:46:45,938 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 00:46:45,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 00:46:45,984 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['113180 memcached\n'], []) 2022-02-08 00:46:45,984 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:46:45,986 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:46:46,066 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:46,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:46:46,355 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:46,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:46:46,369 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:46:46,382 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:46:46,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:46:46,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:46:46,443 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:46:46,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:46:46,477 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:46:46,477 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:46:46,552 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['61262 memcached\n'], []) 2022-02-08 00:46:46,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 00:46:46,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:46:46,615 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['61262 memcached\n'], []) 2022-02-08 00:46:46,617 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:46:46,618 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:46:46,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:46,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:46:46,976 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:46,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:46:46,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:46:47,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:46:47,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:46:47,035 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:46:47,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:46:47,061 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:46:47,076 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:46:47,092 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 00:46:47,119 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:46:47,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:47,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:46:47,403 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:47,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:47,573 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:46:47,591 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 00:46:47,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:46:47,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:47,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:46:47,648 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 00:46:47,648 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:46:47,651 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:46:47,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:47,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:46:48,020 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:48,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:46:48,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:46:48,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:46:48,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:46:48,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:46:48,112 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:46:48,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:46:48,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:46:48,153 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 00:46:48,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:46:48,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:48,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:46:48,520 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:48,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:48,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:46:48,759 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 00:46:48,760 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:46:48,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:48,808 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:46:48,822 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 00:46:48,822 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:46:48,825 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:46:48,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 1399 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:46:49,094 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:46:49,155 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:49,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:46:49,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:46:49,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:46:49,196 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:46:49,213 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:46:49,229 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:46:49,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:46:49,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:46:49,276 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 00:46:49,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:46:49,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:49,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:46:49,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:49,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:49,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:46:49,828 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 00:46:49,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:46:49,844 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:49,892 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:46:49,904 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 00:46:49,904 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:46:49,907 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 00:46:49,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:50,211 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 00:46:50,276 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:50,276 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 00:46:50,290 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 00:46:50,303 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 00:46:50,316 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 00:46:50,334 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 00:46:50,346 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 00:46:50,359 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 00:46:50,374 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:46:50,388 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 00:46:50,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:46:50,448 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:50,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:46:50,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:50,505 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:50,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:46:50,532 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 00:46:50,532 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:46:50,545 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:50,558 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:46:50,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:50,594 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:46:50,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:50,632 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:50,653 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:50,670 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:50,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:50,704 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:50,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:50,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:50,762 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 00:46:50,765 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:46:50,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:51,056 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:46:51,111 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:51,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:46:51,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:46:51,137 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:46:51,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:46:51,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:46:51,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:46:51,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:46:51,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:46:51,220 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 00:46:51,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:46:51,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:51,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:46:51,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:51,369 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:51,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:46:51,404 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 00:46:51,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:46:51,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:51,438 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:46:51,451 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 00:46:51,451 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:46:51,454 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 00:46:51,538 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:51,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 00:46:51,832 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:51,834 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 00:46:51,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 00:46:51,859 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 00:46:51,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 00:46:51,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 00:46:51,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 00:46:51,913 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 00:46:51,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:46:51,940 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 00:46:51,969 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:46:51,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:52,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:46:52,015 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:52,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:52,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:46:52,056 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 00:46:52,056 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:46:52,071 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:52,086 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:46:52,099 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:52,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:46:52,142 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:52,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:52,193 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:52,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:52,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:52,262 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:52,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:52,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:52,315 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 00:46:52,316 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:46:52,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:52,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:46:52,648 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:52,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:46:52,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:46:52,673 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:46:52,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:46:52,698 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:46:52,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:46:52,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:46:52,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:46:52,750 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 00:46:52,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:46:52,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:52,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:46:53,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:53,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:53,336 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:46:53,349 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 00:46:53,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:46:53,364 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:53,382 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:46:53,397 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 00:46:53,397 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:46:53,400 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:46:53,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:53,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:46:53,796 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:53,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:46:53,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:46:53,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:46:53,844 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:46:53,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:46:53,880 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:46:53,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:46:53,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:46:53,927 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 00:46:53,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:46:53,971 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:54,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:46:54,290 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:54,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:54,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:46:54,513 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 00:46:54,515 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:46:54,545 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:54,594 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:46:54,615 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 00:46:54,615 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:46:54,618 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 00:46:54,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:54,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 00:46:54,986 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:54,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 00:46:55,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 00:46:55,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 00:46:55,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 00:46:55,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 00:46:55,053 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 00:46:55,066 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 00:46:55,079 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:46:55,094 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 00:46:55,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:46:55,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:55,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:46:55,165 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:55,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:55,193 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:46:55,207 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 00:46:55,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:46:55,221 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:55,237 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:46:55,252 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:55,272 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:46:55,290 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:55,308 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:55,326 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:55,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:55,367 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:55,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:55,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:55,431 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:55,450 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 00:46:55,453 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 00:46:55,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:55,770 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 00:46:55,828 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:55,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 00:46:55,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 00:46:55,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 00:46:55,867 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 00:46:55,882 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 00:46:55,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 00:46:55,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 00:46:55,920 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:46:55,934 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 00:46:55,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:46:55,974 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:55,989 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:46:56,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:56,015 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:56,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:46:56,039 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 00:46:56,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:46:56,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:56,066 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:46:56,081 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:56,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:46:56,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:56,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:56,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:56,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:56,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:56,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:56,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:56,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:46:56,318 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 00:46:56,319 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:46:56,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:56,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:46:56,703 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:56,704 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:46:56,719 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:46:56,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:46:56,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:46:56,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:46:56,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:46:56,785 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:46:56,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:46:56,812 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 00:46:56,839 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:46:56,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:57,043 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:46:57,229 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:57,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:57,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:46:57,496 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 00:46:57,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:46:57,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:57,545 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:46:57,559 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 00:46:57,561 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:46:57,562 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:46:57,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:57,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:46:57,937 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:57,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:46:57,951 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:46:57,966 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:46:57,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:46:57,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:46:58,012 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:46:58,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:46:58,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:46:58,062 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 00:46:58,091 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:46:58,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:58,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:46:58,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:58,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:46:58,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:46:58,681 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 00:46:58,681 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:46:58,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:58,719 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:46:58,734 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 00:46:58,734 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:46:58,737 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:46:58,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:46:59,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:46:59,361 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:46:59,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:46:59,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:46:59,517 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:46:59,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:46:59,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:46:59,609 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:46:59,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:46:59,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:46:59,657 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 00:46:59,793 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:46:59,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:46:59,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:47:00,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:47:00,211 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:47:00,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:47:00,358 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 00:47:00,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:47:00,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:47:00,456 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:47:00,490 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 00:47:00,490 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:47:00,497 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:47:00,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:47:00,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:47:00,867 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:47:00,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:47:00,882 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:47:00,895 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:47:00,910 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:47:00,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:47:00,938 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:47:00,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:47:00,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:47:00,980 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 00:47:01,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:47:01,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:47:01,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:47:01,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:47:01,434 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:47:01,525 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:47:01,543 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 00:47:01,543 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:47:01,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:47:01,588 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:47:01,604 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 00:47:01,604 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:47:01,605 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 00:47:01,704 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:47:01,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 00:47:01,993 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:47:01,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 00:47:02,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 00:47:02,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 00:47:02,038 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 00:47:02,058 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 00:47:02,073 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 00:47:02,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 00:47:02,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:47:02,121 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 00:47:02,147 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:47:02,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:47:02,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:47:02,188 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:47:02,201 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:47:02,216 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:47:02,229 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 00:47:02,229 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:47:02,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:47:02,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:47:02,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:47:02,293 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:47:02,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:47:02,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:47:02,354 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:47:02,372 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:47:02,392 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:47:02,415 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:47:02,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:47:02,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:47:02,489 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 00:47:02,493 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:47:02,496 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:47:02,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:47:02,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:47:02,865 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:47:02,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:47:02,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:47:02,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:47:02,903 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:47:02,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:47:02,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:47:02,941 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:47:02,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:03,063 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:47:03,069 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:03,069 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:05,071 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:05,210 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:05,210 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:07,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:07,346 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:07,348 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:09,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:09,516 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:09,516 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:11,517 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:11,660 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:11,660 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:13,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:13,805 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:13,805 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:15,808 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:15,953 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:15,953 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:17,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:18,098 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:18,098 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:20,099 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:20,265 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:20,265 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:22,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:22,424 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:22,424 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:24,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:24,569 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:24,569 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:26,571 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:26,719 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:26,720 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:28,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:28,874 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:28,874 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:30,874 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:31,016 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:31,017 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:33,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:33,155 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:33,155 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:35,158 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:35,305 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:35,305 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:37,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:37,453 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:37,453 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:39,454 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:39,601 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:39,601 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:41,601 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:41,750 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:41,750 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:43,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:43,894 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:43,894 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:45,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:46,038 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:46,038 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:48,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:48,186 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:48,186 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 1429 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:47:50,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:50,335 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:50,335 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:52,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:52,528 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:52,528 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:54,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:54,914 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:54,914 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:56,915 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:57,061 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:57,062 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:47:59,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:47:59,210 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:47:59,210 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:48:01,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:01,358 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:48:01,358 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:48:03,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:03,512 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:48:03,512 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:48:05,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:05,661 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:48:05,661 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:48:07,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:07,811 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:48:07,812 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:48:09,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:09,963 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:48:09,964 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:48:11,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:12,111 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:48:12,112 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:48:14,112 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:14,267 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:48:14,267 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:48:16,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:16,421 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:48:16,421 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:48:18,426 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:18,578 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:48:18,578 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 76.0859999657. 2022-02-08 00:48:18,578 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:48:18,582 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:48:19,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:48:19,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:48:19,444 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:48:19,446 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:48:19,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:48:19,487 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:48:19,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:48:19,533 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:48:19,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:48:19,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:48:19,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:19,951 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:48:19,953 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.37400007248. 2022-02-08 00:48:19,953 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:48:19,954 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:48:20,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:48:20,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:48:20,408 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:48:20,408 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:48:20,423 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:48:20,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:48:20,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:48:20,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:48:20,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:48:20,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:48:20,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:21,286 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:48:21,286 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.33399987221. 2022-02-08 00:48:21,286 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:48:21,289 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:48:21,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:48:22,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:48:22,301 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:48:22,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:48:22,323 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:48:22,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:48:22,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:48:22,414 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:48:22,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:48:22,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:48:22,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:22,609 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:48:22,621 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.33400011063. 2022-02-08 00:48:22,621 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:48:22,622 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:48:23,397 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:48:23,653 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:48:23,736 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:48:23,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:48:23,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:48:23,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:48:23,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:48:23,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:48:23,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:48:23,841 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:48:23,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:24,437 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:48:24,437 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.81599998474. 2022-02-08 00:48:24,437 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:48:24,440 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:48:25,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:48:25,283 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:48:25,421 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:48:25,421 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:48:25,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:48:25,456 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:48:25,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:48:25,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:48:25,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:48:25,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:48:25,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:26,033 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:48:26,033 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.59699988365. 2022-02-08 00:48:26,033 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:48:26,036 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:48:26,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:48:27,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:48:27,515 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:48:27,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:48:27,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:48:27,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:48:27,660 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:48:27,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:48:27,729 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:48:27,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:48:27,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:28,434 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:48:28,443 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.40900015831. 2022-02-08 00:48:28,444 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:48:28,446 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:48:29,446 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:48:30,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:48:30,819 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:48:30,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:48:30,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:48:30,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:48:30,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:48:31,015 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:48:31,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:48:31,122 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:48:31,193 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:31,733 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:48:31,733 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.28999996185. 2022-02-08 00:48:31,733 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:48:31,736 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:48:32,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:48:32,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:48:32,887 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:48:32,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:48:32,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:48:32,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:48:32,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:48:32,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:48:33,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:48:33,053 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:48:33,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:33,479 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:48:33,479 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.74500012398. 2022-02-08 00:48:33,479 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:48:33,480 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:48:34,515 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:48:35,130 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:48:35,355 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:48:35,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:48:35,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:48:35,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:48:35,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:48:35,561 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:48:35,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:48:35,635 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:48:35,663 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:48:36,533 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:48:36,535 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.05599999428. ############################################################ # # RAM FootPrint: 3890.44140625 # ############################################################ 2022-02-08 00:48:36,867 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 00:48:36,867 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 92 seconds. Reason: Iteration:5 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 1437 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 1462 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:50:08,871 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 00:50:08,872 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 00:50:13,878 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:50:14,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:14,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:50:14,766 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:14,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:50:14,788 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:50:14,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:50:14,839 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:50:14,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:50:14,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:50:14,914 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:50:14,930 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:50:14,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 00:50:14,996 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['2038 memcached\n'], []) 2022-02-08 00:50:14,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 00:50:15,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 00:50:15,043 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['2038 memcached\n'], []) 2022-02-08 00:50:15,043 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:50:15,046 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:50:15,119 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:15,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:50:15,413 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:15,413 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:50:15,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:50:15,441 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:50:15,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:50:15,482 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:50:15,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:50:15,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:50:15,522 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:50:15,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 00:50:15,584 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['82556 memcached\n'], []) 2022-02-08 00:50:15,584 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 00:50:15,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 00:50:15,634 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['82556 memcached\n'], []) 2022-02-08 00:50:15,634 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:50:15,635 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:50:15,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:15,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:50:16,052 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:16,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:50:16,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:50:16,079 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:50:16,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:50:16,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:50:16,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:50:16,132 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:50:16,148 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:50:16,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 00:50:16,196 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['124950 memcached\n'], []) 2022-02-08 00:50:16,196 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 00:50:16,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 00:50:16,240 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['124950 memcached\n'], []) 2022-02-08 00:50:16,242 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:50:16,244 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:50:16,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:16,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:50:16,595 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:16,596 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:50:16,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:50:16,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:50:16,640 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:50:16,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:50:16,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:50:16,688 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:50:16,704 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:50:16,704 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 00:50:16,746 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['112442 memcached\n'], []) 2022-02-08 00:50:16,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 00:50:16,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 00:50:16,796 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['112442 memcached\n'], []) 2022-02-08 00:50:16,796 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:50:16,809 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:50:16,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:17,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:50:17,178 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:17,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:50:17,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:50:17,206 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:50:17,219 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:50:17,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:50:17,247 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:50:17,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:50:17,275 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:50:17,275 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 00:50:17,319 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['21072 memcached\n'], []) 2022-02-08 00:50:17,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 00:50:17,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 00:50:17,365 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['21072 memcached\n'], []) 2022-02-08 00:50:17,365 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:50:17,367 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:50:17,448 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:17,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:50:17,711 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:17,711 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:50:17,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:50:17,739 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:50:17,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:50:17,767 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:50:17,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:50:17,793 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:50:17,808 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:50:17,808 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 00:50:17,871 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['14876 memcached\n'], []) 2022-02-08 00:50:17,872 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 00:50:17,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 00:50:17,918 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['14876 memcached\n'], []) 2022-02-08 00:50:17,918 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:50:17,921 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:50:18,010 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:18,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:50:18,296 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:18,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:50:18,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:50:18,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:50:18,331 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:50:18,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:50:18,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:50:18,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:50:18,398 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:50:18,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 00:50:18,444 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['55859 memcached\n'], []) 2022-02-08 00:50:18,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 00:50:18,479 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 00:50:18,523 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['55859 memcached\n'], []) 2022-02-08 00:50:18,523 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:50:18,525 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:50:18,624 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:18,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:50:18,907 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:18,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:50:18,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:50:18,938 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:50:18,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:50:18,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:50:18,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:50:19,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:50:19,032 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:50:19,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 00:50:19,086 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['49833 memcached\n'], []) 2022-02-08 00:50:19,086 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 00:50:19,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 00:50:19,137 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['49833 memcached\n'], []) 2022-02-08 00:50:19,137 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:50:19,140 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:50:19,213 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:19,434 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:50:19,516 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:19,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:50:19,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:50:19,551 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:50:19,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:50:19,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:50:19,596 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:50:19,609 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:50:19,622 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:50:19,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 00:50:19,664 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['113916 memcached\n'], []) 2022-02-08 00:50:19,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 00:50:19,688 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 00:50:19,709 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['113916 memcached\n'], []) 2022-02-08 00:50:19,709 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:50:19,710 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:50:19,793 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:20,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:50:20,076 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:20,076 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:50:20,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:50:20,107 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:50:20,119 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:50:20,135 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:50:20,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:50:20,160 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:50:20,174 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:50:20,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:50:20,233 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['62005 memcached\n'], []) 2022-02-08 00:50:20,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 00:50:20,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:50:20,289 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['62005 memcached\n'], []) 2022-02-08 00:50:20,289 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:50:20,292 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:50:20,377 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:20,596 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:50:20,661 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:20,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:50:20,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:50:20,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:50:20,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:50:20,714 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:50:20,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:50:20,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:50:20,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:50:20,767 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 00:50:20,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:50:20,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:22,066 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:50:22,198 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:22,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:22,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:50:22,407 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 00:50:22,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:50:22,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:22,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:50:22,473 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 00:50:22,473 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:50:22,476 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:50:22,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:22,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:50:22,855 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:22,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:50:22,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:50:22,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:50:22,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:50:22,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:50:22,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:50:22,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:50:22,959 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:50:22,974 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 00:50:23,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:50:23,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:23,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:50:23,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:23,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:23,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:50:23,559 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 00:50:23,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:50:23,573 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:23,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:50:23,619 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 00:50:23,619 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:50:23,621 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:50:23,709 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:23,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:50:23,983 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:23,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:50:23,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:50:24,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:50:24,028 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:50:24,043 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:50:24,056 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:50:24,071 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:50:24,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:50:24,098 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 00:50:24,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:50:24,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:25,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:50:25,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:25,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:25,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:50:25,818 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 00:50:25,818 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:50:25,831 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:25,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:50:25,891 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 00:50:25,891 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:50:25,894 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 00:50:25,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:26,188 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 00:50:26,240 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:26,242 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 00:50:26,255 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 00:50:26,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 00:50:26,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 00:50:26,293 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 00:50:26,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 00:50:26,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 00:50:26,334 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:50:26,348 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 00:50:26,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:50:26,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:26,403 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:50:26,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:26,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:26,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:50:26,457 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 00:50:26,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:50:26,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:26,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:50:26,499 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:26,520 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:50:26,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:26,561 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:26,581 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:26,599 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:26,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:26,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:26,660 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:26,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:26,700 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 00:50:26,703 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:50:26,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:27,015 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:50:27,086 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:27,086 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:50:27,102 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:50:27,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:50:27,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:50:27,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:50:27,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:50:27,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:50:27,188 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:50:27,203 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 00:50:27,354 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:50:27,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:27,408 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:50:27,446 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:27,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:27,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:50:27,509 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 00:50:27,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:50:27,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:27,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:50:27,559 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 00:50:27,559 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:50:27,562 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 00:50:27,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:27,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 00:50:27,950 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:27,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 00:50:27,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 00:50:27,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 00:50:27,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 00:50:28,010 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 00:50:28,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 00:50:28,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 00:50:28,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:50:28,062 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 00:50:28,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:50:28,104 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:28,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:50:28,132 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:28,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:28,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:50:28,174 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 00:50:28,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:50:28,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:28,201 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:50:28,213 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:28,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:50:28,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:28,267 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:28,288 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:28,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:28,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:28,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:28,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:28,377 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:28,395 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 00:50:28,398 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:50:28,471 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:28,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:50:28,740 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:28,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:50:28,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:50:28,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:50:28,785 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:50:28,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:50:28,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:50:28,844 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:50:28,862 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:50:28,882 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 00:50:28,928 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:50:28,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:30,349 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:50:30,517 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:30,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:30,734 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:50:30,749 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 00:50:30,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:50:30,762 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:30,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:50:30,813 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 00:50:30,813 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:50:30,815 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:50:30,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:31,102 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:50:31,194 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:31,196 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:50:31,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:50:31,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:50:31,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:50:31,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:50:31,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:50:31,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:50:31,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:50:31,316 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 00:50:31,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:50:31,359 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:31,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:50:31,645 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:31,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:31,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:50:31,851 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 00:50:31,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:50:31,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:31,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:50:31,917 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 00:50:31,917 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:50:31,920 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 00:50:32,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:32,211 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 00:50:32,270 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:32,272 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 00:50:32,285 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 00:50:32,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 00:50:32,308 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 00:50:32,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 00:50:32,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 00:50:32,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 00:50:32,377 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:50:32,391 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 00:50:32,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:50:32,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:32,451 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:50:32,466 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:32,479 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:32,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:50:32,505 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 00:50:32,505 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:50:32,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:32,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:50:32,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:32,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:50:32,586 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:32,604 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:32,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:32,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:32,660 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:32,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:32,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:32,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:32,734 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 00:50:32,736 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 00:50:32,834 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:33,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 00:50:33,114 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:33,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 00:50:33,130 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 00:50:33,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 00:50:33,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 00:50:33,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 00:50:33,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 00:50:33,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 00:50:33,216 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:50:33,230 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 00:50:33,257 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:50:33,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:33,288 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:50:33,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:33,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:33,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:50:33,345 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 00:50:33,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:50:33,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:33,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:50:33,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:33,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:50:33,434 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:33,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:33,474 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:33,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:33,517 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:33,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:33,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:33,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:33,605 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 00:50:33,608 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:50:33,683 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:33,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:50:33,960 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:33,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:50:33,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:50:33,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:50:34,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:50:34,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:50:34,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:50:34,045 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:50:34,058 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:50:34,086 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 00:50:34,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:50:34,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:35,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:50:35,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:36,132 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:36,249 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:50:36,263 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 00:50:36,265 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:50:36,278 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:36,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:50:36,313 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 00:50:36,313 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:50:36,315 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:50:36,405 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:36,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:50:36,687 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:36,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:50:36,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:50:36,711 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:50:36,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:50:36,739 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:50:36,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:50:36,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:50:36,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:50:36,789 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 00:50:36,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:50:36,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:38,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:50:38,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:38,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:38,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:50:38,559 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 00:50:38,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:50:38,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:38,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:50:38,611 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 00:50:38,611 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:50:38,614 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:50:38,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:38,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:50:39,025 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:39,025 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:50:39,038 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:50:39,051 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:50:39,063 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:50:39,079 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:50:39,096 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:50:39,109 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:50:39,127 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:50:39,153 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 00:50:39,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:50:39,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:39,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:50:39,410 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:39,482 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:39,584 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:50:39,598 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 00:50:39,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:50:39,614 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:39,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:50:39,657 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 00:50:39,657 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:50:39,658 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:50:39,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:39,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:50:40,049 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:40,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:50:40,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:50:40,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:50:40,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:50:40,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:50:40,122 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:50:40,135 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:50:40,150 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:50:40,198 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 00:50:40,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:50:40,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:41,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:50:41,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:41,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:41,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:50:41,832 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 00:50:41,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:50:41,846 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:41,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:50:41,897 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 00:50:41,898 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:50:41,900 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 00:50:42,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:42,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 00:50:42,308 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:42,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 00:50:42,323 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 00:50:42,336 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 00:50:42,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 00:50:42,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 00:50:42,385 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 00:50:42,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 00:50:42,413 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:50:42,428 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 00:50:42,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:50:42,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:42,487 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:50:42,502 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:42,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:50:42,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:50:42,542 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 00:50:42,543 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:50:42,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:50:42,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:50:42,584 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:42,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:50:42,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:42,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:42,665 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:42,686 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:42,719 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:42,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:42,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:42,793 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:50:42,813 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 00:50:42,813 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:50:42,815 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:50:42,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:50:43,153 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:50:43,217 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:50:43,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:50:43,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:50:43,247 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:50:43,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:50:43,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:50:43,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:50:43,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:50:43,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:50:43,500 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:50:43,506 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:50:43,506 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:50:45,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:50:45,630 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:50:45,634 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:50:45,634 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:50:47,635 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:50:47,752 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:50:47,757 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:50:47,757 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 1480 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:50:49,757 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:50:49,882 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:50:49,888 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:50:49,888 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:50:51,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:50:52,010 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:50:52,017 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:50:52,019 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:50:54,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:50:54,164 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:50:54,165 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:50:56,165 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:50:56,312 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:50:56,312 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:50:58,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:50:58,466 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:50:58,467 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:00,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:00,619 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:00,621 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:02,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:02,778 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:02,779 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:04,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:04,957 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:04,957 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:06,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:07,105 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:07,105 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:09,107 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:09,252 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:09,253 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:11,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:11,440 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:11,441 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:13,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:13,630 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:13,631 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:15,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:15,799 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:15,799 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:17,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:18,105 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:18,105 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:20,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:20,440 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:20,440 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:22,443 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:22,592 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:22,592 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:24,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:24,992 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:24,993 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:26,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:27,311 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:27,312 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:29,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:29,529 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:29,529 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:31,530 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:31,977 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:31,979 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:33,979 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:34,137 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:34,137 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:36,137 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:36,400 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:36,400 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:38,400 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:38,776 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:38,776 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:40,778 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:41,619 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:41,619 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:43,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:44,453 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:44,453 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:46,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:47,190 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:47,190 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 1480 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:51:49,191 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:50,023 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:51:50,023 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:51:52,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:52,551 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:51:52,551 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 69.737000227. 2022-02-08 00:51:52,552 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:51:52,553 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:51:53,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:51:53,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:51:53,657 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:51:53,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:51:53,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:51:53,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:51:53,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:51:53,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:51:53,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:51:53,757 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:51:53,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:54,233 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:51:54,234 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.68299984932. 2022-02-08 00:51:54,234 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:51:54,236 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:51:55,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:51:56,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:51:56,198 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:51:56,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:51:56,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:51:56,249 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:51:56,278 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:51:56,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:51:56,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:51:56,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:51:56,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:51:57,130 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:51:57,131 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.89600014687. 2022-02-08 00:51:57,131 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:51:57,132 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:51:58,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:51:59,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:51:59,266 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:51:59,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:51:59,290 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:51:59,321 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:51:59,344 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:51:59,372 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:51:59,395 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:51:59,436 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:51:59,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:52:00,223 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:52:00,226 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.09500002861. 2022-02-08 00:52:00,226 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:52:00,227 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:52:01,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:52:02,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:52:02,846 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:52:02,846 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:52:02,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:52:02,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:52:02,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:52:02,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:52:02,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:52:03,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:52:03,040 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:52:03,848 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:52:03,848 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.62199997902. 2022-02-08 00:52:03,848 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:52:03,851 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:52:04,683 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:52:05,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:52:05,450 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:52:05,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:52:05,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:52:05,584 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:52:05,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:52:05,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:52:05,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:52:05,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:52:05,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:52:06,598 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:52:06,598 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.74900007248. 2022-02-08 00:52:06,598 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:52:06,601 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:52:07,762 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:52:08,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:52:09,117 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:52:09,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:52:09,247 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:52:09,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:52:09,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:52:09,780 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:52:09,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:52:09,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:52:10,137 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:52:11,299 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:52:11,299 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.7009999752. 2022-02-08 00:52:11,301 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:52:11,302 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:52:12,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:52:12,727 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:52:13,329 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:52:13,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:52:13,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:52:13,602 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:52:13,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:52:13,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:52:13,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:52:14,107 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:52:14,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:52:15,591 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:52:15,591 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 00:52:17,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:52:18,319 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:52:18,319 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 00:52:20,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:52:20,957 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:52:20,959 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 9.65799999237. 2022-02-08 00:52:20,959 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:52:20,961 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:52:21,716 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:52:22,135 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:52:22,259 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:52:22,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:52:22,275 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:52:22,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:52:22,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:52:22,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:52:22,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:52:22,359 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:52:22,374 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:52:23,088 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:52:23,088 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.12999987602. 2022-02-08 00:52:23,088 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:52:23,091 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:52:24,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:52:25,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:52:25,641 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:52:25,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:52:25,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:52:25,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:52:25,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:52:25,762 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:52:25,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:52:25,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:52:25,918 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:52:26,990 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:52:26,990 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.90199995041. ############################################################ # # RAM FootPrint: 3888.3671875 # ############################################################ 2022-02-08 00:52:27,071 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 00:52:27,071 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 111 seconds. Reason: Iteration:6 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 1493 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 1519 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:54:18,078 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 00:54:18,078 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 00:54:23,085 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:54:23,602 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:23,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:54:23,983 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:23,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:54:24,020 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:54:24,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:54:24,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:54:24,135 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:54:24,178 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:54:24,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:54:24,246 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:54:24,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 00:54:24,355 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['3166 memcached\n'], []) 2022-02-08 00:54:24,357 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 00:54:24,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 00:54:24,459 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['3166 memcached\n'], []) 2022-02-08 00:54:24,459 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:54:24,460 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:54:24,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:24,752 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:54:24,806 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:24,806 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:54:24,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:54:24,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:54:24,844 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:54:24,858 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:54:24,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:54:24,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:54:24,895 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:54:24,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 00:54:24,957 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['83217 memcached\n'], []) 2022-02-08 00:54:24,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 00:54:24,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 00:54:25,002 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['83217 memcached\n'], []) 2022-02-08 00:54:25,002 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:54:25,005 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:54:25,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:25,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:54:25,390 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:25,390 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:54:25,403 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:54:25,415 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:54:25,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:54:25,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:54:25,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:54:25,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:54:25,483 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:54:25,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 00:54:25,533 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['125655 memcached\n'], []) 2022-02-08 00:54:25,533 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 00:54:25,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 00:54:25,585 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['125655 memcached\n'], []) 2022-02-08 00:54:25,585 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:54:25,588 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:54:25,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:25,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:54:25,960 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:25,961 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:54:25,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:54:25,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:54:26,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:54:26,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:54:26,038 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:54:26,051 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:54:26,068 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:54:26,068 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 00:54:26,131 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['113248 memcached\n'], []) 2022-02-08 00:54:26,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 00:54:26,168 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 00:54:26,196 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['113248 memcached\n'], []) 2022-02-08 00:54:26,197 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:54:26,198 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:54:26,290 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:26,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:54:26,592 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:26,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:54:26,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:54:26,617 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:54:26,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:54:26,647 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:54:26,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:54:26,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:54:26,684 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:54:26,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 00:54:26,733 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['21746 memcached\n'], []) 2022-02-08 00:54:26,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 00:54:26,767 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 00:54:26,790 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['21746 memcached\n'], []) 2022-02-08 00:54:26,790 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:54:26,792 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:54:26,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:27,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:54:27,141 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:27,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:54:27,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:54:27,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:54:27,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:54:27,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:54:27,213 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:54:27,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:54:27,242 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:54:27,242 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 00:54:27,292 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['15538 memcached\n'], []) 2022-02-08 00:54:27,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 00:54:27,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 00:54:27,335 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['15538 memcached\n'], []) 2022-02-08 00:54:27,335 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:54:27,338 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:54:27,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:27,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:54:27,690 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:27,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:54:27,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:54:27,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:54:27,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:54:27,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:54:27,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:54:27,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:54:27,785 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:54:27,785 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 00:54:27,834 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['56571 memcached\n'], []) 2022-02-08 00:54:27,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 00:54:27,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 00:54:27,878 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['56571 memcached\n'], []) 2022-02-08 00:54:27,878 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:54:27,880 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:54:28,099 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:28,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:54:28,394 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:28,395 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:54:28,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:54:28,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:54:28,438 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:54:28,454 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:54:28,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:54:28,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:54:28,493 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:54:28,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 00:54:28,548 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['50538 memcached\n'], []) 2022-02-08 00:54:28,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 00:54:28,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 00:54:28,601 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['50538 memcached\n'], []) 2022-02-08 00:54:28,602 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:54:28,605 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:54:28,691 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:28,920 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:54:28,976 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:28,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:54:28,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:54:29,002 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:54:29,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:54:29,038 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:54:29,059 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:54:29,073 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:54:29,091 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:54:29,091 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 00:54:29,151 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['114585 memcached\n'], []) 2022-02-08 00:54:29,153 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 00:54:29,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 00:54:29,201 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['114585 memcached\n'], []) 2022-02-08 00:54:29,203 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:54:29,204 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:54:29,290 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:29,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:54:29,605 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:29,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:54:29,619 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:54:29,632 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:54:29,645 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:54:29,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:54:29,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:54:29,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:54:29,706 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:54:29,706 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:54:29,773 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['62714 memcached\n'], []) 2022-02-08 00:54:29,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 00:54:29,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:54:29,822 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['62714 memcached\n'], []) 2022-02-08 00:54:29,822 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:54:29,823 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:54:29,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:30,127 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:54:30,194 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:30,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:54:30,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:54:30,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:54:30,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:54:30,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:54:30,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:54:30,283 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:54:30,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:54:30,315 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 00:54:30,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:54:30,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:31,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:54:31,474 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:31,576 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:31,665 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:54:31,683 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 00:54:31,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:54:31,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:31,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:54:31,747 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 00:54:31,747 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:54:31,750 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:54:31,823 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:32,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:54:32,108 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:32,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:54:32,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:54:32,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:54:32,147 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:54:32,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:54:32,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:54:32,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:54:32,204 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:54:32,229 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 00:54:32,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:54:32,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:33,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:54:33,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:34,109 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:34,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:54:34,223 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 00:54:34,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:54:34,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:34,280 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:54:34,293 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 00:54:34,293 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:54:34,296 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:54:34,395 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:34,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:54:34,648 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:34,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:54:34,663 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:54:34,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:54:34,686 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:54:34,701 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:54:34,714 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:54:34,727 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:54:34,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:54:34,756 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 00:54:34,790 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:54:34,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:34,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:54:35,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:35,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:35,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:54:35,345 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 00:54:35,346 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:54:35,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:35,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:54:35,397 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 00:54:35,398 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:54:35,400 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 00:54:35,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:35,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 00:54:35,747 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:35,747 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 00:54:35,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 00:54:35,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 00:54:35,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 00:54:35,798 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 00:54:35,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 00:54:35,821 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 00:54:35,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:54:35,845 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 00:54:35,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:54:35,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:35,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:54:35,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:35,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:35,936 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:54:35,948 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 00:54:35,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:54:35,961 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:35,974 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:54:35,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:36,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:54:36,028 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:36,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:36,068 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:36,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:36,104 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:36,122 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:36,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:36,183 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:36,201 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 00:54:36,203 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:54:36,288 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:36,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:54:36,566 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:36,566 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:54:36,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:54:36,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:54:36,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:54:36,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:54:36,632 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:54:36,647 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:54:36,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:54:36,674 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 00:54:36,701 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:54:36,716 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:36,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:54:37,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:37,061 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:37,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:54:37,098 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 00:54:37,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:54:37,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:37,132 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:54:37,150 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 00:54:37,150 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:54:37,153 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 00:54:37,227 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:37,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 00:54:37,505 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:37,505 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 00:54:37,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 00:54:37,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 00:54:37,553 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 00:54:37,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 00:54:37,581 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 00:54:37,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 00:54:37,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:54:37,618 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 00:54:37,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:54:37,670 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:37,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:54:37,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:37,714 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:37,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:54:37,739 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 00:54:37,739 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:54:37,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:37,767 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:54:37,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:37,801 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:54:37,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:37,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:37,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:37,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:37,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:37,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:37,943 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:37,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:37,982 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 00:54:37,983 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:54:38,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:38,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:54:38,315 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:38,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:54:38,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:54:38,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:54:38,354 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:54:38,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:54:38,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:54:38,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:54:38,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:54:38,424 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 00:54:38,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:54:38,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:38,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:54:38,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:38,999 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:39,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:54:39,128 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 00:54:39,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:54:39,142 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:39,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:54:39,177 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 00:54:39,177 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:54:39,178 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:54:39,263 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:39,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:54:39,555 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:39,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:54:39,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:54:39,586 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:54:39,601 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:54:39,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:54:39,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:54:39,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:54:39,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:54:39,687 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 00:54:39,714 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:54:39,732 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:41,186 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:54:41,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:41,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:41,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:54:41,586 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 00:54:41,588 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:54:41,614 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:41,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:54:41,667 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 00:54:41,667 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:54:41,671 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 00:54:41,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:41,966 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 00:54:42,029 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:42,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 00:54:42,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 00:54:42,053 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 00:54:42,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 00:54:42,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 00:54:42,096 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 00:54:42,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 00:54:42,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:54:42,144 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 00:54:42,173 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:54:42,186 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:42,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:54:42,214 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:42,227 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:42,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:54:42,253 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 00:54:42,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:54:42,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:42,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:54:42,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:42,316 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:54:42,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:42,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:42,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:42,408 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:42,428 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:42,448 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:42,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:42,502 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:42,522 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 00:54:42,525 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 00:54:42,599 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:42,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 00:54:42,888 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:42,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 00:54:42,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 00:54:42,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 00:54:42,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 00:54:42,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 00:54:42,977 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 00:54:42,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 00:54:43,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:54:43,029 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 00:54:43,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:54:43,091 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:43,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:54:43,122 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:43,135 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:43,153 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:54:43,167 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 00:54:43,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:54:43,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:43,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:54:43,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:43,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:54:43,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:43,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:43,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:43,403 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:43,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:43,448 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:43,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:43,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:43,515 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 00:54:43,517 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:54:43,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:43,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:54:43,867 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:43,867 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:54:43,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:54:43,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:54:43,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:54:43,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:54:43,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:54:43,948 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:54:43,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:54:43,976 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 00:54:44,002 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:54:44,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:44,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:54:44,431 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:44,571 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:44,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:54:44,700 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 00:54:44,701 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:54:44,714 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:44,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:54:44,755 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 00:54:44,756 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:54:44,757 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:54:44,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:45,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:54:45,114 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:45,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:54:45,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:54:45,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:54:45,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:54:45,170 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:54:45,181 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:54:45,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:54:45,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:54:45,220 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 00:54:45,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:54:45,263 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:45,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:54:45,619 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:45,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:45,846 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:54:45,871 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 00:54:45,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:54:45,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:45,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:54:45,924 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 00:54:45,926 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:54:45,927 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:54:46,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:46,263 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:54:46,331 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:46,331 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:54:46,344 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:54:46,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:54:46,400 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:54:46,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:54:46,489 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:54:46,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:54:46,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:54:46,546 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 00:54:46,573 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:54:46,589 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:46,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:54:46,887 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:46,951 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:47,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:54:47,038 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 00:54:47,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:54:47,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:47,094 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:54:47,107 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 00:54:47,107 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:54:47,108 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:54:47,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:47,420 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:54:47,486 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:47,487 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:54:47,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:54:47,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:54:47,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:54:47,543 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:54:47,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:54:47,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:54:47,589 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:54:47,608 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 00:54:47,635 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:54:47,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:47,818 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:54:47,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:48,091 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:48,186 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:54:48,200 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 00:54:48,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:54:48,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:48,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:54:48,269 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 00:54:48,269 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:54:48,270 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 00:54:48,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:48,604 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 00:54:48,681 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:48,681 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 00:54:48,698 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 00:54:48,711 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 00:54:48,724 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 00:54:48,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 00:54:48,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 00:54:48,770 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 00:54:48,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:54:48,802 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 00:54:48,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:54:48,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:48,857 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:54:48,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:48,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:54:48,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:54:48,911 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 00:54:48,913 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:54:48,926 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:54:48,940 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:54:48,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:48,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:54:48,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:49,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:49,038 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 5 queries submitted, 0 failed, 1545 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:54:49,056 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:49,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:49,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:49,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:49,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:54:49,160 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 00:54:49,161 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:54:49,163 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:54:49,232 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:54:49,443 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:54:49,503 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:54:49,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:54:49,517 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:54:49,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:54:49,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:54:49,563 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:54:49,576 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:54:49,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:54:49,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:54:49,726 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:54:49,730 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:54:49,730 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:54:51,732 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:54:51,845 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:54:51,849 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:54:51,849 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:54:53,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:54:53,973 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:54:53,977 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:54:53,979 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:54:55,979 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:54:56,115 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:54:56,115 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:54:58,117 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:54:58,259 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:54:58,259 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:00,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:00,408 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:00,410 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:02,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:02,573 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:02,575 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:04,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:04,723 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:04,723 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:06,724 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:06,881 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:06,881 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:08,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:09,036 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:09,038 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:11,038 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:11,181 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:11,181 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:13,183 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:13,335 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:13,335 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:15,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:15,490 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:15,490 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:17,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:17,642 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:17,642 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:19,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:19,795 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:19,796 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:21,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:22,538 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:22,538 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:24,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:24,937 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:24,937 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:26,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:27,538 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:27,539 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:29,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:29,822 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:29,823 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:31,823 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:32,197 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:32,197 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:34,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:34,548 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:34,549 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:36,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:36,780 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:36,780 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:38,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:38,961 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:38,961 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:40,961 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:41,361 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:41,361 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:43,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:43,572 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:43,573 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:45,573 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:45,971 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:45,971 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:47,971 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:48,296 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:48,296 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 1548 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:55:50,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:50,523 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:50,523 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:52,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:52,755 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:52,755 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:55:54,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:55,174 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:55:55,176 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 66.0150001049. 2022-02-08 00:55:55,176 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:55:55,178 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:55:55,757 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:55:56,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:55:56,316 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:55:56,318 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:55:56,334 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:55:56,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:55:56,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:55:56,385 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:55:56,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:55:56,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:55:56,443 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:56,880 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:55:56,881 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.70499992371. 2022-02-08 00:55:56,881 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:55:56,882 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:55:57,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:55:57,727 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:55:57,826 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:55:57,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:55:57,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:55:57,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:55:57,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:55:57,923 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:55:57,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:55:57,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:55:58,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:55:58,352 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:55:58,352 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 00:56:00,354 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:00,612 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:56:00,612 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 00:56:02,614 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:03,474 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:56:03,476 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 00:56:05,479 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:06,121 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:56:06,121 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 00:56:08,122 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:08,796 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:56:08,796 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 00:56:10,798 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:11,776 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:56:11,776 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 00:56:13,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:14,345 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:56:14,345 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 00:56:16,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:17,013 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:56:17,015 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 20.1340000629. 2022-02-08 00:56:17,015 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:56:17,016 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:56:17,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:56:18,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:56:18,990 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:56:18,992 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:56:19,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:56:19,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:56:19,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:56:19,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:56:19,872 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:56:20,002 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:56:20,206 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:20,984 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:56:20,984 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 00:56:22,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:23,892 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:56:23,892 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 00:56:25,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:26,736 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:56:26,737 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 00:56:28,739 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:29,423 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:56:29,423 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 00:56:31,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:31,832 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:56:31,832 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 00:56:33,834 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:34,305 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:56:34,306 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 00:56:36,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:36,484 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:56:36,486 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 19.4709999561. 2022-02-08 00:56:36,486 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:56:36,487 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:56:37,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:56:37,793 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:56:38,082 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:56:38,084 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:56:38,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:56:38,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:56:38,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:56:38,276 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:56:38,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:56:38,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:56:38,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:39,230 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:56:39,230 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.74499988556. 2022-02-08 00:56:39,230 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:56:39,233 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:56:40,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:56:41,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:56:41,256 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:56:41,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:56:41,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:56:41,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:56:41,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:56:41,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:56:41,403 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:56:41,441 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:56:41,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:42,730 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:56:42,732 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.5. 2022-02-08 00:56:42,732 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:56:42,733 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:56:43,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:56:44,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:56:44,854 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:56:44,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:56:44,877 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:56:44,908 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:56:44,928 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:56:44,951 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:56:44,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:56:45,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:56:45,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:45,779 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:56:45,779 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.04799985886. 2022-02-08 00:56:45,779 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:56:45,782 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:56:46,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:56:46,839 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:56:46,960 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:56:46,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:56:46,979 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:56:47,007 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:56:47,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:56:47,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:56:47,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:56:47,107 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:56:47,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 1560 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:56:52,088 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:56:52,088 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.30900001526. 2022-02-08 00:56:52,088 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:56:52,091 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:56:52,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:56:53,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:56:53,446 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:56:53,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:56:53,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:56:53,482 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:56:53,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:56:53,510 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:56:53,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:56:53,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:56:53,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:54,016 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:56:54,016 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.9279999733. 2022-02-08 00:56:54,016 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:56:54,019 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:56:54,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:56:54,762 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:56:54,845 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:56:54,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:56:54,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:56:54,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:56:54,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:56:54,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:56:54,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:56:54,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:56:54,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:56:59,329 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:56:59,331 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.3140001297. ############################################################ # # RAM FootPrint: 3895.72265625 # ############################################################ 2022-02-08 00:56:59,369 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 00:56:59,369 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 65 seconds. Reason: Iteration:7 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 1586 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:58:04,377 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 00:58:04,377 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 00:58:09,381 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:58:09,880 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:10,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:58:10,227 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:10,229 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:58:10,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:58:10,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:58:10,295 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:58:10,311 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:58:10,323 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:58:10,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:58:10,361 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:58:10,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 00:58:10,424 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['4260 memcached\n'], []) 2022-02-08 00:58:10,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 00:58:10,466 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 00:58:10,486 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['4260 memcached\n'], []) 2022-02-08 00:58:10,487 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:58:10,490 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:58:10,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:10,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:58:10,838 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:10,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:58:10,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:58:10,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:58:10,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:58:10,908 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:58:10,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:58:10,940 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:58:10,960 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:58:10,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 00:58:11,020 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['83953 memcached\n'], []) 2022-02-08 00:58:11,020 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 00:58:11,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 00:58:11,068 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['83953 memcached\n'], []) 2022-02-08 00:58:11,068 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:58:11,071 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:58:11,650 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:11,914 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:58:12,010 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:12,010 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:58:12,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:58:12,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:58:12,059 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:58:12,073 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:58:12,089 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:58:12,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:58:12,145 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:58:12,145 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 00:58:12,207 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['126390 memcached\n'], []) 2022-02-08 00:58:12,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 00:58:12,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 00:58:12,253 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['126390 memcached\n'], []) 2022-02-08 00:58:12,253 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:58:12,256 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:58:12,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:12,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:58:13,278 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:13,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:58:13,295 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:58:13,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:58:13,380 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:58:13,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:58:13,414 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:58:13,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:58:13,463 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:58:13,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 00:58:13,533 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['114075 memcached\n'], []) 2022-02-08 00:58:13,533 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 00:58:13,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 00:58:13,589 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['114075 memcached\n'], []) 2022-02-08 00:58:13,589 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:58:13,592 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:58:13,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:13,905 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:58:13,963 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:13,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:58:13,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:58:13,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:58:14,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:58:14,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:58:14,035 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:58:14,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:58:14,065 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:58:14,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 00:58:14,109 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['22477 memcached\n'], []) 2022-02-08 00:58:14,109 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 00:58:14,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 00:58:14,158 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['22477 memcached\n'], []) 2022-02-08 00:58:14,158 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:58:14,161 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:58:14,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:14,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:58:14,563 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:14,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:58:14,581 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:58:14,594 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:58:14,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:58:14,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:58:14,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:58:14,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:58:14,670 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:58:14,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 00:58:14,750 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['16317 memcached\n'], []) 2022-02-08 00:58:14,752 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 00:58:14,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 00:58:14,803 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['16317 memcached\n'], []) 2022-02-08 00:58:14,803 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:58:14,806 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:58:14,885 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:15,086 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:58:15,144 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:15,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:58:15,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:58:15,168 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:58:15,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:58:15,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:58:15,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:58:15,221 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:58:15,237 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:58:15,237 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 00:58:15,301 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['57309 memcached\n'], []) 2022-02-08 00:58:15,301 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 00:58:15,334 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 00:58:15,354 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['57309 memcached\n'], []) 2022-02-08 00:58:15,354 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:58:15,357 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:58:15,433 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:15,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:58:15,713 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:15,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:58:15,729 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:58:15,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:58:15,757 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:58:15,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:58:15,788 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:58:15,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:58:15,812 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:58:15,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 00:58:15,859 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['51296 memcached\n'], []) 2022-02-08 00:58:15,859 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 00:58:15,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 00:58:15,920 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['51296 memcached\n'], []) 2022-02-08 00:58:15,920 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:58:15,921 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:58:16,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:16,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:58:16,306 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:16,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:58:16,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:58:16,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:58:16,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:58:16,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:58:16,380 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:58:16,392 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:58:16,407 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:58:16,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 00:58:16,473 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['115320 memcached\n'], []) 2022-02-08 00:58:16,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 00:58:16,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 00:58:16,520 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['115320 memcached\n'], []) 2022-02-08 00:58:16,520 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:58:16,523 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:58:16,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:16,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:58:16,892 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:16,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:58:16,910 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:58:16,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:58:16,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:58:16,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:58:16,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:58:16,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:58:16,996 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 00:58:16,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:58:17,144 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['63447 memcached\n'], []) 2022-02-08 00:58:17,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 00:58:17,186 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 00:58:17,213 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['63447 memcached\n'], []) 2022-02-08 00:58:17,213 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:58:17,216 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 00:58:17,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:17,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 00:58:17,575 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:17,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 00:58:17,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 00:58:17,604 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 00:58:17,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 00:58:17,635 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 00:58:17,653 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 00:58:17,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 00:58:17,688 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:58:17,707 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 00:58:17,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:58:17,760 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:17,913 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:58:18,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:18,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:18,227 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:58:18,252 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 00:58:18,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:58:18,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:18,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:58:18,312 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 00:58:18,312 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 00:58:18,315 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 00:58:18,397 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:18,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 00:58:18,709 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:18,709 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 00:58:18,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 00:58:18,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 00:58:18,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 00:58:18,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 00:58:18,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 00:58:18,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 00:58:18,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:58:18,842 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 00:58:18,872 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:58:18,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:19,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:58:19,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:19,359 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:19,466 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:58:19,480 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 00:58:19,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:58:19,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:19,530 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:58:19,549 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 00:58:19,549 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 00:58:19,552 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 00:58:19,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:19,834 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 00:58:19,894 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:19,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 00:58:19,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 00:58:19,923 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 00:58:19,936 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 00:58:19,956 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 00:58:19,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 00:58:19,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 00:58:19,999 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:58:20,013 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 00:58:20,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:58:20,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:20,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:58:20,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:20,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:20,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:58:21,013 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 00:58:21,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:58:21,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:21,043 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:58:21,055 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 00:58:21,056 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 00:58:21,059 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 00:58:21,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:21,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 00:58:21,391 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:21,392 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 00:58:21,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 00:58:21,415 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 00:58:21,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 00:58:21,441 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 00:58:21,454 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 00:58:21,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 00:58:21,479 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:58:21,493 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 00:58:21,517 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:58:21,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:21,545 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:58:21,558 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:21,571 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:21,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:58:21,595 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 00:58:21,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:58:21,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:21,619 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:58:21,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:21,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:58:21,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:21,693 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:21,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:21,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:21,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:21,778 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:21,801 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:21,823 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:21,851 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 00:58:21,857 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 00:58:21,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:22,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 00:58:22,236 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:22,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 00:58:22,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 00:58:22,265 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 00:58:22,278 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 00:58:22,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 00:58:22,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 00:58:22,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 00:58:22,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:58:22,351 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 00:58:22,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:58:22,477 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:22,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:58:22,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:22,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:22,890 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:58:22,905 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 00:58:22,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:58:22,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:22,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:58:22,960 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 00:58:22,960 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 00:58:22,963 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 00:58:23,056 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:23,275 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 00:58:23,336 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:23,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 00:58:23,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 00:58:23,364 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 00:58:23,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 00:58:23,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 00:58:23,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 00:58:23,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 00:58:23,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:58:23,446 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 00:58:23,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:58:23,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:23,502 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:58:23,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:23,530 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:23,545 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:58:23,559 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 00:58:23,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:58:23,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:23,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:58:23,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:23,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:58:23,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:23,673 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:23,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:23,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:23,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:23,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:23,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:23,836 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:23,858 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 00:58:23,861 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 00:58:23,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:24,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 00:58:24,207 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:24,209 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 00:58:24,221 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 00:58:24,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 00:58:24,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 00:58:24,263 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 00:58:24,276 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 00:58:24,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 00:58:24,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:58:24,319 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 00:58:24,357 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:58:24,372 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:24,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:58:24,739 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:24,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:24,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:58:24,993 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 00:58:24,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:58:25,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:25,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:58:25,040 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 00:58:25,040 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 00:58:25,046 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:58:25,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:25,346 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 00:58:25,407 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:25,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 00:58:25,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 00:58:25,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 00:58:25,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 00:58:25,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 00:58:25,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 00:58:25,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 00:58:25,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:58:25,535 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 00:58:25,563 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:58:25,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:25,747 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:58:25,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:26,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:26,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:58:26,128 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 00:58:26,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:58:26,142 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:26,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:58:26,194 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 00:58:26,194 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 00:58:26,197 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 00:58:26,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:26,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 00:58:26,558 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:26,558 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 00:58:26,571 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 00:58:26,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 00:58:26,596 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 00:58:26,614 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 00:58:26,627 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 00:58:26,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 00:58:26,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:58:26,667 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 00:58:26,693 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:58:26,706 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:26,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:58:26,734 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:26,749 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:26,762 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:58:26,776 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 00:58:26,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:58:26,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:26,808 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:58:26,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:26,846 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:58:26,869 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:26,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:26,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:26,948 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:26,971 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:26,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:27,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:27,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:27,062 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 00:58:27,065 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 00:58:27,135 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:27,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 00:58:27,413 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:27,413 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 00:58:27,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 00:58:27,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 00:58:27,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 00:58:27,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 00:58:27,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 00:58:27,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 00:58:27,507 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:58:27,522 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 00:58:27,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:58:27,563 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:27,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:58:27,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:27,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:27,617 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:58:27,630 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 00:58:27,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:58:27,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:27,655 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:58:27,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:27,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:58:27,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:27,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:27,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:27,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:27,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:27,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:27,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:27,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:27,913 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 00:58:27,915 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:58:28,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:28,237 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:58:28,298 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:28,298 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:58:28,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:58:28,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:58:28,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:58:28,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:58:28,369 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:58:28,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:58:28,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:58:28,413 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 00:58:28,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:58:28,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:28,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:58:28,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:28,979 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:29,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:58:29,118 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 00:58:29,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:58:29,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:29,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:58:29,164 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 00:58:29,164 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:58:29,167 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:58:29,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:29,482 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:58:29,538 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:29,538 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:58:29,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:58:29,563 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:58:29,576 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:58:29,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:58:29,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:58:29,619 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:58:29,632 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:58:29,647 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 00:58:29,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:58:29,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:29,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:58:30,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:30,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:30,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:58:30,253 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 00:58:30,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:58:30,267 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:30,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:58:30,309 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 00:58:30,309 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:58:30,312 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:58:30,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:30,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:58:30,704 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:30,706 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:58:30,727 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:58:30,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:58:30,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:58:30,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:58:30,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:58:30,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:58:30,844 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:58:30,859 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 00:58:30,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:58:30,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:30,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:58:31,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:31,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:31,229 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:58:31,263 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 00:58:31,265 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:58:31,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:31,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:58:31,355 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 00:58:31,355 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:58:31,357 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 00:58:31,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:31,640 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 00:58:31,703 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:31,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 00:58:31,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 00:58:31,729 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 00:58:31,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 00:58:31,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 00:58:31,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 00:58:31,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 00:58:31,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:58:31,806 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 00:58:31,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:58:31,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:32,033 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:58:32,193 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:32,308 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:32,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:58:32,418 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 00:58:32,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:58:32,433 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:32,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:58:32,464 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 00:58:32,466 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 00:58:32,467 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 00:58:32,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:32,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 00:58:32,898 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:32,900 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 00:58:32,913 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 00:58:32,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 00:58:32,940 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 00:58:32,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 00:58:32,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 00:58:32,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 00:58:32,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 00:58:33,013 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 00:58:33,038 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 00:58:33,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:33,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 00:58:33,107 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:33,130 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 00:58:33,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 00:58:33,165 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 00:58:33,165 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 00:58:33,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 00:58:33,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 00:58:33,209 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:33,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 00:58:33,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:33,276 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:33,301 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:33,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:33,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:33,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:33,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:33,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 00:58:33,440 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 00:58:33,440 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:58:33,443 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 00:58:33,528 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:58:33,739 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 00:58:33,819 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:58:33,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 00:58:33,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 00:58:33,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 00:58:33,862 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 00:58:33,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 00:58:33,890 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 00:58:33,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 00:58:33,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:58:34,029 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:58:34,033 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:58:34,033 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:58:36,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:58:36,160 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:58:36,165 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:58:36,165 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:58:38,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:58:38,282 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:58:38,288 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:58:38,288 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:58:40,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:58:40,407 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:58:40,414 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:58:40,414 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:58:42,415 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:58:42,530 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:58:42,536 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:58:42,536 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:58:44,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:58:44,645 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:58:44,651 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:58:44,651 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:58:46,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:58:46,765 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:58:46,769 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:58:46,769 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:58:48,770 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:58:48,888 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 00:58:48,894 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:58:48,894 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 1610 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:58:50,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:58:51,049 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:58:51,051 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:58:53,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:58:53,200 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:58:53,200 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:58:55,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:58:55,358 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:58:55,358 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:58:57,359 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:58:57,509 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:58:57,509 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:58:59,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:58:59,664 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:58:59,664 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:01,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:01,811 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:01,812 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:03,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:03,960 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:03,961 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:05,961 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:06,119 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:06,119 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:08,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:08,273 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:08,273 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:10,275 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:11,073 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:11,073 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:13,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:13,227 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:13,227 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:15,227 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:15,375 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:15,375 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:17,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:18,107 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:18,107 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:20,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:20,756 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:20,756 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:22,757 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:23,026 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:23,026 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:25,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:25,341 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:25,341 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:27,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:27,994 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:27,994 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:29,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:30,319 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:30,319 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:32,321 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:32,767 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:32,767 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:34,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:35,539 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:35,539 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:37,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:37,749 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:37,750 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:39,752 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:40,022 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:40,022 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:42,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:42,565 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:42,565 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:44,566 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:45,180 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:45,180 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:47,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:47,861 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:47,861 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 1610 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 00:59:49,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:50,313 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:50,313 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:52,313 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:52,749 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 00:59:52,750 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 00:59:54,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:55,155 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 00:59:55,155 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 81.7149999142. 2022-02-08 00:59:55,155 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:59:55,157 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 00:59:55,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:59:55,946 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 00:59:56,197 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:59:56,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 00:59:56,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 00:59:56,295 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 00:59:56,321 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 00:59:56,357 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 00:59:56,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 00:59:56,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 00:59:56,420 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:56,855 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 00:59:56,855 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.70000004768. 2022-02-08 00:59:56,855 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:59:56,858 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 00:59:57,719 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 00:59:58,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 00:59:58,546 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 00:59:58,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 00:59:58,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 00:59:58,601 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 00:59:58,647 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 00:59:58,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 00:59:58,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 00:59:58,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 00:59:58,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 00:59:59,115 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 00:59:59,115 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.25900006294. 2022-02-08 00:59:59,115 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 00:59:59,117 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 00:59:59,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:00:02,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:00:03,434 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:00:03,434 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:00:03,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:00:03,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:00:03,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:00:03,783 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:00:03,806 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:00:03,869 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:00:03,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:00:04,729 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:00:04,729 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.61400008202. 2022-02-08 01:00:04,730 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:00:04,732 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:00:05,961 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:00:06,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:00:07,226 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:00:07,227 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:00:07,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:00:07,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:00:07,566 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:00:07,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:00:07,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:00:07,788 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:00:07,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:00:08,592 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:00:08,592 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.86199998856. 2022-02-08 01:00:08,592 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:00:08,594 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:00:09,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:00:09,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:00:10,073 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:00:10,073 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:00:10,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:00:10,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:00:10,255 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:00:10,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:00:10,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:00:10,385 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:00:10,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:00:11,114 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:00:11,115 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.52300000191. 2022-02-08 01:00:11,115 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:00:11,117 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:00:12,150 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:00:12,864 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:00:13,305 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:00:13,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:00:13,558 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:00:13,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:00:13,858 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:00:13,926 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:00:14,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:00:14,214 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:00:14,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:00:15,931 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:00:15,931 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 01:00:17,931 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:00:18,224 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:00:18,224 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.1099998951. 2022-02-08 01:00:18,224 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:00:18,227 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:00:18,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:00:19,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:00:19,430 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:00:19,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:00:19,454 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:00:19,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:00:19,502 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:00:19,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:00:19,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:00:19,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:00:19,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:00:20,128 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:00:20,128 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.90400004387. 2022-02-08 01:00:20,128 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:00:20,131 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:00:20,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:00:20,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:00:20,532 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:00:20,532 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:00:20,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:00:20,561 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:00:20,573 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:00:20,589 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:00:20,604 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:00:20,619 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:00:20,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:00:21,076 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:00:21,078 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.949000120163. 2022-02-08 01:00:21,078 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:00:21,079 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:00:21,749 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:00:22,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:00:22,631 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:00:22,632 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:00:22,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:00:22,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:00:22,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:00:22,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:00:23,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:00:23,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:00:23,181 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:00:24,903 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:00:24,904 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.82500004768. ############################################################ # # RAM FootPrint: 3891.98828125 # ############################################################ 2022-02-08 01:00:25,144 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 01:00:25,144 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 82 seconds. Reason: Iteration:8 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 1626 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:01:47,153 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 01:01:47,153 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 5 queries submitted, 0 failed, 1651 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:01:52,157 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:01:52,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:01:52,721 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:01:52,819 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:01:52,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:01:52,834 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:01:52,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:01:52,862 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:01:52,880 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:01:52,892 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:01:52,913 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:01:52,926 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:01:52,926 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:01:52,987 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['5322 memcached\n'], []) 2022-02-08 01:01:52,987 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 01:01:53,020 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:01:53,042 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['5322 memcached\n'], []) 2022-02-08 01:01:53,042 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:01:53,045 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:01:53,119 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:01:53,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:01:53,384 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:01:53,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:01:53,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:01:53,408 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:01:53,421 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:01:53,436 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:01:53,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:01:53,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:01:53,471 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:01:53,471 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:01:53,520 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['84667 memcached\n'], []) 2022-02-08 01:01:53,520 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 01:01:53,543 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:01:53,563 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['84667 memcached\n'], []) 2022-02-08 01:01:53,563 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:01:53,565 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:01:53,642 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:01:53,880 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:01:53,980 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:01:53,982 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:01:54,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:01:54,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:01:54,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:01:54,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:01:54,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:01:54,081 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:01:54,096 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:01:54,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:01:54,163 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['127189 memcached\n'], []) 2022-02-08 01:01:54,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 01:01:54,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:01:54,210 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['127189 memcached\n'], []) 2022-02-08 01:01:54,210 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:01:54,211 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:01:54,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:01:54,538 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:01:54,607 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:01:54,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:01:54,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:01:54,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:01:54,653 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:01:54,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:01:54,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:01:54,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:01:54,719 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:01:54,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:01:54,763 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['114925 memcached\n'], []) 2022-02-08 01:01:54,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 01:01:54,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:01:54,818 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['114925 memcached\n'], []) 2022-02-08 01:01:54,818 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:01:54,819 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:01:54,900 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:01:55,145 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:01:55,216 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:01:55,216 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:01:55,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:01:55,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:01:55,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:01:55,278 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:01:55,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:01:55,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:01:55,319 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:01:55,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:01:55,371 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['23196 memcached\n'], []) 2022-02-08 01:01:55,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 01:01:55,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:01:55,417 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['23196 memcached\n'], []) 2022-02-08 01:01:55,417 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:01:55,420 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:01:55,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:01:55,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:01:55,763 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:01:55,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:01:55,778 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:01:55,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:01:55,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:01:55,816 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:01:55,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:01:55,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:01:55,854 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:01:55,854 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:01:55,894 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['16990 memcached\n'], []) 2022-02-08 01:01:55,895 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 01:01:55,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:01:55,941 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['16990 memcached\n'], []) 2022-02-08 01:01:55,941 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:01:55,944 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:01:56,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:01:56,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:01:56,303 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:01:56,303 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:01:56,316 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:01:56,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:01:56,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:01:56,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:01:56,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:01:56,382 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:01:56,397 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:01:56,397 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:01:56,443 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['58025 memcached\n'], []) 2022-02-08 01:01:56,443 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 01:01:56,464 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:01:56,486 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['58025 memcached\n'], []) 2022-02-08 01:01:56,486 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:01:56,487 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:01:56,571 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:01:56,785 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:01:56,848 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:01:56,849 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:01:56,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:01:56,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:01:56,895 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:01:56,913 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:01:56,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:01:56,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:01:56,950 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:01:56,951 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:01:57,000 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['51965 memcached\n'], []) 2022-02-08 01:01:57,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 01:01:57,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:01:57,072 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['51965 memcached\n'], []) 2022-02-08 01:01:57,072 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:01:57,075 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:01:57,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:01:57,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:01:57,424 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:01:57,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:01:57,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:01:57,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:01:57,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:01:57,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:01:57,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:01:57,502 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:01:57,522 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:01:57,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:01:57,555 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['116038 memcached\n'], []) 2022-02-08 01:01:57,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 01:01:57,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:01:57,601 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['116038 memcached\n'], []) 2022-02-08 01:01:57,601 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:01:57,604 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:01:57,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:01:57,926 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:01:57,987 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:01:57,987 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:01:58,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:01:58,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:01:58,035 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:01:58,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:01:58,066 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:01:58,081 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:01:58,095 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:01:58,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:01:58,180 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['64163 memcached\n'], []) 2022-02-08 01:01:58,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 01:01:58,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:01:58,239 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['64163 memcached\n'], []) 2022-02-08 01:01:58,240 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:01:58,242 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:01:58,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:01:58,533 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:01:58,601 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:01:58,601 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:01:58,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:01:58,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:01:58,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:01:58,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:01:58,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:01:58,691 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:01:58,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:01:58,730 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 01:01:58,755 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:01:58,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:01:58,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:01:59,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:01:59,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:01:59,276 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:01:59,292 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 01:01:59,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:01:59,308 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:01:59,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:01:59,361 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 01:01:59,361 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:01:59,364 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:01:59,451 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:01:59,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:01:59,717 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:01:59,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:01:59,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:01:59,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:01:59,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:01:59,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:01:59,783 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:01:59,798 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:01:59,811 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:01:59,825 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 01:01:59,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:01:59,864 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:00,068 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:02:00,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:00,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:00,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:02:00,483 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 01:02:00,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:02:00,505 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:00,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:02:00,559 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 01:02:00,559 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:02:00,562 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:02:00,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:02:00,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:02:00,904 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:02:00,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:02:00,931 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:02:00,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:02:00,961 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:02:00,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:02:01,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:02:01,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:02:01,035 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:02:01,051 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 01:02:01,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:02:01,089 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:01,252 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:02:01,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:01,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:01,599 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:02:01,612 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 01:02:01,612 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:02:01,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:01,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:02:01,684 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 01:02:01,684 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:02:01,687 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 01:02:01,765 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:02:01,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 01:02:02,062 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:02:02,063 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 01:02:02,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 01:02:02,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 01:02:02,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 01:02:02,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 01:02:02,158 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 01:02:02,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 01:02:02,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:02:02,201 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 01:02:02,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:02:02,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:02,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:02:02,276 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:02,290 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:02,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:02:02,315 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 01:02:02,316 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:02:02,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:02,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:02:02,357 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:02,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:02:02,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:02,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:02,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:02,474 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:02,497 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:02,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:02,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:02,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:02,585 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 01:02:02,586 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:02:02,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:02:02,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:02:02,971 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:02:02,971 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:02:02,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:02:03,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:02:03,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:02:03,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:02:03,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:02:03,068 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:02:03,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:02:03,104 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 01:02:03,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:02:03,145 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:03,206 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:02:03,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:03,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:03,311 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:02:03,326 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 01:02:03,326 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:02:03,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:03,369 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:02:03,394 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 01:02:03,394 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:02:03,401 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 01:02:03,474 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:02:03,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 01:02:03,756 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:02:03,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 01:02:03,770 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 01:02:03,785 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 01:02:03,798 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 01:02:03,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 01:02:03,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 01:02:03,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 01:02:03,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:02:03,868 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 01:02:03,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:02:03,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:03,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:02:03,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:03,948 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:03,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:02:03,973 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 01:02:03,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:02:03,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:04,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:02:04,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:04,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:02:04,071 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:04,091 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:04,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:04,135 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:04,155 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:04,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:04,196 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:04,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:04,234 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 01:02:04,237 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:02:04,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:02:04,505 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:02:04,559 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:02:04,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:02:04,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:02:04,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:02:04,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:02:04,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:02:04,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:02:04,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:02:04,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:02:04,661 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 01:02:04,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:02:04,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:04,880 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:02:05,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:05,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:05,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:02:05,315 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 01:02:05,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:02:05,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:05,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:02:05,378 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 01:02:05,378 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:02:05,382 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:02:05,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:02:05,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:02:05,744 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:02:05,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:02:05,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:02:05,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:02:05,803 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:02:05,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:02:05,836 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:02:05,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:02:05,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:02:05,884 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 01:02:05,910 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:02:05,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:06,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:02:06,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:06,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:06,471 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:02:06,490 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 01:02:06,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:02:06,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:06,543 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:02:06,559 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 01:02:06,559 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:02:06,561 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 01:02:06,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:02:06,874 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 01:02:06,933 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:02:06,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 01:02:06,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 01:02:06,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 01:02:06,982 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 01:02:07,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 01:02:07,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 01:02:07,025 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 01:02:07,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:02:07,059 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 01:02:07,089 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:02:07,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:07,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:02:07,147 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:07,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:07,178 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:02:07,265 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 01:02:07,265 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:02:07,283 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:07,303 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:02:07,318 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:07,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:02:07,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:07,385 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:07,408 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:07,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:07,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:07,474 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:07,497 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:07,520 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:07,542 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 01:02:07,543 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 01:02:07,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:02:07,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 01:02:07,904 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:02:07,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 01:02:07,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 01:02:07,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 01:02:07,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 01:02:07,971 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 01:02:07,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 01:02:08,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 01:02:08,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:02:08,039 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 01:02:08,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:02:08,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:08,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:02:08,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:08,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:08,153 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:02:08,165 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 01:02:08,165 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:02:08,178 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:08,191 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:02:08,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:08,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:02:08,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:08,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:08,293 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:08,323 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:08,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:08,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:08,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:08,410 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:08,430 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 01:02:08,431 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:02:08,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:02:08,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:02:08,825 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:02:08,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:02:08,839 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:02:08,854 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:02:08,867 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:02:08,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:02:08,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:02:08,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:02:08,926 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:02:08,940 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 01:02:08,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:02:08,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:10,173 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:02:10,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:10,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:10,663 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:02:10,677 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 01:02:10,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:02:10,691 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:10,719 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:02:10,733 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 01:02:10,733 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:02:10,734 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:02:10,803 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:02:11,035 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:02:11,096 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:02:11,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:02:11,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:02:11,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:02:11,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:02:11,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:02:11,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:02:11,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:02:11,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:02:11,219 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 01:02:11,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:02:11,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:11,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:02:11,665 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:11,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:11,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:02:11,927 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 01:02:11,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:02:11,951 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:11,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:02:12,019 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 01:02:12,019 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:02:12,022 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:02:12,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:02:12,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:02:12,405 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:02:12,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:02:12,428 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:02:12,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:02:12,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:02:12,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:02:12,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:02:12,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:02:12,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:02:12,542 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 01:02:12,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:02:12,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:12,691 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:02:12,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:12,867 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:12,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:02:12,954 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 01:02:12,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:02:12,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:13,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:02:13,062 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 01:02:13,062 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:02:13,063 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:02:13,150 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:02:13,354 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:02:13,408 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:02:13,410 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:02:13,423 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:02:13,434 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:02:13,446 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:02:13,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:02:13,471 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:02:13,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:02:13,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:02:13,529 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 01:02:13,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:02:13,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:13,739 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:02:13,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:14,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:14,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:02:14,148 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 01:02:14,150 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:02:14,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:14,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:02:14,211 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 01:02:14,211 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:02:14,217 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 01:02:14,311 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:02:14,545 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 01:02:14,614 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:02:14,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 01:02:14,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 01:02:14,642 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 01:02:14,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 01:02:14,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 01:02:14,691 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 01:02:14,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 01:02:14,721 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:02:14,739 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 01:02:14,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:02:14,780 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:14,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:02:14,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:14,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:02:14,841 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:02:14,857 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 01:02:14,857 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:02:14,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:02:14,885 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:02:14,900 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:14,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:02:14,956 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:14,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:15,007 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:15,030 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:15,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:15,079 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:15,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:15,130 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:02:15,153 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 01:02:15,153 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:02:15,154 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:02:15,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:02:15,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:02:15,535 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:02:15,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:02:15,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:02:15,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:02:15,581 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:02:15,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:02:15,609 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:02:15,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:02:15,637 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:15,798 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:02:15,802 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:15,802 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:17,803 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:17,920 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:02:17,924 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:17,924 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:19,926 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:20,036 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:02:20,042 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:20,042 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:22,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:22,160 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:02:22,164 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:22,165 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:24,165 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:24,283 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:02:24,289 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:24,289 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:26,290 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:26,434 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:26,434 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:28,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:28,582 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:28,582 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:30,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:30,734 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:30,736 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:32,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:32,885 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:32,887 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:34,887 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:35,048 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:35,048 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:37,051 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:37,196 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:37,197 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:39,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:39,344 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:39,345 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:41,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:41,500 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:41,506 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:43,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:43,645 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:43,645 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:45,647 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:45,801 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:45,801 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:47,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:47,940 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:47,941 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 1663 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:02:49,941 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:50,109 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:50,111 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:52,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:52,260 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:52,260 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:54,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:54,407 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:54,407 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:56,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:56,658 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:56,658 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:02:58,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:02:59,114 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:02:59,114 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:01,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:01,309 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:03:01,309 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:03,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:04,088 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:03:04,088 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:06,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:06,456 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:03:06,457 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:08,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:08,795 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:03:08,795 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:10,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:11,407 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:03:11,407 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:13,408 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:13,770 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:03:13,770 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:15,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:16,424 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:03:16,426 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:18,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:18,640 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:03:18,640 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:20,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:20,823 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:03:20,825 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:22,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:23,092 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:03:23,092 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:25,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:25,282 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:03:25,282 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:27,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:27,470 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:03:27,470 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:29,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:29,691 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:03:29,693 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:31,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:32,209 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:03:32,210 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:34,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:34,970 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:03:34,970 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:36,971 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:37,732 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:03:37,732 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:39,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:40,312 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:03:40,312 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:03:42,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:43,216 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:03:43,216 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 88.0629999638. 2022-02-08 01:03:43,217 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:03:43,219 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:03:43,892 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:03:44,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:03:44,407 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:03:44,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:03:44,423 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:03:44,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:03:44,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:03:44,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:03:44,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:03:44,507 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:03:44,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:44,973 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:03:44,980 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.76300001144. 2022-02-08 01:03:44,980 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:03:44,983 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:03:46,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:03:46,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:03:46,927 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:03:46,928 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:03:46,948 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:03:46,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:03:47,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:03:47,043 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:03:47,063 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:03:47,104 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:03:47,132 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:47,466 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:03:47,467 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.48699998856. 2022-02-08 01:03:47,467 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:03:47,470 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:03:48,326 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:03:48,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 5 queries submitted, 0 failed, 1669 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:03:49,163 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:03:49,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:03:49,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:03:49,331 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:03:49,377 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:03:49,438 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:03:49,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:03:49,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:03:49,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:51,178 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:03:51,183 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.71600008011. 2022-02-08 01:03:51,183 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:03:51,184 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:03:52,290 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:03:52,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:03:53,355 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:03:53,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:03:53,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:03:53,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:03:53,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:03:53,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:03:53,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:03:54,010 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:03:54,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:03:56,440 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:03:56,440 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.257999897. 2022-02-08 01:03:56,440 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:03:56,443 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:03:57,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:03:58,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:03:58,611 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:03:58,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:03:58,663 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:03:58,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:03:58,711 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:03:58,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:03:58,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:03:58,793 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:03:58,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:04:00,177 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:04:00,178 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.73799991608. 2022-02-08 01:04:00,178 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:04:00,180 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:04:01,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:04:02,414 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:04:02,762 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:04:02,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:04:02,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:04:02,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:04:03,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:04:03,346 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:04:03,466 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:04:03,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:04:03,683 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:04:04,857 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:04:04,857 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.67800021172. 2022-02-08 01:04:04,858 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:04:04,859 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:04:06,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:04:07,040 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:04:07,529 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:04:07,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:04:07,586 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:04:07,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:04:07,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:04:07,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:04:07,969 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:04:08,068 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:04:08,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:04:09,217 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:04:09,220 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.36199998856. 2022-02-08 01:04:09,220 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:04:09,223 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:04:09,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:04:10,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:04:10,415 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:04:10,415 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:04:10,434 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:04:10,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:04:10,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:04:10,492 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:04:10,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:04:10,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:04:10,553 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:04:11,075 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:04:11,075 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.85500001907. 2022-02-08 01:04:11,075 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:04:11,078 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:04:12,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:04:13,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:04:13,430 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:04:13,431 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:04:13,479 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:04:13,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:04:13,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:04:13,693 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:04:13,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:04:13,778 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:04:13,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:04:14,990 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:04:14,992 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.91600012779. ############################################################ # # RAM FootPrint: 3894.68359375 # ############################################################ 2022-02-08 01:04:15,052 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 01:04:15,052 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 113 seconds. Reason: Iteration:9 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 1690 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 1714 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:06:08,059 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 01:06:08,062 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 01:06:13,063 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:06:13,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:14,178 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:06:14,336 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:14,336 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:06:14,352 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:06:14,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:06:14,382 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:06:14,400 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:06:14,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:06:14,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:06:14,454 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:06:14,456 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:06:14,523 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['6426 memcached\n'], []) 2022-02-08 01:06:14,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 01:06:14,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:06:14,566 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['6426 memcached\n'], []) 2022-02-08 01:06:14,568 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:06:14,569 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:06:14,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:14,862 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:06:14,920 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:14,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:06:14,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:06:14,946 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:06:14,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:06:14,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:06:14,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:06:14,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:06:15,010 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:06:15,010 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:06:15,071 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['85395 memcached\n'], []) 2022-02-08 01:06:15,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 01:06:15,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:06:15,121 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['85395 memcached\n'], []) 2022-02-08 01:06:15,122 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:06:15,125 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:06:15,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:15,434 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:06:15,493 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:15,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:06:15,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:06:15,520 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:06:15,533 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:06:15,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:06:15,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:06:15,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:06:15,631 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:06:15,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:06:15,684 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['127873 memcached\n'], []) 2022-02-08 01:06:15,686 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 01:06:15,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:06:15,739 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['127873 memcached\n'], []) 2022-02-08 01:06:15,739 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:06:15,742 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:06:15,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:16,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:06:16,109 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:16,109 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:06:16,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:06:16,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:06:16,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:06:16,170 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:06:16,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:06:16,201 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:06:16,216 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:06:16,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:06:16,259 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['115750 memcached\n'], []) 2022-02-08 01:06:16,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 01:06:16,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:06:16,309 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['115750 memcached\n'], []) 2022-02-08 01:06:16,309 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:06:16,311 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:06:16,410 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:16,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:06:16,696 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:16,696 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:06:16,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:06:16,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:06:16,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:06:16,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:06:16,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:06:16,785 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:06:16,799 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:06:16,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:06:16,845 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['23922 memcached\n'], []) 2022-02-08 01:06:16,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 01:06:16,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:06:16,891 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['23922 memcached\n'], []) 2022-02-08 01:06:16,891 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:06:16,892 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:06:17,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:17,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:06:17,293 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:17,293 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:06:17,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:06:17,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:06:17,336 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:06:17,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:06:17,369 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:06:17,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:06:17,398 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:06:17,400 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:06:17,453 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['17714 memcached\n'], []) 2022-02-08 01:06:17,454 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 01:06:17,479 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:06:17,509 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['17714 memcached\n'], []) 2022-02-08 01:06:17,509 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:06:17,510 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:06:17,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:17,798 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:06:17,858 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:17,859 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:06:17,872 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:06:17,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:06:17,903 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:06:17,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:06:17,938 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:06:17,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:06:17,964 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:06:17,966 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:06:18,006 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['58771 memcached\n'], []) 2022-02-08 01:06:18,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 01:06:18,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:06:18,052 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['58771 memcached\n'], []) 2022-02-08 01:06:18,052 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:06:18,055 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:06:18,145 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:18,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:06:18,414 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:18,414 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:06:18,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:06:18,446 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:06:18,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:06:18,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:06:18,489 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:06:18,502 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:06:18,516 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:06:18,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:06:18,584 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['52692 memcached\n'], []) 2022-02-08 01:06:18,584 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 01:06:18,614 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:06:18,638 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['52692 memcached\n'], []) 2022-02-08 01:06:18,638 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:06:18,641 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:06:18,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:18,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:06:18,982 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:18,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:06:18,999 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:06:19,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:06:19,025 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:06:19,040 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:06:19,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:06:19,066 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:06:19,079 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:06:19,081 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:06:19,135 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['116769 memcached\n'], []) 2022-02-08 01:06:19,135 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 01:06:19,160 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:06:19,181 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['116769 memcached\n'], []) 2022-02-08 01:06:19,183 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:06:19,184 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:06:19,270 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:19,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:06:19,582 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:19,584 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:06:19,602 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:06:19,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:06:19,637 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:06:19,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:06:19,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:06:19,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:06:19,700 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:06:19,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:06:19,769 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['64860 memcached\n'], []) 2022-02-08 01:06:19,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 01:06:19,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:06:19,823 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['64860 memcached\n'], []) 2022-02-08 01:06:19,823 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:06:19,825 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:06:20,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:20,221 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:06:20,292 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:20,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:06:20,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:06:20,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:06:20,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:06:20,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:06:20,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:06:20,413 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:06:20,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:06:20,447 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 01:06:20,474 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:06:20,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:21,653 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:06:21,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:21,905 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:21,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:06:22,010 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 01:06:22,010 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:06:22,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:22,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:06:22,078 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 01:06:22,078 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:06:22,081 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:06:22,160 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:22,377 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:06:22,443 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:22,443 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:06:22,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:06:22,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:06:22,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:06:22,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:06:22,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:06:22,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:06:22,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:06:22,569 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 01:06:22,596 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:06:22,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:22,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:06:22,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:23,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:23,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:06:23,214 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 01:06:23,214 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:06:23,227 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:23,265 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:06:23,278 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 01:06:23,278 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:06:23,279 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:06:23,357 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:23,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:06:23,618 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:23,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:06:23,632 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:06:23,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:06:23,660 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:06:23,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:06:23,693 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:06:23,709 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:06:23,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:06:23,742 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 01:06:23,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:06:23,783 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:25,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:06:25,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:25,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:25,933 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:06:25,947 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 01:06:25,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:06:25,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:26,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:06:26,019 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 01:06:26,019 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:06:26,022 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 01:06:26,109 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:26,334 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 01:06:26,391 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:26,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 01:06:26,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 01:06:26,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 01:06:26,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 01:06:26,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 01:06:26,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 01:06:26,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 01:06:26,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:06:26,505 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 01:06:26,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:06:26,543 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:26,558 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:06:26,571 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:26,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:26,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:06:26,608 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 01:06:26,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:06:26,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:26,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:06:26,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:26,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:06:26,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:26,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:26,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:26,780 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:26,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:26,829 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:26,852 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:26,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:26,900 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 01:06:26,901 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:06:27,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:27,209 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:06:27,267 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:27,267 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:06:27,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:06:27,293 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:06:27,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:06:27,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:06:27,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:06:27,346 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:06:27,359 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:06:27,375 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 01:06:27,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:06:27,415 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:27,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:06:27,798 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:27,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:27,867 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:06:27,881 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 01:06:27,882 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:06:27,895 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:28,059 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:06:28,073 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 01:06:28,073 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:06:28,075 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 01:06:28,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:28,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 01:06:28,430 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:28,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 01:06:28,443 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 01:06:28,454 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 01:06:28,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 01:06:28,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 01:06:28,497 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 01:06:28,512 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 01:06:28,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:06:28,539 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 01:06:28,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:06:28,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:28,599 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:06:28,614 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:28,627 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:28,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:06:28,654 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 01:06:28,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:06:28,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:28,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:06:28,698 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:28,724 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:06:28,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:28,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:28,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:28,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:28,852 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:28,877 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:28,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:28,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:28,946 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 01:06:28,947 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:06:29,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:29,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:06:29,313 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:29,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:06:29,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:06:29,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:06:29,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:06:29,372 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:06:29,385 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:06:29,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:06:29,413 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:06:29,433 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 01:06:29,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:06:29,484 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:31,191 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:06:31,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:31,515 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:31,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:06:31,648 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 01:06:31,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:06:31,663 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:31,698 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:06:31,713 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 01:06:31,713 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:06:31,717 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:06:31,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:32,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:06:32,085 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:32,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:06:32,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:06:32,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:06:32,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:06:32,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:06:32,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:06:32,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:06:32,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:06:32,217 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 01:06:32,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:06:32,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:32,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:06:32,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:32,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:32,836 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:06:32,852 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 01:06:32,854 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:06:32,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:32,908 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:06:32,927 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 01:06:32,927 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:06:32,930 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 01:06:33,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:33,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 01:06:33,289 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:33,290 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 01:06:33,303 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 01:06:33,316 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 01:06:33,331 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 01:06:33,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 01:06:33,359 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 01:06:33,372 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 01:06:33,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:06:33,401 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 01:06:33,433 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:06:33,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:33,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:06:33,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:33,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:33,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:06:33,520 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 01:06:33,520 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:06:33,533 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:33,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:06:33,563 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:33,588 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:06:33,612 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:33,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:33,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:33,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:33,711 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:33,734 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:33,757 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:33,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:33,803 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 01:06:33,805 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 01:06:33,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:34,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 01:06:34,220 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:34,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 01:06:34,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 01:06:34,249 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 01:06:34,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 01:06:34,278 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 01:06:34,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 01:06:34,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 01:06:34,316 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:06:34,332 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 01:06:34,364 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:06:34,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:34,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:06:34,413 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:34,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:34,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:06:34,451 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 01:06:34,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:06:34,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:34,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:06:34,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:34,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:06:34,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:34,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:34,602 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:34,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:34,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:34,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:34,706 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:34,739 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:34,762 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 01:06:34,763 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:06:34,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:35,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:06:35,118 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:35,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:06:35,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:06:35,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:06:35,155 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:06:35,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:06:35,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:06:35,196 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:06:35,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:06:35,221 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 01:06:35,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:06:35,265 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:37,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:06:37,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:37,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:37,601 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:06:37,615 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 01:06:37,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:06:37,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:37,653 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:06:37,667 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 01:06:37,667 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:06:37,670 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:06:37,762 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:37,982 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:06:38,042 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:38,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:06:38,053 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:06:38,066 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:06:38,079 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:06:38,096 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:06:38,109 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:06:38,122 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:06:38,137 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:06:38,151 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 01:06:38,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:06:38,193 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:38,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:06:39,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:39,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:39,385 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:06:39,403 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 01:06:39,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:06:39,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:39,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:06:39,460 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 01:06:39,461 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:06:39,463 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:06:39,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:39,801 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:06:39,926 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:39,926 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:06:39,951 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:06:39,971 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:06:39,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:06:40,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:06:40,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:06:40,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:06:40,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:06:40,210 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 01:06:40,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:06:40,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:40,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:06:40,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:40,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:40,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:06:40,822 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 01:06:40,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:06:40,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:40,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:06:40,918 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 01:06:40,918 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:06:40,921 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:06:41,005 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:41,216 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:06:41,290 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:41,290 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:06:41,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:06:41,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:06:41,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:06:41,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:06:41,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:06:41,374 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:06:41,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:06:41,401 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 01:06:41,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:06:41,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:42,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:06:43,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:43,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:43,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:06:43,299 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 01:06:43,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:06:43,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:43,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:06:43,357 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 01:06:43,358 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:06:43,359 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 01:06:43,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:43,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 01:06:43,773 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:43,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 01:06:43,790 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 01:06:43,803 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 01:06:43,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 01:06:43,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 01:06:43,858 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 01:06:43,874 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 01:06:43,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:06:43,904 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 01:06:43,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:06:43,946 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:43,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:06:43,974 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:43,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:06:44,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:06:44,013 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 01:06:44,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:06:44,028 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:06:44,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:06:44,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:44,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:06:44,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:44,137 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:44,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:44,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:44,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:44,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:44,257 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:44,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:06:44,305 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 01:06:44,305 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:06:44,308 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:06:44,395 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:06:44,614 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:06:44,683 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:06:44,683 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:06:44,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:06:44,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:06:44,727 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:06:44,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:06:44,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:06:44,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:06:44,793 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:06:44,917 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:06:44,923 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:06:44,923 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:06:46,923 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:06:47,036 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:06:47,042 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:06:47,042 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:06:49,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 1737 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:06:49,151 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:06:49,157 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:06:49,157 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:06:51,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:06:51,276 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:06:51,282 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:06:51,282 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:06:53,283 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:06:53,394 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:06:53,400 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:06:53,400 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:06:55,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:06:55,522 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:06:55,528 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:06:55,529 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:06:57,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:06:57,642 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:06:57,648 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:06:57,648 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:06:59,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:06:59,775 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:06:59,779 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:06:59,779 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:01,780 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:01,897 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:07:01,901 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:01,901 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:03,903 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:04,052 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:04,052 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:06,053 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:06,197 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:06,197 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:08,198 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:08,355 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:08,355 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:10,357 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:10,517 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:10,517 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:12,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:12,674 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:12,674 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:14,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:14,823 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:14,823 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:16,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:16,970 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:16,970 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:18,971 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:19,138 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:19,138 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:21,140 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:21,703 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:21,704 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:23,704 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:23,989 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:23,989 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:25,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:26,200 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:26,200 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:28,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:28,450 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:28,450 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:30,451 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:30,608 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:30,608 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:32,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:32,818 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:32,818 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:34,818 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:34,994 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:34,994 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:36,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:37,150 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:37,151 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:39,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:39,328 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:39,328 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:41,331 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:41,585 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:41,585 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:43,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:43,747 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:43,749 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:45,749 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:46,349 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:46,349 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:48,349 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 1741 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:07:49,137 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:49,138 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:51,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:51,987 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:51,987 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:53,989 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:54,697 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:54,697 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:56,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:57,289 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:57,289 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:07:59,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:07:59,608 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:07:59,608 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:08:01,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:08:01,845 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:08:01,845 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 77.5390000343. 2022-02-08 01:08:01,845 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:08:01,848 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:08:02,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:08:02,632 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:08:02,720 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:08:02,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:08:02,734 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:08:02,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:08:02,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:08:02,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:08:02,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:08:02,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:08:02,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:08:03,220 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:08:03,220 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.37400007248. 2022-02-08 01:08:03,220 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:08:03,223 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:08:04,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:08:04,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:08:04,894 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:08:04,895 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:08:04,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:08:04,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:08:05,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:08:05,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:08:05,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:08:05,104 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:08:05,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:08:06,104 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:08:06,104 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.88300013542. 2022-02-08 01:08:06,104 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:08:06,105 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:08:06,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:08:07,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:08:07,766 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:08:07,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:08:07,793 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:08:07,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:08:07,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:08:07,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:08:07,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:08:08,028 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:08:08,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:08:08,842 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:08:08,842 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.73799991608. 2022-02-08 01:08:08,842 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:08:08,845 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:08:10,214 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:08:10,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:08:11,112 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:08:11,112 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:08:11,137 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:08:11,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:08:11,204 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:08:11,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:08:11,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:08:11,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:08:11,382 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:08:12,292 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:08:12,292 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.45000004768. 2022-02-08 01:08:12,292 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:08:12,296 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:08:13,311 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:08:13,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:08:14,453 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:08:14,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:08:14,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:08:14,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:08:15,028 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:08:15,224 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:08:15,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:08:15,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:08:15,691 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:08:17,119 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:08:17,119 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 01:08:19,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:08:19,486 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:08:19,486 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.1930000782. 2022-02-08 01:08:19,486 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:08:19,489 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:08:19,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:08:19,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:08:20,030 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:08:20,030 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:08:20,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:08:20,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:08:20,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:08:20,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:08:20,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:08:20,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:08:20,181 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:08:20,778 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:08:20,778 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.29100012779. 2022-02-08 01:08:20,779 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:08:20,779 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:08:21,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:08:22,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:08:22,792 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:08:22,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:08:22,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:08:23,035 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:08:23,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:08:23,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:08:23,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:08:23,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:08:23,532 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:08:24,665 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:08:24,665 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 01:08:26,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:08:26,888 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:08:26,890 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.11100006104. 2022-02-08 01:08:26,890 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:08:26,892 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:08:27,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:08:28,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:08:28,381 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:08:28,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:08:28,395 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:08:28,415 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:08:28,428 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:08:28,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:08:28,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:08:28,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:08:28,502 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:08:28,967 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:08:28,967 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.07699990273. 2022-02-08 01:08:28,967 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:08:28,969 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:08:29,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:08:30,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:08:30,694 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:08:30,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:08:30,795 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:08:30,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:08:30,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:08:30,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:08:31,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:08:31,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:08:31,086 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:08:32,119 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:08:32,119 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.15299987793. ############################################################ # # RAM FootPrint: 3896.76171875 # ############################################################ 2022-02-08 01:08:32,157 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 01:08:32,157 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 108 seconds. Reason: Iteration:10 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 1756 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 1782 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:10:20,161 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 01:10:20,161 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 01:10:25,165 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:10:25,624 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:25,852 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:10:25,937 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:25,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:10:25,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:10:25,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:10:25,992 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:10:26,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:10:26,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:10:26,056 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:10:26,069 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:10:26,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:10:26,128 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['7669 memcached\n'], []) 2022-02-08 01:10:26,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 01:10:26,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:10:26,213 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['7669 memcached\n'], []) 2022-02-08 01:10:26,213 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:10:26,217 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:10:26,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:26,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:10:26,607 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:26,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:10:26,624 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:10:26,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:10:26,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:10:26,673 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:10:26,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:10:26,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:10:26,710 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:10:26,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:10:26,757 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['86137 memcached\n'], []) 2022-02-08 01:10:26,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 01:10:26,780 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:10:26,801 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['86137 memcached\n'], []) 2022-02-08 01:10:26,802 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:10:26,803 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:10:26,882 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:27,119 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:10:27,177 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:27,178 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:10:27,191 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:10:27,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:10:27,216 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:10:27,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:10:27,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:10:27,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:10:27,273 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:10:27,275 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:10:27,349 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['128662 memcached\n'], []) 2022-02-08 01:10:27,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 01:10:27,374 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:10:27,395 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['128662 memcached\n'], []) 2022-02-08 01:10:27,395 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:10:27,398 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:10:27,469 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:27,698 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:10:27,763 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:27,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:10:27,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:10:27,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:10:27,808 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:10:27,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:10:27,841 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:10:27,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:10:27,871 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:10:27,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:10:27,917 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['116590 memcached\n'], []) 2022-02-08 01:10:27,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 01:10:27,941 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:10:27,967 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['116590 memcached\n'], []) 2022-02-08 01:10:27,967 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:10:27,969 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:10:28,059 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:28,283 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:10:28,345 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:28,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:10:28,359 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:10:28,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:10:28,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:10:28,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:10:28,415 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:10:28,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:10:28,444 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:10:28,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:10:28,503 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['24667 memcached\n'], []) 2022-02-08 01:10:28,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 01:10:28,538 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:10:28,568 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['24667 memcached\n'], []) 2022-02-08 01:10:28,568 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:10:28,569 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:10:28,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:28,880 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:10:28,971 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:28,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:10:28,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:10:29,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:10:29,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:10:29,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:10:29,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:10:29,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:10:29,094 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:10:29,094 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:10:29,145 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['18458 memcached\n'], []) 2022-02-08 01:10:29,145 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 01:10:29,173 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:10:29,194 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['18458 memcached\n'], []) 2022-02-08 01:10:29,194 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:10:29,197 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:10:29,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:29,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:10:29,542 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:29,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:10:29,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:10:29,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:10:29,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:10:29,596 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:10:29,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:10:29,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:10:29,644 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:10:29,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:10:29,683 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['59516 memcached\n'], []) 2022-02-08 01:10:29,683 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 01:10:29,709 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:10:29,730 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['59516 memcached\n'], []) 2022-02-08 01:10:29,732 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:10:29,733 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:10:30,015 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:30,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:10:30,303 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:30,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:10:30,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:10:30,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:10:30,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:10:30,420 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:10:30,433 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:10:30,551 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:10:30,569 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:10:30,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:10:30,641 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['53433 memcached\n'], []) 2022-02-08 01:10:30,642 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 01:10:30,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:10:30,688 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['53433 memcached\n'], []) 2022-02-08 01:10:30,690 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:10:30,690 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:10:30,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:31,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:10:31,108 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:31,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:10:31,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:10:31,132 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:10:31,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:10:31,160 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:10:31,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:10:31,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:10:31,197 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:10:31,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:10:31,246 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['117516 memcached\n'], []) 2022-02-08 01:10:31,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 01:10:31,272 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:10:31,293 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['117516 memcached\n'], []) 2022-02-08 01:10:31,293 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:10:31,296 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:10:31,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:31,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:10:31,657 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:31,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:10:31,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:10:31,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:10:31,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:10:31,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:10:31,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:10:31,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:10:31,755 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:10:31,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:10:31,799 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['65635 memcached\n'], []) 2022-02-08 01:10:31,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 01:10:31,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:10:31,845 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['65635 memcached\n'], []) 2022-02-08 01:10:31,846 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:10:31,848 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:10:31,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:32,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:10:32,204 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:32,206 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:10:32,219 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:10:32,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:10:32,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:10:32,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:10:32,283 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:10:32,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:10:32,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:10:32,329 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 01:10:32,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:10:32,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:32,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:10:32,696 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:32,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:32,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:10:32,894 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 01:10:32,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:10:32,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:32,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:10:32,960 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 01:10:32,960 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:10:32,963 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:10:33,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:33,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:10:33,328 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:33,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:10:33,349 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:10:33,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:10:33,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:10:33,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:10:33,405 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:10:33,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:10:33,431 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:10:33,447 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 01:10:33,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:10:33,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:35,193 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:10:35,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:35,502 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:35,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:10:35,621 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 01:10:35,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:10:35,635 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:35,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:10:35,772 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 01:10:35,773 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:10:35,775 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:10:35,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:36,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:10:36,111 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:36,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:10:36,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:10:36,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:10:36,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:10:36,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:10:36,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:10:36,204 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:10:36,219 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:10:36,233 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 01:10:36,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:10:36,295 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:36,471 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:10:36,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:36,762 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:36,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:10:36,875 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 01:10:36,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:10:36,890 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:36,931 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:10:36,944 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 01:10:36,944 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:10:36,947 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 01:10:37,043 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:37,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 01:10:37,322 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:37,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 01:10:37,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 01:10:37,349 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 01:10:37,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 01:10:37,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 01:10:37,395 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 01:10:37,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 01:10:37,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:10:37,437 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 01:10:37,464 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:10:37,492 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:37,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:10:37,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:37,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:37,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:10:37,561 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 01:10:37,561 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:10:37,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:37,589 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:10:37,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:37,632 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:10:37,665 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:37,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:37,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:37,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:37,767 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:37,790 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:37,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:37,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:37,864 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 01:10:37,865 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:10:37,938 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:38,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:10:38,230 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:38,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:10:38,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:10:38,257 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:10:38,270 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:10:38,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:10:38,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:10:38,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:10:38,326 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:10:38,342 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 01:10:38,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:10:38,385 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:38,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:10:38,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:38,811 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:38,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:10:38,857 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 01:10:38,857 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:10:38,869 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:38,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:10:38,913 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 01:10:38,913 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:10:38,914 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 01:10:39,012 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:39,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 01:10:39,296 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:39,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 01:10:39,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 01:10:39,323 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 01:10:39,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 01:10:39,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 01:10:39,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 01:10:39,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 01:10:39,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:10:39,401 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 01:10:39,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:10:39,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:39,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:10:39,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:39,487 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:39,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:10:39,513 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 01:10:39,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:10:39,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:39,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:10:39,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:39,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:10:39,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:39,645 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:39,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:39,693 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:39,714 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:39,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:39,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:39,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:39,808 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 01:10:39,809 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:10:39,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:40,096 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:10:40,151 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:40,153 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:10:40,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:10:40,178 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:10:40,193 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:10:40,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:10:40,221 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:10:40,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:10:40,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:10:40,266 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 01:10:40,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:10:40,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:40,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:10:40,780 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:40,914 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:41,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:10:41,049 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 01:10:41,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:10:41,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:41,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:10:41,107 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 01:10:41,107 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:10:41,108 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:10:41,181 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:41,392 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:10:41,460 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:41,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:10:41,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:10:41,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:10:41,510 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:10:41,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:10:41,543 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:10:41,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:10:41,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:10:41,592 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 01:10:41,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:10:41,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:41,811 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:10:41,969 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:42,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:42,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:10:42,194 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 01:10:42,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:10:42,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:42,252 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:10:42,267 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 01:10:42,267 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:10:42,269 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 01:10:42,372 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:42,594 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 01:10:42,650 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:42,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 01:10:42,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 01:10:42,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 01:10:42,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 01:10:42,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 01:10:42,719 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 01:10:42,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 01:10:42,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:10:42,762 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 01:10:42,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:10:42,803 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:42,821 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:10:42,836 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:42,849 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:42,862 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:10:42,875 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 01:10:42,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:10:42,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:42,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:10:42,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:42,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:10:42,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:42,994 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:43,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:43,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:43,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:43,099 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:43,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:43,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:43,170 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 01:10:43,173 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 01:10:43,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:43,477 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 01:10:43,542 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:43,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 01:10:43,558 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 01:10:43,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 01:10:43,586 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 01:10:43,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 01:10:43,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 01:10:43,642 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 01:10:43,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:10:43,673 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 01:10:43,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:10:43,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:43,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:10:43,749 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:43,760 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:43,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:10:43,786 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 01:10:43,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:10:43,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:43,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:10:43,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:43,854 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:10:43,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:43,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:43,928 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:43,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:43,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:43,994 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:44,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:44,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:44,073 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 01:10:44,076 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:10:44,150 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:44,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:10:44,457 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:44,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:10:44,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:10:44,484 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:10:44,497 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:10:44,515 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:10:44,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:10:44,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:10:44,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:10:44,569 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 01:10:44,594 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:10:44,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:44,831 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:10:45,030 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:45,168 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:45,293 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:10:45,308 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 01:10:45,308 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:10:45,321 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:45,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:10:45,358 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 01:10:45,358 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:10:45,361 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:10:45,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:45,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:10:45,744 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:45,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:10:45,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:10:45,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:10:45,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:10:45,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:10:45,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:10:45,841 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:10:45,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:10:45,871 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 01:10:45,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:10:45,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:46,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:10:46,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:46,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:46,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:10:46,509 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 01:10:46,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:10:46,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:46,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:10:46,566 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 01:10:46,568 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:10:46,569 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:10:46,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:46,915 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:10:47,148 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:47,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:10:47,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:10:47,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:10:47,193 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:10:47,209 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:10:47,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:10:47,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:10:47,247 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:10:47,263 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 01:10:47,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:10:47,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:47,762 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:10:47,858 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:47,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:47,989 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:10:48,003 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 01:10:48,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:10:48,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:48,051 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:10:48,065 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 01:10:48,065 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:10:48,068 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:10:48,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:48,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:10:48,407 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:48,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:10:48,421 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:10:48,438 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:10:48,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:10:48,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:10:48,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:10:48,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:10:48,512 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:10:48,529 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 01:10:48,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:10:48,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 1806 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:10:50,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:10:50,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:50,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:50,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:10:50,526 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 01:10:50,528 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:10:50,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:50,581 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:10:50,594 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 01:10:50,594 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:10:50,595 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 01:10:50,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:50,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 01:10:50,964 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:50,966 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 01:10:50,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 01:10:50,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 01:10:51,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 01:10:51,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 01:10:51,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 01:10:51,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 01:10:51,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:10:51,081 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 01:10:51,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:10:51,119 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:51,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:10:51,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:51,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:10:51,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:10:51,187 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 01:10:51,188 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:10:51,201 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:10:51,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:10:51,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:51,257 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:10:51,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:51,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:51,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:51,359 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:51,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:51,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:51,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:51,466 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:10:51,490 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 01:10:51,496 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:10:51,497 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:10:51,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:10:51,783 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:10:51,842 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:10:51,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:10:51,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:10:51,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:10:51,880 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:10:51,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:10:51,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:10:51,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:10:51,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:10:52,048 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:10:52,052 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:10:52,052 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:10:54,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:10:54,190 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:10:54,190 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:10:56,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:10:56,339 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:10:56,341 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:10:58,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:10:58,486 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:10:58,487 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:00,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:00,631 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:00,632 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:02,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:02,801 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:02,801 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:04,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:04,957 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:04,959 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:06,959 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:07,098 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:07,098 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:09,099 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:09,249 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:09,249 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:11,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:11,390 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:11,390 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:13,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:13,530 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:13,530 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:15,532 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:15,674 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:15,676 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:17,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:17,835 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:17,835 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:19,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:20,421 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:20,421 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:22,423 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:22,671 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:22,673 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:24,673 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:25,009 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:25,009 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:27,010 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:27,582 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:27,582 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:29,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:30,171 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:30,173 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:32,173 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:32,638 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:32,638 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:34,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:34,813 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:34,815 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:36,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:37,095 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:37,095 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:39,096 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:39,325 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:39,325 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:41,326 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:41,523 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:41,523 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:43,525 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:44,030 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:44,030 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:46,030 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:46,677 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:46,677 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:48,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 1808 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:11:49,164 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:49,164 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:51,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:51,482 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:51,483 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:53,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:54,009 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:54,009 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:56,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:56,512 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:56,513 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:11:58,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:11:58,711 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:11:58,711 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:12:00,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:01,148 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:12:01,150 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 69.6540000439. 2022-02-08 01:12:01,150 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:12:01,151 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:12:01,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:12:02,556 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:12:02,730 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:12:02,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:12:02,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:12:02,765 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:12:02,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:12:02,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:12:02,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:12:02,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:12:02,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:03,469 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:12:03,470 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.31999993324. 2022-02-08 01:12:03,470 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:12:03,471 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:12:04,551 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:12:04,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:12:05,220 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:12:05,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:12:05,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:12:05,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:12:05,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:12:05,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:12:05,397 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:12:05,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:12:05,466 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:06,266 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:12:06,267 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.79699993134. 2022-02-08 01:12:06,267 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:12:06,269 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:12:07,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:12:08,084 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:12:08,562 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:12:08,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:12:08,594 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:12:08,619 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:12:08,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:12:08,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:12:08,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:12:08,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:12:08,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:09,410 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:12:09,411 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.14400005341. 2022-02-08 01:12:09,411 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:12:09,414 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:12:09,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:12:10,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:12:10,838 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:12:10,839 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:12:10,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:12:10,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:12:11,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:12:11,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:12:11,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:12:11,262 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:12:11,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:12,142 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:12:12,142 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 01:12:14,142 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:14,834 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:12:14,835 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 01:12:16,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:17,398 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:12:17,400 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 01:12:19,400 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:20,302 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:12:20,302 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 01:12:22,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:23,000 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:12:23,000 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 01:12:25,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:25,434 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:12:25,434 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 01:12:27,436 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:28,365 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:12:28,365 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 01:12:30,367 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:30,911 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:12:30,911 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 01:12:32,913 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:33,378 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:12:33,378 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 01:12:35,380 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:35,816 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:12:35,816 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 01:12:37,818 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:38,342 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:12:38,344 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 01:12:40,344 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:40,670 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:12:40,671 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 31.2590000629. 2022-02-08 01:12:40,671 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:12:40,674 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:12:40,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:12:41,119 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:12:41,391 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:12:41,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:12:41,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:12:41,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:12:41,538 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:12:41,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:12:41,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:12:41,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:12:41,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:42,628 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:12:42,631 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.96000003815. 2022-02-08 01:12:42,631 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:12:42,632 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:12:43,551 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:12:44,079 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:12:44,197 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:12:44,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:12:44,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:12:44,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:12:44,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:12:44,352 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:12:44,456 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:12:44,584 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:12:44,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:45,895 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:12:45,897 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.26600003242. 2022-02-08 01:12:45,897 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:12:45,898 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:12:46,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:12:46,829 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:12:47,023 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:12:47,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:12:47,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:12:47,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:12:47,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:12:47,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:12:47,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:12:47,280 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:12:47,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:48,375 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:12:48,375 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.47899985313. 2022-02-08 01:12:48,377 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:12:48,378 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:12:48,727 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:12:48,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:12:49,051 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:12:49,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:12:49,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 5 queries submitted, 0 failed, 1817 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:12:49,089 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:12:49,102 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:12:49,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:12:49,132 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:12:49,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:12:49,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:49,634 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:12:49,641 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.26499986649. 2022-02-08 01:12:49,642 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:12:49,644 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:12:50,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:12:50,943 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:12:51,387 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:12:51,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:12:51,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:12:51,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:12:51,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:12:51,520 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:12:51,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:12:51,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:12:51,617 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:12:53,871 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:12:53,871 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.22900009155. ############################################################ # # RAM FootPrint: 3895.01953125 # ############################################################ 2022-02-08 01:12:53,904 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 01:12:53,904 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 111 seconds. Reason: Iteration:11 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 1840 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:14:44,931 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 01:14:44,933 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 5 queries submitted, 0 failed, 1861 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:14:49,938 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:14:50,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:14:50,801 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:14:50,901 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:14:50,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:14:50,915 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:14:50,940 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:14:50,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:14:50,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:14:50,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:14:51,005 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:14:51,019 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:14:51,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:14:51,066 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['8815 memcached\n'], []) 2022-02-08 01:14:51,066 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 01:14:51,096 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:14:51,117 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['8815 memcached\n'], []) 2022-02-08 01:14:51,117 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:14:51,118 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:14:51,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:14:51,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:14:51,493 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:14:51,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:14:51,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:14:51,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:14:51,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:14:51,558 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:14:51,571 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:14:51,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:14:51,596 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:14:51,596 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:14:51,651 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['86877 memcached\n'], []) 2022-02-08 01:14:51,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 01:14:51,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:14:51,697 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['86877 memcached\n'], []) 2022-02-08 01:14:51,697 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:14:51,700 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:14:51,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:14:52,030 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:14:52,098 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:14:52,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:14:52,112 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:14:52,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:14:52,140 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:14:52,158 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:14:52,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:14:52,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:14:52,198 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:14:52,198 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:14:52,272 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['129399 memcached\n'], []) 2022-02-08 01:14:52,272 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 01:14:52,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:14:52,329 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['129399 memcached\n'], []) 2022-02-08 01:14:52,331 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:14:52,332 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:14:52,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:14:52,640 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:14:52,704 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:14:52,704 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:14:52,721 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:14:52,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:14:52,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:14:52,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:14:52,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:14:52,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:14:52,829 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:14:52,829 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:14:52,871 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['117472 memcached\n'], []) 2022-02-08 01:14:52,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 01:14:52,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:14:52,918 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['117472 memcached\n'], []) 2022-02-08 01:14:52,920 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:14:52,923 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:14:53,943 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:14:54,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:14:54,493 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:14:54,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:14:54,507 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:14:54,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:14:54,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:14:54,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:14:54,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:14:54,617 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:14:54,628 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:14:54,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:14:54,690 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['25406 memcached\n'], []) 2022-02-08 01:14:54,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 01:14:54,716 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:14:54,740 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['25406 memcached\n'], []) 2022-02-08 01:14:54,740 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:14:54,743 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:14:54,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:14:55,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:14:55,084 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:14:55,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:14:55,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:14:55,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:14:55,127 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:14:55,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:14:55,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:14:55,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:14:55,184 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:14:55,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:14:55,230 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['19251 memcached\n'], []) 2022-02-08 01:14:55,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 01:14:55,255 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:14:55,276 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['19251 memcached\n'], []) 2022-02-08 01:14:55,276 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:14:55,279 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:14:55,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:14:55,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:14:55,615 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:14:55,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:14:55,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:14:55,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:14:55,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:14:55,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:14:55,681 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:14:55,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:14:55,709 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:14:55,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:14:55,806 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['60261 memcached\n'], []) 2022-02-08 01:14:55,806 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 01:14:55,831 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:14:55,851 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['60261 memcached\n'], []) 2022-02-08 01:14:55,851 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:14:55,854 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:14:55,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:14:56,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:14:56,204 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:14:56,206 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:14:56,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:14:56,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:14:56,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:14:56,285 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:14:56,298 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:14:56,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:14:56,325 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:14:56,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:14:56,375 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['54226 memcached\n'], []) 2022-02-08 01:14:56,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 01:14:56,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:14:56,427 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['54226 memcached\n'], []) 2022-02-08 01:14:56,427 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:14:56,428 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:14:56,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:14:56,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:14:56,799 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:14:56,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:14:56,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:14:56,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:14:56,836 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:14:56,852 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:14:56,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:14:56,877 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:14:56,890 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:14:56,890 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:14:56,927 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['118260 memcached\n'], []) 2022-02-08 01:14:56,928 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 01:14:56,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:14:56,980 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['118260 memcached\n'], []) 2022-02-08 01:14:56,980 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:14:56,983 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:14:57,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:14:57,301 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:14:57,364 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:14:57,364 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:14:57,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:14:57,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:14:57,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:14:57,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:14:57,436 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:14:57,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:14:57,463 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:14:57,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:14:57,533 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['66383 memcached\n'], []) 2022-02-08 01:14:57,533 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 01:14:57,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:14:57,581 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['66383 memcached\n'], []) 2022-02-08 01:14:57,581 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:14:57,586 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:14:57,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:14:57,900 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:14:57,976 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:14:57,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:14:57,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:14:58,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:14:58,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:14:58,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:14:58,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:14:58,059 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:14:58,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:14:58,086 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 01:14:58,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:14:58,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:14:58,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:14:58,428 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:14:58,528 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:14:58,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:14:58,631 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 01:14:58,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:14:58,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:14:58,686 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:14:58,700 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 01:14:58,700 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:14:58,703 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:14:58,778 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:14:58,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:14:59,052 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:14:59,053 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:14:59,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:14:59,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:14:59,089 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:14:59,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:14:59,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:14:59,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:14:59,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:14:59,160 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 01:14:59,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:14:59,201 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:14:59,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:14:59,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:00,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:00,224 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:15:00,237 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 01:15:00,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:15:00,252 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:00,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:15:00,325 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 01:15:00,325 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:15:00,328 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:15:00,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:15:00,594 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:15:00,644 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:15:00,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:15:00,655 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:15:00,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:15:00,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:15:00,691 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:15:00,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:15:00,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:15:00,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:15:00,746 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 01:15:00,775 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:15:00,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:00,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:15:01,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:01,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:01,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:15:01,394 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 01:15:01,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:15:01,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:01,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:15:01,496 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 01:15:01,496 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:15:01,500 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 01:15:01,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:15:01,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 01:15:01,862 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:15:01,864 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 01:15:01,877 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 01:15:01,890 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 01:15:01,903 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 01:15:01,918 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 01:15:01,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 01:15:01,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 01:15:01,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:15:01,971 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 01:15:01,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:15:02,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:02,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:15:02,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:02,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:02,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:15:02,075 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 01:15:02,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:15:02,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:02,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:15:02,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:02,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:15:02,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:02,186 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:02,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:02,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:02,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:02,280 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:02,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:02,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:02,351 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 01:15:02,354 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:15:02,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:15:02,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:15:02,717 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:15:02,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:15:02,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:15:02,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:15:02,757 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:15:02,775 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:15:02,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:15:02,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:15:02,811 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:15:02,825 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 01:15:02,849 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:15:02,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:02,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:15:02,987 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:03,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:03,061 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:15:03,075 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 01:15:03,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:15:03,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:03,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:15:03,137 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 01:15:03,137 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:15:03,138 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 01:15:03,224 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:15:03,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 01:15:03,493 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:15:03,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 01:15:03,507 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 01:15:03,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 01:15:03,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 01:15:03,545 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 01:15:03,556 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 01:15:03,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 01:15:03,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:15:03,592 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 01:15:03,617 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:15:03,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:03,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:15:03,655 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:03,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:03,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:15:03,691 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 01:15:03,691 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:15:03,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:03,716 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:15:03,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:03,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:15:03,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:03,798 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:03,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:03,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:03,872 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:03,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:03,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:03,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:03,977 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 01:15:03,980 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:15:04,073 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:15:04,272 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:15:04,332 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:15:04,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:15:04,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:15:04,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:15:04,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:15:04,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:15:04,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:15:04,421 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:15:04,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:15:04,451 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 01:15:04,477 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:15:04,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:04,693 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:15:04,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:05,028 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:05,145 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:15:05,160 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 01:15:05,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:15:05,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:05,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:15:05,221 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 01:15:05,223 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:15:05,224 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:15:05,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:15:05,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:15:05,582 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:15:05,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:15:05,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:15:05,612 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:15:05,627 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:15:05,650 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:15:05,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:15:05,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:15:05,714 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:15:05,732 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 01:15:05,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:15:05,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:06,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:15:06,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:07,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:07,201 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:15:07,217 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 01:15:07,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:15:07,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:07,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:15:07,293 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 01:15:07,293 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:15:07,298 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 01:15:07,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:15:07,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 01:15:07,653 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:15:07,653 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 01:15:07,665 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 01:15:07,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 01:15:07,693 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 01:15:07,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 01:15:07,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 01:15:07,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 01:15:07,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:15:07,759 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 01:15:07,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:15:07,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:07,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:15:07,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:07,839 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:07,852 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:15:07,865 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 01:15:07,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:15:07,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:07,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:15:07,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:07,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:15:07,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:07,982 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:08,007 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:08,030 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:08,053 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:08,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:08,102 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:08,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:08,151 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 01:15:08,153 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 01:15:08,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:15:08,456 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 01:15:08,515 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:15:08,515 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 01:15:08,528 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 01:15:08,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 01:15:08,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 01:15:08,566 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 01:15:08,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 01:15:08,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 01:15:08,602 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:15:08,618 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 01:15:08,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:15:08,655 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:08,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:15:08,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:08,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:08,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:15:08,724 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 01:15:08,724 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:15:08,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:08,762 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:15:08,775 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:08,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:15:08,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:08,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:08,882 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:08,908 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:08,936 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:08,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:08,992 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:09,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:09,046 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 01:15:09,049 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:15:09,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:15:09,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:15:09,410 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:15:09,410 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:15:09,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:15:09,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:15:09,448 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:15:09,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:15:09,482 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:15:09,499 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:15:09,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:15:09,538 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 01:15:09,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:15:09,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:10,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:15:10,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:10,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:11,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:15:11,078 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 01:15:11,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:15:11,091 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:11,127 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:15:11,140 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 01:15:11,140 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:15:11,141 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:15:11,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:15:11,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:15:11,500 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:15:11,502 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:15:11,515 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:15:11,528 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:15:11,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:15:11,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:15:11,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:15:11,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:15:11,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:15:11,608 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 01:15:11,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:15:11,645 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:13,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:15:13,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:13,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:13,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:15:13,611 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 01:15:13,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:15:13,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:13,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:15:13,687 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 01:15:13,687 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:15:13,690 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:15:13,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:15:14,005 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:15:14,075 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:15:14,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:15:14,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:15:14,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:15:14,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:15:14,130 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:15:14,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:15:14,158 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:15:14,188 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:15:14,210 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 01:15:14,247 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:15:14,263 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:14,538 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:15:14,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:14,732 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:14,808 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:15:14,825 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 01:15:14,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:15:14,854 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:14,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:15:14,890 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 01:15:14,891 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:15:14,892 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:15:14,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:15:15,178 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:15:15,234 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:15:15,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:15:15,247 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:15:15,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:15:15,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:15:15,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:15:15,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:15:15,313 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:15:15,326 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:15:15,364 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 01:15:15,392 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:15:15,405 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:15,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:15:15,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:15,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:16,015 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:15:16,029 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 01:15:16,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:15:16,040 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:16,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:15:16,088 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 01:15:16,088 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:15:16,095 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 01:15:16,188 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:15:16,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 01:15:16,497 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:15:16,497 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 01:15:16,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 01:15:16,532 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 01:15:16,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 01:15:16,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 01:15:16,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 01:15:16,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 01:15:16,614 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:15:16,631 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 01:15:16,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:15:16,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:16,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:15:16,704 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:16,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:15:16,732 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:15:16,746 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 01:15:16,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:15:16,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:15:16,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:15:16,788 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:16,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:15:16,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:16,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:16,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:16,926 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:16,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:16,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:17,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:17,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:15:17,068 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 01:15:17,069 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:15:17,072 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:15:17,150 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:15:17,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:15:17,427 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:15:17,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:15:17,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:15:17,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:15:17,464 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:15:17,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:15:17,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:15:17,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:15:17,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:17,647 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:15:17,651 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:17,653 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:19,653 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:19,767 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:15:19,773 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:19,773 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:21,775 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:21,887 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:15:21,892 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:21,892 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:23,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:24,033 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:24,035 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:26,035 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:26,174 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:26,174 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:28,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:28,315 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:28,315 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:30,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:30,453 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:30,453 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:32,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:32,615 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:32,615 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:34,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:34,769 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:34,769 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:36,770 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:36,921 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:36,921 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:38,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:39,061 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:39,061 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:41,061 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:41,210 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:41,210 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:43,211 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:43,375 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:43,375 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:45,377 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:45,526 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:45,526 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:47,528 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:47,680 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:47,680 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 1875 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:15:49,681 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:49,826 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:49,826 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:51,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:51,973 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:51,973 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:53,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:54,128 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:54,128 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:56,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:56,282 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:56,282 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:15:58,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:15:58,490 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:15:58,492 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:16:00,492 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:00,644 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:16:00,644 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:16:02,645 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:02,838 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:16:02,838 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:16:04,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:05,282 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:16:05,282 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:16:07,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:07,677 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:16:07,677 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:16:09,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:10,104 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:16:10,105 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:16:12,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:12,326 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:16:12,328 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:16:14,331 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:14,612 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:16:14,614 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:16:16,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:16,934 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:16:16,934 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:16:18,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:19,305 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:16:19,306 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:16:21,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:21,502 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:16:21,502 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:16:23,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:24,279 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:16:24,280 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:16:26,280 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:27,076 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:16:27,076 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:16:29,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:29,834 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:16:29,834 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:16:31,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:32,309 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:16:32,309 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:16:34,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:34,582 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:16:34,582 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:16:36,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:36,835 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:16:36,836 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:16:38,836 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:39,072 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:16:39,072 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 82.0039999485. 2022-02-08 01:16:39,072 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:16:39,075 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:16:39,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:16:39,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:16:40,042 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:16:40,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:16:40,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:16:40,155 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:16:40,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:16:40,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:16:40,275 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:16:40,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:16:40,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:40,868 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:16:40,868 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.79500007629. 2022-02-08 01:16:40,868 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:16:40,869 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:16:41,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:16:42,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:16:42,456 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:16:42,456 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:16:42,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:16:42,551 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:16:42,614 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:16:42,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:16:42,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:16:42,719 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:16:42,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:43,647 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:16:43,648 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.77999997139. 2022-02-08 01:16:43,648 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:16:43,650 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:16:45,063 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:16:45,982 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:16:47,216 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:16:47,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:16:47,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:16:47,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:16:47,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:16:47,892 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:16:48,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:16:48,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:16:48,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 1881 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:16:49,878 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:16:49,878 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 01:16:51,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:52,311 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:16:52,311 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 8.66300010681. 2022-02-08 01:16:52,311 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:16:52,312 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:16:53,415 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:16:53,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:16:53,961 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:16:53,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:16:53,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:16:54,035 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:16:54,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:16:54,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:16:54,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:16:54,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:16:54,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:16:54,596 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:16:54,598 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.28699994087. 2022-02-08 01:16:54,598 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:16:54,599 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:16:55,747 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:16:56,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:16:56,592 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:16:56,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:16:56,642 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:16:56,686 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:16:56,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:16:56,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:16:56,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:16:57,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:16:57,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:17:00,346 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:17:00,348 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.75. 2022-02-08 01:17:00,348 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:17:00,349 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:17:01,170 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:17:01,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:17:01,868 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:17:01,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:17:01,890 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:17:01,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:17:01,946 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:17:01,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:17:01,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:17:02,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:17:02,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:17:03,244 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:17:03,246 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.89699983597. 2022-02-08 01:17:03,246 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:17:03,247 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:17:04,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:17:05,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:17:05,901 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:17:05,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:17:05,946 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:17:05,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:17:06,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:17:06,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:17:06,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:17:06,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:17:06,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:17:08,684 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:17:08,686 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.43899989128. 2022-02-08 01:17:08,686 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:17:08,687 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:17:09,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:17:09,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:17:10,119 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:17:10,119 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:17:10,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:17:10,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:17:10,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:17:10,191 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:17:10,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:17:10,232 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:17:10,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:17:10,755 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:17:10,756 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.06999993324. 2022-02-08 01:17:10,756 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:17:10,757 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:17:11,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:17:12,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:17:12,259 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:17:12,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:17:12,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:17:12,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:17:12,563 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:17:12,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:17:12,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:17:12,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:17:12,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:17:15,447 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:17:15,454 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.69899988174. ############################################################ # # RAM FootPrint: 3928.6015625 # ############################################################ 2022-02-08 01:17:15,670 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 01:17:15,670 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 61 seconds. Reason: Iteration:12 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 1904 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:18:16,674 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 01:18:16,674 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 01:18:21,677 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:18:22,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:22,403 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:18:22,486 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:22,487 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:18:22,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:18:22,520 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:18:22,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:18:22,556 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:18:22,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:18:22,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:18:22,657 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:18:22,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:18:22,756 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['9925 memcached\n'], []) 2022-02-08 01:18:22,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 01:18:22,806 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:18:22,838 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['9925 memcached\n'], []) 2022-02-08 01:18:22,839 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:18:22,842 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:18:22,941 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:23,160 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:18:23,217 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:23,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:18:23,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:18:23,242 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:18:23,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:18:23,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:18:23,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:18:23,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:18:23,312 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:18:23,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:18:23,371 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['87673 memcached\n'], []) 2022-02-08 01:18:23,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 01:18:23,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:18:23,415 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['87673 memcached\n'], []) 2022-02-08 01:18:23,415 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:18:23,417 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:18:23,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:23,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:18:23,819 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:23,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:18:23,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:18:23,844 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:18:23,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:18:23,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:18:23,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:18:23,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:18:23,914 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:18:23,914 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:18:23,957 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['130150 memcached\n'], []) 2022-02-08 01:18:23,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 01:18:23,984 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:18:24,007 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['130150 memcached\n'], []) 2022-02-08 01:18:24,007 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:18:24,012 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:18:24,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:24,339 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:18:24,410 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:24,410 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:18:24,428 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:18:24,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:18:24,459 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:18:24,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:18:24,499 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:18:24,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:18:24,536 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:18:24,538 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:18:24,581 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['118316 memcached\n'], []) 2022-02-08 01:18:24,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 01:18:24,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:18:24,631 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['118316 memcached\n'], []) 2022-02-08 01:18:24,631 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:18:24,634 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:18:24,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:24,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:18:25,006 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:25,007 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:18:25,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:18:25,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:18:25,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:18:25,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:18:25,076 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:18:25,094 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:18:25,108 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:18:25,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:18:25,155 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['26316 memcached\n'], []) 2022-02-08 01:18:25,155 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 01:18:25,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:18:25,201 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['26316 memcached\n'], []) 2022-02-08 01:18:25,203 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:18:25,203 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:18:25,285 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:25,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:18:25,556 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:25,556 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:18:25,571 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:18:25,584 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:18:25,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:18:25,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:18:25,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:18:25,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:18:25,650 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:18:25,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:18:25,697 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['20002 memcached\n'], []) 2022-02-08 01:18:25,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 01:18:25,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:18:25,746 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['20002 memcached\n'], []) 2022-02-08 01:18:25,746 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:18:25,749 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:18:25,846 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:26,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:18:26,111 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:26,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:18:26,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:18:26,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:18:26,150 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:18:26,165 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:18:26,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:18:26,188 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:18:26,201 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:18:26,201 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:18:26,243 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['61041 memcached\n'], []) 2022-02-08 01:18:26,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 01:18:26,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:18:26,296 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['61041 memcached\n'], []) 2022-02-08 01:18:26,296 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:18:26,299 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:18:26,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:26,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:18:26,697 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:26,698 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:18:26,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:18:26,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:18:26,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:18:26,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:18:26,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:18:26,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:18:26,802 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:18:26,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:18:26,859 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['54979 memcached\n'], []) 2022-02-08 01:18:26,859 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 01:18:26,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:18:26,907 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['54979 memcached\n'], []) 2022-02-08 01:18:26,907 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:18:26,910 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:18:26,992 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:27,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:18:27,265 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:27,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:18:27,280 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:18:27,293 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:18:27,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:18:27,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:18:27,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:18:27,369 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:18:27,388 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:18:27,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:18:27,459 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['119055 memcached\n'], []) 2022-02-08 01:18:27,459 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 01:18:27,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:18:27,509 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['119055 memcached\n'], []) 2022-02-08 01:18:27,509 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:18:27,512 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:18:27,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:27,849 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:18:27,908 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:27,908 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:18:27,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:18:27,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:18:27,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:18:27,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:18:27,989 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:18:28,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:18:28,036 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:18:28,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:18:28,102 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['67131 memcached\n'], []) 2022-02-08 01:18:28,102 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 01:18:28,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:18:28,151 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['67131 memcached\n'], []) 2022-02-08 01:18:28,151 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:18:28,155 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:18:28,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:28,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:18:28,536 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:28,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:18:28,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:18:28,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:18:28,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:18:28,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:18:28,609 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:18:28,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:18:28,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:18:28,674 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 01:18:28,724 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:18:28,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:29,528 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:18:29,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:29,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:29,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:18:29,882 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 01:18:29,882 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:18:29,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:29,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:18:29,960 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 01:18:29,960 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:18:29,963 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:18:30,035 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:30,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:18:30,358 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:30,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:18:30,372 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:18:30,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:18:30,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:18:30,428 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:18:30,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:18:30,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:18:30,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:18:30,484 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 01:18:30,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:18:30,525 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:30,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:18:30,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:31,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:31,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:18:31,163 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 01:18:31,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:18:31,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:31,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:18:31,234 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 01:18:31,234 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:18:31,236 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:18:31,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:31,533 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:18:31,594 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:31,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:18:31,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:18:31,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:18:31,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:18:31,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:18:31,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:18:31,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:18:31,688 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:18:31,703 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 01:18:31,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:18:31,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:31,938 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:18:32,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:32,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:32,352 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:18:32,368 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 01:18:32,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:18:32,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:32,438 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:18:32,454 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 01:18:32,456 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:18:32,463 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 01:18:32,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:32,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 01:18:32,832 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:32,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 01:18:32,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 01:18:32,858 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 01:18:32,869 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 01:18:32,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 01:18:32,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 01:18:32,908 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 01:18:32,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:18:32,936 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 01:18:32,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:18:32,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:32,994 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:18:33,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:33,020 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:33,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:18:33,046 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 01:18:33,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:18:33,059 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:33,073 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:18:33,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:33,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:18:33,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:33,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:33,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:33,221 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:33,252 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:33,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:33,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:33,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:33,364 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 01:18:33,365 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:18:33,446 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:33,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:18:33,747 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:33,747 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:18:33,765 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:18:33,778 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:18:33,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:18:33,806 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:18:33,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:18:33,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:18:33,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:18:33,858 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 01:18:33,882 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:18:33,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:34,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:18:34,331 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:34,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:34,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:18:34,431 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 01:18:34,433 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:18:34,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:34,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:18:34,486 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 01:18:34,487 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:18:34,490 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 01:18:34,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:34,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 01:18:34,884 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:34,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 01:18:34,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 01:18:34,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 01:18:34,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 01:18:34,938 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 01:18:34,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 01:18:34,961 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 01:18:34,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:18:34,989 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 01:18:35,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:18:35,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:35,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:18:35,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:35,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:35,076 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:18:35,088 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 01:18:35,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:18:35,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:35,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:18:35,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:35,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:18:35,176 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:35,198 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:35,224 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:35,247 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:35,272 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:35,303 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:35,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:35,357 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:35,382 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 01:18:35,384 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:18:35,477 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:35,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:18:35,740 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:35,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:18:35,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:18:35,770 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:18:35,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:18:35,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:18:35,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:18:35,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:18:35,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:18:35,865 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 01:18:35,892 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:18:35,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:36,887 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:18:37,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:37,224 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:37,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:18:37,358 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 01:18:37,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:18:37,372 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:37,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:18:37,426 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 01:18:37,427 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:18:37,430 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:18:37,520 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:37,739 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:18:37,802 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:37,803 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:18:37,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:18:37,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:18:37,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:18:37,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:18:37,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:18:37,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:18:37,914 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:18:37,931 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 01:18:37,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:18:37,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:38,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:18:38,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:38,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:38,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:18:38,584 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 01:18:38,584 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:18:38,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:38,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:18:38,654 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 01:18:38,654 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:18:38,657 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 01:18:38,747 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:38,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 01:18:39,063 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:39,063 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 01:18:39,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 01:18:39,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 01:18:39,104 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 01:18:39,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 01:18:39,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 01:18:39,145 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 01:18:39,158 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:18:39,174 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 01:18:39,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:18:39,216 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:39,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:18:39,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:39,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:39,275 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:18:39,289 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 01:18:39,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:18:39,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:39,323 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:18:39,336 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:39,367 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:18:39,395 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:39,423 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:39,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:39,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:39,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:39,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:39,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:39,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:39,634 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 01:18:39,638 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 01:18:39,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:39,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 01:18:40,020 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:40,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 01:18:40,035 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 01:18:40,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 01:18:40,061 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 01:18:40,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 01:18:40,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 01:18:40,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 01:18:40,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:18:40,128 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 01:18:40,155 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:18:40,170 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:40,183 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:18:40,196 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:40,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:40,221 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:18:40,233 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 01:18:40,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:18:40,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:40,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:18:40,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:40,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:18:40,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:40,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:40,380 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:40,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:40,428 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:40,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:40,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:40,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:40,540 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 01:18:40,543 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:18:40,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:40,846 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:18:40,914 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:40,915 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:18:40,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:18:40,943 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:18:40,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:18:40,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:18:40,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:18:41,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:18:41,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:18:41,040 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 01:18:41,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:18:41,079 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:41,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:18:41,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:41,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:41,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:18:41,980 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 01:18:41,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:18:41,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:42,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:18:42,039 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 01:18:42,039 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:18:42,042 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:18:42,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:42,334 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:18:42,398 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:42,400 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:18:42,415 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:18:42,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:18:42,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:18:42,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:18:42,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:18:42,492 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:18:42,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:18:42,523 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 01:18:42,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:18:42,563 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:42,757 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:18:42,936 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:43,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:43,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:18:43,203 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 01:18:43,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:18:43,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:43,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:18:43,259 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 01:18:43,259 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:18:43,262 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:18:43,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:43,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:18:43,792 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:43,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:18:43,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:18:43,859 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:18:43,903 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:18:43,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:18:43,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:18:43,989 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:18:44,007 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:18:44,086 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 01:18:44,112 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:18:44,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:44,423 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:18:44,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:44,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:44,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:18:44,730 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 01:18:44,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:18:44,747 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:44,801 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:18:44,813 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 01:18:44,813 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:18:44,815 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:18:44,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:45,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:18:45,153 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:45,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:18:45,165 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:18:45,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:18:45,188 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:18:45,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:18:45,216 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:18:45,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:18:45,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:18:45,253 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 01:18:45,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:18:45,295 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:46,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:18:47,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:47,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:47,346 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:18:47,365 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 01:18:47,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:18:47,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:47,413 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:18:47,424 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 01:18:47,426 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:18:47,427 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 01:18:47,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:47,732 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 01:18:47,796 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:47,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 01:18:47,811 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 01:18:47,823 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 01:18:47,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 01:18:47,854 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 01:18:47,867 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 01:18:47,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 01:18:47,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:18:47,910 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 01:18:47,936 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:18:47,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:47,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:18:47,979 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:47,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:18:48,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:18:48,023 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 01:18:48,025 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:18:48,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:18:48,061 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:18:48,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:48,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:18:48,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:48,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:48,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:48,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:48,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:48,265 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:48,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:48,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:18:48,351 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 01:18:48,351 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:18:48,354 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:18:48,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:18:48,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:18:48,733 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:18:48,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:18:48,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:18:48,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:18:48,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:18:48,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:18:48,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:18:48,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:18:48,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:18:49,022 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:18:49,028 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:18:49,028 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 1932 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:18:51,028 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:18:51,141 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:18:51,148 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:18:51,148 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:18:53,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:18:53,269 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:18:53,276 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:18:53,276 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:18:55,278 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:18:55,427 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:18:55,427 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:18:57,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:18:57,575 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:18:57,576 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:18:59,576 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:18:59,724 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:18:59,726 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:01,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:01,867 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:01,868 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:03,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:04,015 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:04,015 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:06,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:06,174 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:06,174 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:08,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:08,328 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:08,328 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:10,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:10,483 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:10,483 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:12,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:12,782 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:12,782 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:14,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:14,940 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:14,940 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:16,940 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:17,101 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:17,101 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:19,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:19,631 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:19,631 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:21,632 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:22,019 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:22,019 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:24,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:24,342 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:24,342 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:26,344 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:26,720 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:26,720 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:28,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:29,069 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:29,069 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:31,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:31,232 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:31,233 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:33,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:33,555 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:33,555 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:35,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:35,757 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:35,757 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:37,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:37,931 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:37,933 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:39,933 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:40,323 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:40,323 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:42,323 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:42,609 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:42,609 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:44,609 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:45,124 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:45,125 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:47,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:47,793 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:47,793 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 1943 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:19:49,795 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:50,200 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:50,201 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:52,201 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:53,125 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:53,125 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:55,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:55,628 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:55,630 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:19:57,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:19:58,006 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:19:58,006 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:20:00,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:20:00,305 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:20:00,305 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:20:02,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:20:02,855 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:20:02,855 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 74.5039999485. 2022-02-08 01:20:02,857 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:20:02,858 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:20:03,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:20:03,982 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:20:04,114 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:20:04,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:20:04,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:20:04,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:20:04,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:20:04,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:20:04,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:20:04,219 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:20:04,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:20:04,674 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:20:04,676 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.81799983978. 2022-02-08 01:20:04,676 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:20:04,677 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:20:05,385 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:20:05,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:20:06,035 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:20:06,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:20:06,056 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:20:06,081 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:20:06,130 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:20:06,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:20:06,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:20:06,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:20:06,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:20:07,273 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:20:07,273 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.59799981117. 2022-02-08 01:20:07,275 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:20:07,278 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:20:07,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:20:07,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:20:08,197 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:20:08,198 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:20:08,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:20:08,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:20:08,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:20:08,311 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:20:08,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:20:08,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:20:08,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:20:08,881 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:20:08,882 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.60699987411. 2022-02-08 01:20:08,882 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:20:08,884 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:20:09,701 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:20:10,099 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:20:10,279 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:20:10,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:20:10,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:20:10,367 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:20:10,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:20:10,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:20:10,431 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:20:10,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:20:10,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:20:11,155 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:20:11,157 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.27300000191. 2022-02-08 01:20:11,157 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:20:11,158 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:20:12,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:20:12,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:20:12,816 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:20:12,816 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:20:12,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:20:12,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:20:12,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:20:12,905 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:20:12,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:20:12,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:20:12,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:20:13,710 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:20:13,710 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.5529999733. 2022-02-08 01:20:13,710 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:20:13,711 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:20:14,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:20:14,966 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:20:15,108 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:20:15,109 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:20:15,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:20:15,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:20:15,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:20:15,198 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:20:15,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:20:15,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:20:15,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:20:15,908 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:20:15,910 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.19900012016. 2022-02-08 01:20:15,910 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:20:15,911 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:20:16,538 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:20:16,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:20:17,346 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:20:17,346 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:20:17,403 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:20:17,479 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:20:17,637 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:20:17,869 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:20:18,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:20:18,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:20:18,191 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:20:19,408 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:20:19,408 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 01:20:21,410 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:20:21,615 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:20:21,615 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 01:20:23,617 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:20:24,000 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:20:24,002 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 01:20:26,002 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:20:26,196 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:20:26,197 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 01:20:28,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:20:28,375 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:20:28,377 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 01:20:30,377 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:20:30,542 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:20:30,542 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 14.6319999695. 2022-02-08 01:20:30,542 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:20:30,543 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:20:30,941 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:20:31,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:20:31,298 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:20:31,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:20:31,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:20:31,334 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:20:31,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:20:31,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:20:31,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:20:31,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:20:31,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:20:32,003 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:20:32,005 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.46199989319. 2022-02-08 01:20:32,005 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:20:32,006 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:20:33,030 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:20:33,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:20:33,805 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:20:33,806 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:20:33,831 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:20:33,857 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:20:33,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:20:33,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:20:33,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:20:33,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:20:34,007 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:20:35,128 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:20:35,128 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.12299990654. ############################################################ # # RAM FootPrint: 3965.234375 # ############################################################ 2022-02-08 01:20:35,322 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 01:20:35,322 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 94 seconds. Reason: Iteration:13 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 1956 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 1979 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:22:09,390 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 01:22:09,390 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 01:22:14,418 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:22:15,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:16,168 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:22:16,430 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:16,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:22:16,466 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:22:16,492 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:22:16,512 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:22:16,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:22:16,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:22:16,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:22:16,703 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:22:16,704 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:22:16,900 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['11057 memcached\n'], []) 2022-02-08 01:22:16,900 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 01:22:16,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:22:17,009 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['11057 memcached\n'], []) 2022-02-08 01:22:17,009 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:22:17,012 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:22:17,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:18,206 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:22:18,487 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:18,487 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:22:18,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:22:18,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:22:18,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:22:18,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:22:18,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:22:18,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:22:18,624 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:22:18,624 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:22:18,683 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['88328 memcached\n'], []) 2022-02-08 01:22:18,683 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 01:22:18,721 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:22:18,746 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['88328 memcached\n'], []) 2022-02-08 01:22:18,746 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:22:18,749 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:22:19,642 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:19,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:22:20,147 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:20,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:22:20,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:22:20,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:22:20,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:22:20,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:22:20,280 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:22:20,326 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:22:20,352 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:22:20,352 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:22:20,426 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['130848 memcached\n'], []) 2022-02-08 01:22:20,426 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 01:22:20,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:22:20,519 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['130848 memcached\n'], []) 2022-02-08 01:22:20,519 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:22:20,522 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:22:21,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:21,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:22:22,012 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:22,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:22:22,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:22:22,073 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:22:22,112 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:22:22,137 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:22:22,153 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:22:22,209 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:22:22,229 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:22:22,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:22:22,335 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['119122 memcached\n'], []) 2022-02-08 01:22:22,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 01:22:22,369 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:22:22,401 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['119122 memcached\n'], []) 2022-02-08 01:22:22,401 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:22:22,403 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:22:23,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:24,313 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:22:24,549 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:24,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:22:24,571 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:22:24,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:22:24,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:22:24,665 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:22:24,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:22:24,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:22:24,737 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:22:24,737 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:22:24,805 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['26986 memcached\n'], []) 2022-02-08 01:22:24,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 01:22:24,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:22:24,865 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['26986 memcached\n'], []) 2022-02-08 01:22:24,865 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:22:24,867 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:22:25,846 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:26,525 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:22:26,759 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:26,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:22:26,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:22:26,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:22:26,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:22:26,844 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:22:26,858 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:22:26,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:22:26,908 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:22:26,908 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:22:27,012 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['20656 memcached\n'], []) 2022-02-08 01:22:27,012 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 01:22:27,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:22:27,065 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['20656 memcached\n'], []) 2022-02-08 01:22:27,065 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:22:27,069 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:22:28,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:29,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:22:29,378 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:29,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:22:29,392 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:22:29,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:22:29,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:22:29,456 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:22:29,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:22:29,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:22:29,607 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:22:29,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:22:29,690 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['61708 memcached\n'], []) 2022-02-08 01:22:29,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 01:22:29,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:22:29,740 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['61708 memcached\n'], []) 2022-02-08 01:22:29,740 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:22:29,743 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:22:31,193 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:31,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:22:31,996 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:31,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:22:32,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:22:32,030 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:22:32,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:22:32,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:22:32,076 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:22:32,107 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:22:32,125 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:22:32,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:22:32,190 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['55676 memcached\n'], []) 2022-02-08 01:22:32,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 01:22:32,242 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:22:32,263 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['55676 memcached\n'], []) 2022-02-08 01:22:32,263 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:22:32,266 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:22:33,153 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:33,790 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:22:34,023 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:34,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:22:34,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:22:34,073 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:22:34,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:22:34,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:22:34,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:22:34,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:22:34,173 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:22:34,173 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:22:34,226 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['119712 memcached\n'], []) 2022-02-08 01:22:34,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 01:22:34,252 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:22:34,282 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['119712 memcached\n'], []) 2022-02-08 01:22:34,282 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:22:34,285 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:22:34,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:35,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:22:35,349 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:35,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:22:35,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:22:35,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:22:35,405 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:22:35,469 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:22:35,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:22:35,558 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:22:35,579 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:22:35,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:22:35,687 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['67830 memcached\n'], []) 2022-02-08 01:22:35,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 01:22:35,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:22:35,733 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['67830 memcached\n'], []) 2022-02-08 01:22:35,733 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:22:35,736 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:22:35,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:36,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:22:36,098 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:36,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:22:36,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:22:36,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:22:36,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:22:36,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:22:36,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:22:36,191 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:22:36,206 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:22:36,220 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 01:22:36,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:22:36,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:38,107 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:22:38,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:38,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:38,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:22:38,500 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 01:22:38,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:22:38,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:38,556 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:22:38,569 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 01:22:38,569 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:22:38,572 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:22:38,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:38,887 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:22:38,947 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:38,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:22:38,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:22:38,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:22:38,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:22:39,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:22:39,015 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:22:39,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:22:39,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:22:39,059 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 01:22:39,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:22:39,099 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:41,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:22:41,397 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:41,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:41,647 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:22:41,661 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 01:22:41,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:22:41,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:41,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:22:41,740 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 01:22:41,740 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:22:41,743 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:22:41,839 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:42,038 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:22:42,099 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:42,099 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:22:42,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:22:42,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:22:42,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:22:42,153 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:22:42,165 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:22:42,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:22:42,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:22:42,204 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 01:22:42,229 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:22:42,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:44,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:22:44,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:44,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:44,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:22:44,509 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 01:22:44,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:22:44,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:44,563 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:22:44,576 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 01:22:44,576 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:22:44,578 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 01:22:44,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:44,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 01:22:44,953 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:44,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 01:22:44,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 01:22:44,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 01:22:44,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 01:22:45,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 01:22:45,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 01:22:45,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 01:22:45,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:22:45,065 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 01:22:45,091 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:22:45,107 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:45,122 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:22:45,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:45,150 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:45,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:22:45,176 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 01:22:45,176 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:22:45,188 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:45,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:22:45,216 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:45,249 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:22:45,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:45,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:45,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:45,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:45,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:45,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:45,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:45,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:45,517 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 01:22:45,519 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:22:45,589 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:45,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:22:45,872 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:45,872 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:22:45,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:22:45,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:22:45,915 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:22:45,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:22:45,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:22:45,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:22:45,974 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:22:45,990 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 01:22:46,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:22:46,028 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:46,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:22:46,538 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:46,581 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:46,619 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:22:46,632 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 01:22:46,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:22:46,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:46,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:22:46,700 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 01:22:46,701 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:22:46,704 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 01:22:46,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:47,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 01:22:47,066 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:47,068 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 01:22:47,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 01:22:47,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 01:22:47,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 01:22:47,130 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 01:22:47,142 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 01:22:47,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 01:22:47,168 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:22:47,183 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 01:22:47,211 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:22:47,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:47,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:22:47,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:47,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:47,278 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:22:47,292 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 01:22:47,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:22:47,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:47,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:22:47,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:47,372 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:22:47,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:47,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:47,466 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:47,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:47,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:47,561 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:47,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:47,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:47,651 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 01:22:47,653 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:22:47,732 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:47,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:22:48,000 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:48,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:22:48,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:22:48,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:22:48,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:22:48,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:22:48,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:22:48,076 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:22:48,089 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:22:48,104 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 01:22:48,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:22:48,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 2001 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:22:49,867 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:22:50,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:50,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:50,339 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:22:50,355 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 01:22:50,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:22:50,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:50,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:22:50,414 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 01:22:50,414 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:22:50,417 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:22:50,492 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:50,721 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:22:50,786 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:50,788 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:22:50,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:22:50,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:22:50,829 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:22:50,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:22:50,864 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:22:50,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:22:50,892 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:22:50,908 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 01:22:50,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:22:50,948 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:52,576 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:22:52,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:52,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:52,977 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:22:52,994 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 01:22:52,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:22:53,010 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:53,053 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:22:53,069 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 01:22:53,069 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:22:53,072 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 01:22:53,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:53,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 01:22:53,440 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:53,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 01:22:53,456 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 01:22:53,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 01:22:53,482 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 01:22:53,497 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 01:22:53,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 01:22:53,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 01:22:53,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:22:53,549 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 01:22:53,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:22:53,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:53,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:22:53,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:53,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:53,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:22:53,678 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 01:22:53,678 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:22:53,693 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:53,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:22:53,721 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:53,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:22:53,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:53,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:53,852 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:53,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:53,915 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:53,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:53,982 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:54,015 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:54,042 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 01:22:54,045 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 01:22:54,122 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:54,359 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 01:22:54,437 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:54,438 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 01:22:54,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 01:22:54,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 01:22:54,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 01:22:54,499 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 01:22:54,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 01:22:54,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 01:22:54,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:22:54,563 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 01:22:54,599 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:22:54,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:54,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:22:54,647 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:54,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:54,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:22:54,688 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 01:22:54,688 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:22:54,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:54,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:22:54,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:54,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:22:54,803 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:54,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:54,867 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:54,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:54,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:54,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:54,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:55,030 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:22:55,059 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 01:22:55,061 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:22:55,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:55,352 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:22:55,413 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:55,413 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:22:55,426 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:22:55,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:22:55,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:22:55,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:22:55,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:22:55,497 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:22:55,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:22:55,528 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 01:22:55,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:22:55,566 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:57,706 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:22:57,928 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:58,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:22:58,209 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:22:58,221 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 01:22:58,221 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:22:58,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:22:58,263 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:22:58,276 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 01:22:58,276 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:22:58,278 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:22:58,352 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:22:58,588 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:22:58,651 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:22:58,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:22:58,668 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:22:58,683 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:22:58,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:22:58,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:22:58,734 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:22:58,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:22:58,765 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:22:58,782 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 01:22:58,806 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:22:58,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:23:00,624 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:23:00,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:23:00,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:23:01,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:23:01,092 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 01:23:01,094 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:23:01,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:23:01,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:23:01,153 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 01:23:01,153 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:23:01,155 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:23:01,984 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:23:02,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:23:02,516 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:23:02,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:23:02,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:23:02,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:23:02,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:23:02,749 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:23:02,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:23:02,858 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:23:02,940 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:23:02,983 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 01:23:03,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:23:03,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:23:04,091 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:23:04,272 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:23:04,469 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:23:04,635 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:23:04,720 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 01:23:04,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:23:04,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:23:04,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:23:04,808 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 01:23:04,809 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:23:04,811 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:23:04,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:23:05,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:23:05,187 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:23:05,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:23:05,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:23:05,213 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:23:05,227 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:23:05,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:23:05,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:23:05,272 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:23:05,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:23:05,302 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 01:23:05,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:23:05,344 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:23:07,056 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:23:07,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:23:07,400 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:23:07,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:23:07,533 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 01:23:07,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:23:07,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:23:07,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:23:07,608 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 01:23:07,608 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:23:07,609 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 01:23:07,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:23:07,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 01:23:08,009 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:23:08,010 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 01:23:08,025 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 01:23:08,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 01:23:08,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 01:23:08,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 01:23:08,086 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 01:23:08,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 01:23:08,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:23:08,134 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 01:23:08,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:23:08,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:23:08,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:23:08,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:23:08,221 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:23:08,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:23:08,250 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 01:23:08,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:23:08,265 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:23:08,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:23:08,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:23:08,334 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:23:08,369 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:23:08,403 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:23:08,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:23:08,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:23:08,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:23:08,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:23:08,573 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:23:08,609 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:23:08,644 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 01:23:08,644 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:23:08,647 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:23:08,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:23:08,940 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:23:09,039 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:23:09,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:23:09,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:23:09,068 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:23:09,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:23:09,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:23:09,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:23:09,127 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:23:09,142 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:09,601 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:09,602 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:11,602 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:11,973 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:11,974 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:13,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:14,148 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:14,148 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:16,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:16,447 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:16,447 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:18,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:18,625 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:18,627 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:20,627 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:20,937 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:20,937 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:22,946 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:23,269 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:23,270 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:25,270 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:25,496 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:25,496 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:27,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:27,732 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:27,732 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:29,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:30,121 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:30,121 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:32,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:32,398 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:32,398 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:34,400 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:34,957 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:34,957 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:36,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:37,546 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:37,546 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:39,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:40,049 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:40,049 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:42,051 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:42,463 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:42,463 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:44,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:44,832 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:44,832 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:46,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:47,158 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:47,158 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 2026 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:23:49,160 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:49,414 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:49,414 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:51,415 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:51,963 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:51,963 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:53,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:54,615 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:54,615 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:56,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:57,012 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:57,013 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:23:59,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:23:59,270 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:23:59,270 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:24:01,270 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:24:01,614 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:24:01,614 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:24:03,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:24:03,980 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:24:03,980 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:24:05,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:24:06,362 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:24:06,365 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:24:08,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:24:08,640 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:24:08,641 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 59.9960000515. 2022-02-08 01:24:08,641 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:24:08,671 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:24:08,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:24:09,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:24:09,089 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:24:09,099 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:24:09,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:24:09,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:24:09,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:24:09,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:24:09,173 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:24:09,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:24:09,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:24:09,778 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:24:09,778 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.13700008392. 2022-02-08 01:24:09,779 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:24:09,780 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:24:10,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:24:11,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:24:11,875 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:24:11,877 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:24:11,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:24:11,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:24:12,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:24:12,247 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:24:12,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:24:12,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:24:12,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:24:13,927 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:24:13,927 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 01:24:15,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:24:16,243 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:24:16,243 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 01:24:18,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:24:18,493 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:24:18,493 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 8.71599984169. 2022-02-08 01:24:18,494 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:24:18,496 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:24:19,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:24:19,788 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:24:19,969 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:24:19,969 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:24:20,015 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:24:20,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:24:20,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:24:20,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:24:20,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:24:20,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:24:20,211 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:24:21,009 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:24:21,012 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.51800012589. 2022-02-08 01:24:21,012 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:24:21,013 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:24:22,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:24:22,589 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:24:22,773 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:24:22,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:24:22,793 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:24:22,823 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:24:22,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:24:22,895 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:24:22,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:24:22,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:24:22,977 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:24:23,693 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:24:23,696 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.68299984932. 2022-02-08 01:24:23,696 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:24:23,697 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:24:24,762 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:24:25,096 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:24:25,233 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:24:25,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:24:25,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:24:25,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:24:25,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:24:25,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:24:25,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:24:25,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:24:25,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:24:26,197 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:24:26,198 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.50299978256. 2022-02-08 01:24:26,198 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:24:26,200 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:24:27,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:24:27,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:24:27,473 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:24:27,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:24:27,497 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:24:27,525 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:24:27,573 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:24:27,601 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:24:27,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:24:27,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:24:27,691 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:24:28,476 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:24:28,477 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.27900004387. 2022-02-08 01:24:28,477 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:24:28,480 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:24:29,397 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:24:30,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:24:30,168 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:24:30,168 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:24:30,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:24:30,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:24:30,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:24:30,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:24:30,311 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:24:30,334 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:24:30,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:24:31,147 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:24:31,148 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.66900014877. 2022-02-08 01:24:31,148 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:24:31,150 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:24:31,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:24:32,059 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:24:32,151 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:24:32,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:24:32,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:24:32,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:24:32,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:24:32,224 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:24:32,247 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:24:32,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:24:32,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:24:32,928 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:24:32,930 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.78200006485. 2022-02-08 01:24:32,930 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:24:32,931 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:24:33,854 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:24:34,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:24:34,788 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:24:34,788 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:24:34,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:24:34,890 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:24:34,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:24:34,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:24:34,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:24:34,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:24:35,020 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:24:36,157 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:24:36,157 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.22699999809. ############################################################ # # RAM FootPrint: 3982.78125 # ############################################################ 2022-02-08 01:24:36,484 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 01:24:36,486 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 120 seconds. Reason: Iteration:14 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 2051 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 2076 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:26:36,490 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 01:26:36,490 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 01:26:41,493 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:26:42,561 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:26:43,374 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:26:43,786 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:26:43,788 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:26:43,895 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:26:43,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:26:44,028 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:26:44,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:26:44,272 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:26:44,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:26:44,496 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:26:44,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:26:44,730 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['12162 memcached\n'], []) 2022-02-08 01:26:44,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 01:26:44,920 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:26:44,956 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['12162 memcached\n'], []) 2022-02-08 01:26:44,956 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:26:44,957 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:26:45,086 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:26:45,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:26:45,384 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:26:45,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:26:45,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:26:45,413 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:26:45,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:26:45,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:26:45,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:26:45,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:26:45,484 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:26:45,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:26:45,546 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['89053 memcached\n'], []) 2022-02-08 01:26:45,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 01:26:45,571 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:26:45,592 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['89053 memcached\n'], []) 2022-02-08 01:26:45,592 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:26:45,594 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:26:47,252 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:26:47,877 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:26:48,204 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:26:48,206 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:26:48,224 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:26:48,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:26:48,283 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:26:48,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:26:48,323 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:26:48,359 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:26:48,377 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:26:48,377 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:26:48,440 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['862 memcached\n'], []) 2022-02-08 01:26:48,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 01:26:48,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:26:48,523 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['862 memcached\n'], []) 2022-02-08 01:26:48,523 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:26:48,525 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 5 queries submitted, 0 failed, 2101 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:26:49,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:26:50,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:26:50,269 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:26:50,270 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:26:50,290 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:26:50,349 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:26:50,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:26:50,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:26:50,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:26:50,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:26:50,473 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:26:50,474 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:26:50,540 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['119916 memcached\n'], []) 2022-02-08 01:26:50,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 01:26:50,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:26:50,609 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['119916 memcached\n'], []) 2022-02-08 01:26:50,611 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:26:50,614 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:26:51,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:26:52,344 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:26:52,651 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:26:52,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:26:52,670 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:26:52,698 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:26:52,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:26:52,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:26:52,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:26:52,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:26:52,801 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:26:52,801 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:26:52,868 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['27715 memcached\n'], []) 2022-02-08 01:26:52,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 01:26:52,903 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:26:52,926 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['27715 memcached\n'], []) 2022-02-08 01:26:52,926 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:26:52,927 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:26:53,545 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:26:54,183 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:26:54,668 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:26:54,670 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:26:54,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:26:55,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:26:55,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:26:55,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:26:55,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:26:55,408 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:26:55,443 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:26:55,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:26:55,687 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['21403 memcached\n'], []) 2022-02-08 01:26:55,688 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 01:26:55,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:26:55,888 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['21403 memcached\n'], []) 2022-02-08 01:26:55,888 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:26:55,890 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:26:56,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:26:57,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:26:57,461 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:26:57,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:26:57,487 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:26:57,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:26:57,543 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:26:57,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:26:57,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:26:57,816 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:26:57,937 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:26:57,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:26:58,138 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['62445 memcached\n'], []) 2022-02-08 01:26:58,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 01:26:58,213 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:26:58,240 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['62445 memcached\n'], []) 2022-02-08 01:26:58,240 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:26:58,243 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:26:59,484 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:00,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:27:00,424 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:00,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:27:00,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:27:00,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:27:00,464 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:27:00,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:27:00,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:27:00,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:27:00,540 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:27:00,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:27:00,605 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['56467 memcached\n'], []) 2022-02-08 01:27:00,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 01:27:00,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:27:00,651 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['56467 memcached\n'], []) 2022-02-08 01:27:00,651 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:27:00,654 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:27:01,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:01,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:27:01,703 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:01,704 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:27:01,719 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:27:01,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:27:01,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:27:01,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:27:01,783 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:27:01,811 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:27:01,826 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:27:01,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:27:01,887 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['120451 memcached\n'], []) 2022-02-08 01:27:01,887 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 01:27:01,915 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:27:01,940 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['120451 memcached\n'], []) 2022-02-08 01:27:01,940 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:27:01,941 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:27:02,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:03,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:27:03,206 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:03,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:27:03,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:27:03,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:27:03,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:27:03,275 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:27:03,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:27:03,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:27:03,325 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:27:03,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:27:03,391 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['68570 memcached\n'], []) 2022-02-08 01:27:03,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 01:27:03,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:27:03,447 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['68570 memcached\n'], []) 2022-02-08 01:27:03,447 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:27:03,448 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:27:03,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:03,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:27:03,857 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:03,857 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:27:03,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:27:03,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:27:03,895 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:27:03,913 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:27:03,926 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:27:03,940 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:27:03,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:27:03,970 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 01:27:03,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:27:04,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:05,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:27:05,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:06,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:06,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:27:06,114 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 01:27:06,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:27:06,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:06,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:27:06,180 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 01:27:06,181 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:27:06,184 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:27:06,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:06,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:27:06,565 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:06,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:27:06,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:27:06,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:27:06,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:27:06,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:27:06,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:27:06,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:27:06,665 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:27:06,690 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 01:27:06,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:27:06,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:08,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:27:08,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:08,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:09,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:27:09,096 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 01:27:09,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:27:09,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:09,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:27:09,276 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 01:27:09,276 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:27:09,278 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:27:09,359 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:09,558 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:27:09,611 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:09,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:27:09,624 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:27:09,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:27:09,647 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:27:09,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:27:09,673 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:27:09,686 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:27:09,698 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:27:09,713 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 01:27:09,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:27:09,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:11,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:27:11,737 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:11,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:11,989 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:27:12,003 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 01:27:12,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:27:12,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:12,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:27:12,076 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 01:27:12,076 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:27:12,078 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 01:27:12,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:12,367 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 01:27:12,423 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:12,423 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 01:27:12,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 01:27:12,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 01:27:12,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 01:27:12,479 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 01:27:12,492 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 01:27:12,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 01:27:12,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:27:12,533 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 01:27:12,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:27:12,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:12,586 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:27:12,599 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:12,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:12,624 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:27:12,637 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 01:27:12,637 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:27:12,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:12,663 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:27:12,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:12,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:27:12,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:12,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:12,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:12,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:12,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:12,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:12,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:12,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:13,006 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 01:27:13,009 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:27:13,099 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:13,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:27:13,382 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:13,382 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:27:13,397 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:27:13,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:27:13,423 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:27:13,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:27:13,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:27:13,466 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:27:13,479 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:27:13,493 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 01:27:13,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:27:13,532 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:14,012 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:27:14,091 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:14,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:14,183 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:27:14,203 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 01:27:14,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:27:14,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:14,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:27:14,269 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 01:27:14,269 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:27:14,272 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 01:27:14,352 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:14,586 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 01:27:14,645 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:14,647 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 01:27:14,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 01:27:14,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 01:27:14,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 01:27:14,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 01:27:14,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 01:27:14,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 01:27:14,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:27:14,762 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 01:27:14,790 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:27:14,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:14,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:27:14,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:14,864 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:14,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:27:14,888 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 01:27:14,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:27:14,903 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:14,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:27:14,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:14,966 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:27:15,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:15,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:15,073 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:15,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:15,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:15,173 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:15,206 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:15,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:15,269 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 01:27:15,272 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:27:15,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:15,566 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:27:15,625 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:15,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:27:15,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:27:15,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:27:15,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:27:15,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:27:15,698 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:27:15,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:27:15,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:27:15,742 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 01:27:15,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:27:15,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:17,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:27:18,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:18,173 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:18,293 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:27:18,308 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 01:27:18,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:27:18,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:18,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:27:18,369 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 01:27:18,369 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:27:18,371 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:27:18,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:18,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:27:18,737 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:18,739 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:27:18,755 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:27:18,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:27:18,783 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:27:18,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:27:18,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:27:18,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:27:18,849 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:27:18,867 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 01:27:18,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:27:18,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:20,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:27:20,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:20,895 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:20,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:27:21,019 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 01:27:21,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:27:21,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:21,079 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:27:21,095 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 01:27:21,096 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:27:21,099 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 01:27:21,196 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:21,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 01:27:21,512 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:21,512 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 01:27:21,528 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 01:27:21,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 01:27:21,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 01:27:21,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 01:27:21,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 01:27:21,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 01:27:21,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:27:21,634 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 01:27:21,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:27:21,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:21,688 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:27:21,701 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:21,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:21,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:27:21,739 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 01:27:21,739 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:27:21,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:21,765 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:27:21,778 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:21,811 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:27:21,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:21,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:21,913 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:21,946 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:21,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:22,012 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:22,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:22,081 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:22,111 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 01:27:22,114 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 01:27:22,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:22,414 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 01:27:22,470 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:22,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 01:27:22,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 01:27:22,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 01:27:22,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 01:27:22,525 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 01:27:22,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 01:27:22,551 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 01:27:22,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:27:22,579 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 01:27:22,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:27:22,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:22,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:27:22,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:22,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:22,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:27:22,690 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 01:27:22,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:27:22,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:22,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:27:22,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:22,762 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:27:22,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:22,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:22,862 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:22,895 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:22,931 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:22,971 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:23,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:23,059 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:23,092 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 01:27:23,095 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:27:23,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:23,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:27:23,459 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:23,459 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:27:23,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:27:23,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:27:23,502 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:27:23,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:27:23,532 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:27:23,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:27:23,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:27:23,575 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 01:27:23,599 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:27:23,612 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:25,640 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:27:25,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:26,020 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:26,153 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:27:26,171 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 01:27:26,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:27:26,186 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:26,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:27:26,229 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 01:27:26,229 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:27:26,230 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:27:26,313 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:26,533 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:27:26,591 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:26,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:27:26,604 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:27:26,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:27:26,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:27:26,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:27:26,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:27:26,670 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:27:26,683 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:27:26,697 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 01:27:26,721 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:27:26,734 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:28,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:27:28,931 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:29,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:29,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:27:29,186 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 01:27:29,186 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:27:29,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:29,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:27:29,252 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 01:27:29,252 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:27:29,253 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:27:29,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:30,263 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:27:30,401 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:30,403 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:27:30,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:27:30,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:27:30,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:27:30,482 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:27:30,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:27:30,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:27:30,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:27:30,551 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 01:27:30,576 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:27:30,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:31,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:27:31,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:31,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:31,989 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:27:32,003 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 01:27:32,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:27:32,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:32,045 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:27:32,059 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 01:27:32,061 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:27:32,062 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:27:32,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:32,374 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:27:32,470 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:32,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:27:32,487 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:27:32,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:27:32,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:27:32,538 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:27:32,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:27:32,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:27:32,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:27:32,598 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 01:27:32,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:27:32,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:34,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:27:34,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:34,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:34,920 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:27:34,936 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 01:27:34,936 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:27:34,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:34,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:27:35,000 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 01:27:35,000 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:27:35,002 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 01:27:35,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:35,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 01:27:35,369 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:35,369 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 01:27:35,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 01:27:35,397 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 01:27:35,410 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 01:27:35,426 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 01:27:35,438 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 01:27:35,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 01:27:35,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:27:35,480 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 01:27:35,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:27:35,520 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:35,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:27:35,556 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:35,571 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:27:35,599 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:27:35,617 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 01:27:35,617 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:27:35,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:27:35,645 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:27:35,660 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:35,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:27:35,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:35,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:35,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:35,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:35,874 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:35,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:35,943 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:35,979 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:27:36,010 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 01:27:36,010 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:27:36,013 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:27:36,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:27:36,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:27:36,417 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:27:36,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:27:36,436 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:27:36,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:27:36,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:27:36,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:27:36,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:27:36,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:27:36,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:27:36,802 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:27:36,802 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:27:38,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:27:38,967 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:27:38,979 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:27:40,979 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:27:41,131 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:27:41,132 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:27:43,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:27:43,278 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:27:43,279 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:27:45,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:27:45,552 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:27:45,552 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:27:47,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:27:47,717 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:27:47,717 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 2124 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:27:49,719 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:27:49,914 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:27:49,914 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:27:51,915 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:27:52,444 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:27:52,444 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:27:54,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:27:54,992 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:27:54,992 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:27:56,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:27:57,187 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:27:57,187 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:27:59,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:27:59,489 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:27:59,489 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:01,489 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:01,713 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:28:01,713 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:03,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:04,227 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:28:04,229 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:06,229 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:06,611 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:28:06,611 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:08,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:08,801 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:28:08,801 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:10,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:11,065 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:28:11,065 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:13,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:13,447 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:28:13,447 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:15,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:16,036 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:28:16,036 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:18,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:18,476 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:28:18,477 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:20,477 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:21,085 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:28:21,086 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:23,086 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:23,500 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:28:23,500 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:25,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:25,809 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:28:25,809 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:27,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:28,302 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:28:28,302 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:30,303 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:30,779 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:28:30,779 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:32,780 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:33,171 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:28:33,171 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:35,181 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:35,650 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:28:35,650 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:37,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:38,125 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:28:38,125 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:40,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:40,489 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:28:40,489 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:28:42,489 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:42,698 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:28:42,700 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 66.6890001297. 2022-02-08 01:28:42,700 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:28:42,701 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:28:42,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:28:43,096 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:28:43,173 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:28:43,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:28:43,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:28:43,214 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:28:43,227 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:28:43,247 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:28:43,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:28:43,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:28:43,372 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:43,882 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:28:43,884 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.18299984932. 2022-02-08 01:28:43,884 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:28:43,885 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:28:44,969 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:28:45,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:28:45,927 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:28:45,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:28:45,974 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:28:46,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:28:46,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:28:46,318 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:28:46,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:28:46,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:28:46,482 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:47,273 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:28:47,275 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.39100003242. 2022-02-08 01:28:47,275 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:28:47,276 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:28:47,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:28:47,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:28:47,914 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:28:47,915 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:28:47,941 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:28:47,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:28:47,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:28:48,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:28:48,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:28:48,071 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:28:48,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:48,740 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:28:48,740 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.46499991417. 2022-02-08 01:28:48,740 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:28:48,743 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:28:48,846 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 2148 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:28:49,150 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:28:49,262 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:28:49,262 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:28:49,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:28:49,308 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:28:49,331 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:28:49,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:28:49,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:28:49,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:28:49,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:50,111 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:28:50,111 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.3710000515. 2022-02-08 01:28:50,111 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:28:50,114 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:28:50,224 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:28:50,545 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:28:50,642 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:28:50,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:28:50,665 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:28:50,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:28:50,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:28:50,727 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:28:50,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:28:50,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:28:50,811 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:51,584 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:28:51,585 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.47200012207. 2022-02-08 01:28:51,585 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:28:51,585 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:28:51,701 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:28:51,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:28:52,088 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:28:52,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:28:52,107 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:28:52,122 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:28:52,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:28:52,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:28:52,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:28:52,206 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:28:52,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:52,911 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:28:52,911 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.3259999752. 2022-02-08 01:28:52,911 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:28:52,913 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:28:53,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:28:53,326 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:28:53,427 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:28:53,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:28:53,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:28:53,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:28:53,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:28:53,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:28:53,586 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:28:53,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:28:53,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:54,279 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:28:54,280 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.36999988556. 2022-02-08 01:28:54,280 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:28:54,283 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:28:54,943 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:28:55,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:28:55,500 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:28:55,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:28:55,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:28:55,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:28:55,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:28:55,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:28:55,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:28:55,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:28:55,627 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:28:56,200 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:28:56,206 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.92500019073. 2022-02-08 01:28:56,207 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:28:56,207 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:28:57,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:28:57,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:28:58,193 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:28:58,193 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:28:58,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:28:58,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:28:58,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:28:58,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:28:58,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:28:58,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:28:58,984 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:29:00,161 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:29:00,161 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:29:02,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:29:02,671 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:29:02,673 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.46600008011. ############################################################ # # RAM FootPrint: 4001.87109375 # ############################################################ 2022-02-08 01:29:03,124 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 01:29:03,124 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 106 seconds. Reason: Iteration:15 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 2173 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 2196 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:30:49,128 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 01:30:49,128 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 01:30:54,131 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:30:55,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:30:55,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:30:56,168 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:30:56,170 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:30:56,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:30:56,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:30:56,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:30:56,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:30:56,482 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:30:56,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:30:56,566 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:30:56,566 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:30:56,686 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['13210 memcached\n'], []) 2022-02-08 01:30:56,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 01:30:56,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:30:56,812 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['13210 memcached\n'], []) 2022-02-08 01:30:56,812 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:30:56,815 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:30:57,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:30:57,727 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:30:57,898 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:30:57,900 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:30:57,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:30:57,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:30:57,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:30:57,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:30:58,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:30:58,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:30:58,052 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:30:58,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:30:58,117 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['89852 memcached\n'], []) 2022-02-08 01:30:58,117 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 01:30:58,142 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:30:58,167 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['89852 memcached\n'], []) 2022-02-08 01:30:58,168 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:30:58,170 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:30:59,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:30:59,602 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:30:59,931 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:30:59,931 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:30:59,951 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:31:00,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:31:00,038 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:31:00,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:31:00,086 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:31:00,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:31:00,177 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:31:00,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:31:00,236 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['1651 memcached\n'], []) 2022-02-08 01:31:00,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 01:31:00,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:31:00,368 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['1651 memcached\n'], []) 2022-02-08 01:31:00,368 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:31:00,369 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:31:01,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:01,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:31:01,641 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:01,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:31:01,660 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:31:01,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:31:01,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:31:01,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:31:01,765 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:31:01,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:31:01,819 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:31:01,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:31:01,898 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['120792 memcached\n'], []) 2022-02-08 01:31:01,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 01:31:01,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:31:01,969 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['120792 memcached\n'], []) 2022-02-08 01:31:01,969 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:31:01,970 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:31:02,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:03,357 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:31:03,523 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:03,525 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:31:03,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:31:03,573 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:31:03,614 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:31:03,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:31:03,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:31:03,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:31:03,723 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:31:03,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:31:03,808 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['28496 memcached\n'], []) 2022-02-08 01:31:03,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 01:31:03,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:31:03,924 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['28496 memcached\n'], []) 2022-02-08 01:31:03,924 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:31:03,927 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:31:05,316 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:05,941 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:31:06,319 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:06,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:31:06,354 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:31:06,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:31:06,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:31:06,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:31:06,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:31:06,686 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:31:06,730 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:31:06,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:31:06,864 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['22176 memcached\n'], []) 2022-02-08 01:31:06,864 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 01:31:06,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:31:06,983 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['22176 memcached\n'], []) 2022-02-08 01:31:06,983 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:31:06,986 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:31:07,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:08,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:31:08,546 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:08,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:31:08,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:31:08,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:31:08,637 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:31:08,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:31:08,678 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:31:08,704 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:31:08,723 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:31:08,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:31:08,811 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['63224 memcached\n'], []) 2022-02-08 01:31:08,811 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 01:31:08,839 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:31:08,862 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['63224 memcached\n'], []) 2022-02-08 01:31:08,864 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:31:08,867 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:31:09,642 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:10,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:31:10,595 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:10,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:31:10,665 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:31:10,711 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:31:10,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:31:10,864 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:31:10,877 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:31:10,905 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:31:10,921 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:31:10,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:31:11,019 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['57214 memcached\n'], []) 2022-02-08 01:31:11,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 01:31:11,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:31:11,253 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['57214 memcached\n'], []) 2022-02-08 01:31:11,253 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:31:11,256 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:31:12,102 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:12,464 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:31:12,645 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:12,645 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:31:12,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:31:12,688 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:31:12,701 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:31:12,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:31:12,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:31:12,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:31:12,780 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:31:12,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:31:12,845 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['121247 memcached\n'], []) 2022-02-08 01:31:12,846 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 01:31:12,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:31:12,898 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['121247 memcached\n'], []) 2022-02-08 01:31:12,898 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:31:12,901 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:31:13,158 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:13,448 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:31:13,608 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:13,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:31:13,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:31:13,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:31:13,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:31:13,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:31:13,691 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:31:13,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:31:13,759 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:31:13,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:31:13,806 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['69316 memcached\n'], []) 2022-02-08 01:31:13,808 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 01:31:13,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:31:13,854 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['69316 memcached\n'], []) 2022-02-08 01:31:13,854 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:31:13,855 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:31:13,918 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:14,137 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:31:14,196 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:14,196 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:31:14,209 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:31:14,221 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:31:14,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:31:14,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:31:14,265 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:31:14,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:31:14,293 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:31:14,309 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 01:31:14,344 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:31:14,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:15,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:31:15,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:16,015 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:16,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:31:16,127 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 01:31:16,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:31:16,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:16,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:31:16,191 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 01:31:16,191 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:31:16,194 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:31:16,295 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:16,499 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:31:16,558 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:16,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:31:16,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:31:16,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:31:16,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:31:16,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:31:16,627 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:31:16,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:31:16,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:31:16,690 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 01:31:16,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:31:16,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:19,188 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:31:19,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:19,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:19,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:31:19,657 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 01:31:19,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:31:19,670 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:19,719 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:31:19,733 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 01:31:19,733 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:31:19,734 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:31:19,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:20,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:31:20,062 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:20,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:31:20,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:31:20,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:31:20,099 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:31:20,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:31:20,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:31:20,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:31:20,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:31:20,177 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 01:31:20,201 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:31:20,214 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:23,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:31:23,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:23,683 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:23,793 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:31:23,809 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 01:31:23,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:31:23,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:23,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:31:23,888 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 01:31:23,888 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:31:23,891 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 01:31:23,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:24,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 01:31:24,260 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:24,262 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 01:31:24,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 01:31:24,293 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 01:31:24,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 01:31:24,323 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 01:31:24,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 01:31:24,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 01:31:24,382 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:31:24,398 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 01:31:24,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:31:24,443 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:24,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:31:24,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:24,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:24,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:31:24,515 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 01:31:24,515 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:31:24,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:24,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:31:24,556 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:24,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:31:24,632 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:24,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:24,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:24,734 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:24,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:24,808 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:24,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:24,882 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:24,914 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 01:31:24,917 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:31:24,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:25,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:31:25,289 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:25,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:31:25,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:31:25,336 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:31:25,349 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:31:25,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:31:25,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:31:25,413 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:31:25,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:31:25,453 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 01:31:25,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:31:25,512 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:27,265 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:31:27,349 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:27,397 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:27,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:31:27,469 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 01:31:27,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:31:27,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:27,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:31:27,568 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 01:31:27,568 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:31:27,572 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 01:31:27,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:27,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 01:31:27,943 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:27,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 01:31:27,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 01:31:27,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 01:31:27,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 01:31:27,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 01:31:28,007 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 01:31:28,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 01:31:28,033 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:31:28,048 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 01:31:28,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:31:28,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:28,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:31:28,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:28,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:28,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:31:28,171 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 01:31:28,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:31:28,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:28,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:31:28,214 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:28,249 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:31:28,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:28,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:28,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:28,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:28,414 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:28,448 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:28,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:28,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:28,552 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 01:31:28,555 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:31:28,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:28,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:31:28,892 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:28,892 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:31:28,905 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:31:28,918 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:31:28,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:31:28,946 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:31:28,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:31:28,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:31:28,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:31:29,000 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 01:31:29,025 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:31:29,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:32,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:31:32,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:32,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:32,668 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:31:32,683 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 01:31:32,683 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:31:32,696 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:32,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:31:32,815 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 01:31:32,815 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:31:32,818 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:31:32,892 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:33,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:31:33,198 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:33,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:31:33,214 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:31:33,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:31:33,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:31:33,267 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:31:33,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:31:33,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:31:33,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:31:33,335 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 01:31:33,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:31:33,380 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:35,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:31:36,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:36,227 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:36,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:31:36,384 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 01:31:36,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:31:36,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:36,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:31:36,558 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 01:31:36,558 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:31:36,561 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 01:31:36,637 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:36,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 01:31:36,913 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:36,914 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 01:31:36,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 01:31:36,940 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 01:31:36,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 01:31:36,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 01:31:36,982 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 01:31:36,994 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 01:31:37,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:31:37,023 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 01:31:37,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:31:37,063 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:37,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:31:37,094 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:37,107 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:37,119 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:31:37,132 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 01:31:37,132 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:31:37,147 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:37,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:31:37,176 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:37,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:31:37,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:37,275 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:37,311 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:37,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:37,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:37,408 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:37,443 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:37,477 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:37,507 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 01:31:37,509 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 01:31:37,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:37,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 01:31:37,881 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:37,882 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 01:31:37,895 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 01:31:37,908 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 01:31:37,923 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 01:31:37,938 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 01:31:37,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 01:31:37,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 01:31:37,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:31:37,992 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 01:31:38,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:31:38,030 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:38,045 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:31:38,058 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:38,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:38,084 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:31:38,095 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 01:31:38,096 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:31:38,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:38,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:31:38,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:38,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:31:38,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:38,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:38,276 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:38,308 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:38,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:38,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:38,426 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:38,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:38,499 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 01:31:38,500 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:31:38,596 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:38,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:31:38,878 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:38,880 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:31:38,892 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:31:38,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:31:38,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:31:38,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:31:38,951 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:31:38,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:31:38,984 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:31:39,000 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 01:31:39,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:31:39,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:41,933 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:31:42,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:42,357 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:42,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:31:42,510 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 01:31:42,510 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:31:42,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:42,561 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:31:42,575 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 01:31:42,575 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:31:42,578 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:31:42,668 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:42,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:31:42,970 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:42,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:31:42,984 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:31:43,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:31:43,020 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:31:43,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:31:43,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:31:43,066 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:31:43,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:31:43,098 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 01:31:43,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:31:43,135 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:45,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:31:45,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:45,556 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:45,678 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:31:45,694 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 01:31:45,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:31:45,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:45,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:31:45,828 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 01:31:45,828 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:31:45,831 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:31:46,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:47,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:31:47,421 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:47,421 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:31:47,436 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:31:47,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:31:47,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:31:47,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:31:47,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:31:47,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:31:47,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:31:47,555 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 01:31:47,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:31:47,594 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:48,940 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:31:49,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 2221 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:31:49,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:49,216 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:31:49,233 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 01:31:49,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:31:49,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:49,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:31:49,295 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 01:31:49,296 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:31:49,298 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:31:49,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:49,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:31:49,714 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:49,716 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:31:49,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:31:49,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:31:49,762 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:31:49,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:31:49,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:31:49,806 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:31:49,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:31:49,838 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 01:31:49,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:31:49,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:51,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:31:51,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:51,969 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:52,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:31:52,132 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 01:31:52,132 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:31:52,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:52,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:31:52,209 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 01:31:52,210 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:31:52,211 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 01:31:52,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:52,517 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 01:31:52,585 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:52,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 01:31:52,601 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 01:31:52,614 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 01:31:52,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 01:31:52,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 01:31:52,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 01:31:52,670 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 01:31:52,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:31:52,698 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 01:31:52,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:31:52,737 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:52,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:31:52,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:52,783 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:31:52,798 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:31:52,812 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 01:31:52,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:31:52,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:31:52,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:31:52,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:52,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:31:52,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:52,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:53,007 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:53,040 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:53,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:53,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:53,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:53,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:31:53,250 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 01:31:53,250 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:31:53,253 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:31:53,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:31:53,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:31:53,637 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:31:53,637 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:31:53,683 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:31:53,724 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:31:53,767 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:31:53,857 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:31:53,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:31:53,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:31:54,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:31:54,460 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:31:54,460 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:31:56,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:31:56,858 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:31:56,858 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:31:58,858 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:31:59,092 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:31:59,094 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:32:01,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:01,437 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:01,437 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:32:03,438 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:03,668 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:03,668 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:32:05,670 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:06,219 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:06,219 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:32:08,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:08,552 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:08,552 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:32:10,553 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:10,986 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:10,986 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:32:12,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:13,461 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:13,463 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:32:15,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:15,848 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:15,848 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:32:17,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:18,098 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:18,098 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:32:20,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:20,440 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:20,440 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:32:22,441 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:22,842 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:22,842 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:32:24,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:25,091 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:25,091 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:32:27,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:27,403 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:27,403 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:32:29,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:29,746 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:29,746 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:32:31,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:32,153 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:32,153 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:32:34,153 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:34,424 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:34,424 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:32:36,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:36,799 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:36,799 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:32:38,801 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:38,999 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:32:38,999 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 45.7490000725. 2022-02-08 01:32:38,999 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:32:39,000 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:32:39,104 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:32:39,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:32:39,424 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:32:39,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:32:39,438 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:32:39,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:32:39,464 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:32:39,487 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:32:39,505 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:32:39,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:32:39,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:40,023 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:32:40,023 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.02399992943. 2022-02-08 01:32:40,023 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:32:40,026 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:32:40,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:32:40,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:32:41,072 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:32:41,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:32:41,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:32:41,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:32:41,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:32:41,433 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:32:41,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:32:41,545 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:32:41,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:42,818 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:42,818 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 01:32:44,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:45,365 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:45,368 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 01:32:47,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:47,609 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:32:47,611 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.58800005913. 2022-02-08 01:32:47,611 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:32:47,612 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:32:48,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 2242 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:32:49,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:32:49,677 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:32:49,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:32:49,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:32:49,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:32:50,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:32:50,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:32:50,227 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:32:50,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:32:50,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:52,055 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:52,056 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 01:32:54,056 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:54,505 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:32:54,505 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 01:32:56,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:56,720 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:32:56,721 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 9.11099982262. 2022-02-08 01:32:56,721 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:32:56,723 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:32:57,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:32:58,104 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:32:58,316 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:32:58,318 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:32:58,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:32:58,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:32:58,410 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:32:58,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:32:58,456 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:32:58,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:32:58,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:32:59,585 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:32:59,585 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.8630001545. 2022-02-08 01:32:59,585 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:32:59,586 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:33:00,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:33:01,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:33:01,194 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:33:01,196 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:33:01,249 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:33:01,293 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:33:01,318 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:33:01,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:33:01,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:33:01,408 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:33:01,434 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:33:02,094 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:33:02,101 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.51600003242. 2022-02-08 01:33:02,101 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:33:02,104 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:33:03,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:33:03,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:33:03,937 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:33:03,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:33:03,956 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:33:03,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:33:04,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:33:04,035 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:33:04,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:33:04,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:33:04,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:33:05,016 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:33:05,016 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.91499996185. 2022-02-08 01:33:05,016 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:33:05,019 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:33:05,489 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:33:05,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:33:06,092 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:33:06,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:33:06,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:33:06,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:33:06,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:33:06,183 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:33:06,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:33:06,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:33:06,262 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:33:06,839 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:33:06,841 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.82400012016. 2022-02-08 01:33:06,841 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:33:06,842 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:33:07,770 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:33:08,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:33:08,506 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:33:08,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:33:08,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:33:08,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:33:08,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:33:08,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:33:08,612 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:33:08,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:33:08,683 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:33:09,253 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:33:09,255 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.41299986839. 2022-02-08 01:33:09,255 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:33:09,256 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:33:10,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:33:10,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:33:10,948 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:33:10,948 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:33:10,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:33:10,994 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:33:11,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:33:11,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:33:11,073 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:33:11,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:33:11,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:33:11,766 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:33:11,767 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.51299977303. ############################################################ # # RAM FootPrint: 4023.6328125 # ############################################################ 2022-02-08 01:33:12,036 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 01:33:12,038 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 89 seconds. Reason: Iteration:16 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 2269 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:34:41,042 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 01:34:41,042 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 01:34:46,046 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:34:47,089 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:34:47,859 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:34:48,332 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:34:48,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:34:48,395 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:34:48,487 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:34:48,624 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:34:48,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:34:48,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:34:48,839 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:34:48,895 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:34:48,895 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:34:49,072 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['14281 memcached\n'], []) 2022-02-08 01:34:49,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 5 queries submitted, 0 failed, 2291 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:34:49,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:34:49,161 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['14281 memcached\n'], []) 2022-02-08 01:34:49,161 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:34:49,164 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:34:52,614 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:34:52,977 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:34:53,392 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:34:53,392 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:34:53,415 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:34:53,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:34:53,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:34:53,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:34:53,637 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:34:53,905 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:34:53,927 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:34:53,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:34:55,102 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['90589 memcached\n'], []) 2022-02-08 01:34:55,104 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 01:34:55,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:34:55,151 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['90589 memcached\n'], []) 2022-02-08 01:34:55,151 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:34:55,154 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:34:56,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:34:56,601 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:34:56,900 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:34:56,900 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:34:56,933 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:34:57,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:34:57,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:34:57,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:34:57,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:34:57,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:34:57,174 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:34:57,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:34:57,255 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['2393 memcached\n'], []) 2022-02-08 01:34:57,255 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 01:34:57,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:34:57,321 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['2393 memcached\n'], []) 2022-02-08 01:34:57,321 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:34:57,322 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:34:58,051 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:34:58,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:34:59,039 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:34:59,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:34:59,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:34:59,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:34:59,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:34:59,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:34:59,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:34:59,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:34:59,467 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:34:59,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:34:59,543 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['121627 memcached\n'], []) 2022-02-08 01:34:59,543 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 01:34:59,601 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:34:59,634 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['121627 memcached\n'], []) 2022-02-08 01:34:59,634 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:34:59,637 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:35:00,609 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:01,219 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:35:01,538 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:01,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:35:01,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:35:01,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:35:01,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:35:01,642 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:35:01,681 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:35:01,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:35:01,763 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:35:01,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:35:01,878 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['29234 memcached\n'], []) 2022-02-08 01:35:01,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 01:35:01,926 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:35:01,956 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['29234 memcached\n'], []) 2022-02-08 01:35:01,957 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:35:01,957 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:35:02,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:03,507 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:35:03,598 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:03,599 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:35:03,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:35:03,632 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:35:03,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:35:03,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:35:03,688 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:35:03,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:35:03,742 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:35:03,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:35:03,801 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['22913 memcached\n'], []) 2022-02-08 01:35:03,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 01:35:03,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:35:03,849 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['22913 memcached\n'], []) 2022-02-08 01:35:03,851 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:35:03,852 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:35:06,806 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:07,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:35:07,709 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:07,709 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:35:07,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:35:07,762 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:35:07,938 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:35:07,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:35:07,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:35:08,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:35:08,144 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:35:08,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:35:08,653 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['63968 memcached\n'], []) 2022-02-08 01:35:08,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 01:35:08,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:35:08,720 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['63968 memcached\n'], []) 2022-02-08 01:35:08,721 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:35:08,726 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:35:10,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:11,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:35:11,204 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:11,206 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:35:11,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:35:11,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:35:11,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:35:11,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:35:11,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:35:11,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:35:11,898 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:35:11,900 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:35:12,042 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['57951 memcached\n'], []) 2022-02-08 01:35:12,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 01:35:12,068 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:35:12,089 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['57951 memcached\n'], []) 2022-02-08 01:35:12,089 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:35:12,092 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:35:13,408 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:14,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:35:14,253 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:14,255 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:35:14,280 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:35:14,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:35:14,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:35:14,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:35:14,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:35:14,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:35:14,394 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:35:14,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:35:14,447 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['121988 memcached\n'], []) 2022-02-08 01:35:14,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 01:35:14,474 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:35:14,502 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['121988 memcached\n'], []) 2022-02-08 01:35:14,503 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:35:14,505 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:35:15,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:15,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:35:16,094 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:16,094 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:35:16,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:35:16,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:35:16,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:35:16,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:35:16,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:35:16,288 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:35:16,302 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:35:16,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:35:16,371 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['70069 memcached\n'], []) 2022-02-08 01:35:16,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 01:35:16,405 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:35:16,427 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['70069 memcached\n'], []) 2022-02-08 01:35:16,427 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:35:16,428 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:35:16,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:16,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:35:16,793 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:16,793 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:35:16,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:35:16,823 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:35:16,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:35:16,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:35:16,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:35:16,885 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:35:16,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:35:16,917 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 01:35:16,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:35:16,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:19,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:35:19,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:19,346 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:19,443 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:35:19,459 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 01:35:19,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:35:19,471 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:19,589 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:35:19,605 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 01:35:19,605 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:35:19,607 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:35:19,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:19,900 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:35:19,967 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:19,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:35:19,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:35:19,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:35:20,005 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:35:20,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:35:20,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:35:20,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:35:20,063 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:35:20,078 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 01:35:20,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:35:20,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:22,211 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:35:22,428 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:22,566 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:22,686 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:35:22,700 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 01:35:22,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:35:22,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:22,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:35:22,779 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 01:35:22,780 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:35:22,782 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:35:22,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:23,063 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:35:23,125 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:23,127 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:35:23,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:35:23,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:35:23,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:35:23,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:35:23,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:35:23,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:35:23,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:35:23,243 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 01:35:23,267 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:35:23,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:25,994 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:35:26,204 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:26,334 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:26,451 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:35:26,466 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 01:35:26,466 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:35:26,484 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:26,543 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:35:26,568 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 01:35:26,569 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:35:26,571 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 01:35:26,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:26,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 01:35:26,960 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:26,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 01:35:26,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 01:35:26,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 01:35:27,005 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 01:35:27,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 01:35:27,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 01:35:27,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 01:35:27,066 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:35:27,084 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 01:35:27,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:35:27,122 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:27,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:35:27,155 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:27,170 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:27,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:35:27,198 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 01:35:27,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:35:27,213 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:27,229 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:35:27,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:27,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:35:27,276 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:27,293 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:27,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:27,326 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:27,344 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:27,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:27,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:27,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:27,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:27,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:35:27,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:27,466 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:27,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:27,502 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:27,517 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:27,533 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:27,551 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:27,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:27,595 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 01:35:27,598 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:35:27,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:27,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:35:28,006 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:28,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:35:28,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:35:28,045 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:35:28,058 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:35:28,076 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:35:28,091 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:35:28,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:35:28,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:35:28,160 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 01:35:28,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:35:28,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:28,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:35:28,836 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:28,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:28,994 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:35:29,104 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 01:35:29,104 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:35:29,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:29,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:35:29,588 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 01:35:29,589 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:35:29,592 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 01:35:29,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:29,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 01:35:29,974 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:29,974 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 01:35:29,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 01:35:30,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 01:35:30,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 01:35:30,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 01:35:30,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 01:35:30,068 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 01:35:30,086 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:35:30,108 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 01:35:30,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:35:30,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:30,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:35:30,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:30,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:30,209 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:35:30,223 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 01:35:30,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:35:30,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:30,252 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:35:30,265 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:30,280 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:35:30,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:30,313 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:30,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:30,354 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:30,369 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:30,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:30,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:30,420 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:30,436 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:30,454 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:35:30,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:30,489 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:30,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:30,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:30,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:30,558 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:30,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:30,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:30,621 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 01:35:30,625 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:35:30,724 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:30,943 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:35:31,006 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:31,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:35:31,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:35:31,035 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:35:31,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:35:31,066 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:35:31,081 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:35:31,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:35:31,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:35:31,124 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 01:35:31,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:35:31,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:33,068 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:35:33,290 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:33,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:33,563 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:35:33,579 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 01:35:33,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:35:33,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:33,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:35:33,644 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 01:35:33,645 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:35:33,648 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:35:33,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:33,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:35:34,076 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:34,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:35:34,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:35:34,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:35:34,127 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:35:34,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:35:34,160 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:35:34,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:35:34,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:35:34,210 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 01:35:34,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:35:34,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:36,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:35:36,290 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:36,423 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:36,532 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:35:36,551 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 01:35:36,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:35:36,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:36,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:35:36,657 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 01:35:36,657 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:35:36,661 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 01:35:36,752 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:37,012 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 01:35:37,088 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:37,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 01:35:37,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 01:35:37,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 01:35:37,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 01:35:37,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 01:35:37,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 01:35:37,186 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 01:35:37,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:35:37,220 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 01:35:37,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:35:37,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:37,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:35:37,303 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:37,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:37,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:35:37,368 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 01:35:37,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:35:37,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:37,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:35:37,414 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:37,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:35:37,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:37,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:37,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:37,507 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:37,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:37,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:37,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:37,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:37,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:37,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:35:37,670 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:37,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:37,716 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:37,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:37,755 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:37,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:37,790 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:37,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:37,832 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 01:35:37,834 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 01:35:37,915 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:38,145 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 01:35:38,230 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:38,232 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 01:35:38,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 01:35:38,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 01:35:38,276 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 01:35:38,295 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 01:35:38,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 01:35:38,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 01:35:38,349 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:35:38,365 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 01:35:38,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:35:38,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:38,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:35:38,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:38,459 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:38,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:35:38,486 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 01:35:38,487 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:35:38,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:38,517 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:35:38,532 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:38,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:35:38,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:38,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:38,604 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:38,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:38,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:38,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:38,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:38,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:38,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:38,734 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:35:38,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:38,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:38,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:38,806 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:38,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:38,844 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:38,864 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:38,882 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:38,900 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 01:35:38,901 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:35:39,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:39,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:35:39,944 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:39,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:35:40,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:35:40,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:35:40,367 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:35:40,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:35:40,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:35:40,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:35:40,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:35:40,586 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 01:35:40,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:35:40,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:42,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:35:43,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:43,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:43,816 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:35:43,901 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 01:35:43,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:35:44,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:44,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:35:44,101 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 01:35:44,101 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:35:44,102 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:35:44,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:44,482 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:35:44,661 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:44,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:35:44,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:35:44,688 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:35:44,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:35:44,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:35:44,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:35:44,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:35:44,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:35:44,802 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 01:35:44,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:35:44,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:46,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:35:46,969 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:47,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:47,318 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:35:47,391 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 01:35:47,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:35:47,421 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:47,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:35:47,499 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 01:35:47,500 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:35:47,506 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:35:47,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:48,206 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:35:48,367 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:48,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:35:48,392 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:35:48,420 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:35:48,434 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:35:48,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:35:48,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:35:48,484 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:35:48,510 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:35:48,529 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 01:35:48,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:35:48,571 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 2318 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:35:49,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:35:49,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:49,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:50,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:35:50,052 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 01:35:50,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:35:50,071 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:50,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:35:50,122 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 01:35:50,124 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:35:50,125 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:35:50,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:50,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:35:50,573 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:50,573 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:35:50,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:35:50,614 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:35:50,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:35:50,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:35:50,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:35:50,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:35:50,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:35:50,742 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 01:35:50,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:35:50,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:52,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:35:52,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:53,127 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:53,263 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:35:53,286 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 01:35:53,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:35:53,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:53,346 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:35:53,368 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 01:35:53,368 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:35:53,369 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 01:35:53,469 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:53,709 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 01:35:53,769 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:53,770 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 01:35:53,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 01:35:53,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 01:35:53,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 01:35:53,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 01:35:53,849 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 01:35:53,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 01:35:53,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:35:53,901 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 01:35:53,928 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:35:53,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:53,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:35:53,992 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:54,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:35:54,030 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:35:54,053 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 01:35:54,053 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:35:54,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:35:54,094 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:35:54,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:54,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:35:54,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:54,165 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:54,183 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:54,198 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:54,214 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:54,232 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:54,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:54,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:35:54,285 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:54,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:35:54,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:54,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:54,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:54,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:54,392 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:54,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:54,434 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:54,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:35:54,473 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 01:35:54,473 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:35:54,474 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:35:54,681 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:35:55,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:35:55,595 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:35:55,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:35:55,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:35:55,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:35:55,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:35:55,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:35:55,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:35:56,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:35:56,094 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:35:57,420 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:35:57,421 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:35:59,421 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:35:59,740 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:35:59,740 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:36:01,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:02,128 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:02,130 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:36:04,130 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:04,605 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:04,605 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:36:06,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:07,197 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:07,197 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:36:09,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:09,460 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:09,461 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:36:11,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:11,868 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:11,868 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:36:13,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:14,140 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:14,140 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:36:16,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:16,378 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:16,378 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:36:18,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:18,684 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:18,684 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:36:20,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:21,049 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:21,051 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:36:23,051 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:23,525 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:23,525 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:36:25,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:25,756 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:25,756 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:36:27,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:28,019 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:36:28,023 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 33.5499999523. 2022-02-08 01:36:28,023 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:36:28,026 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:36:28,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:36:28,380 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:36:28,456 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:36:28,456 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:36:28,471 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:36:28,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:36:28,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:36:28,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:36:28,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:36:28,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:36:28,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:29,144 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:36:29,144 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.12199997902. 2022-02-08 01:36:29,144 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:36:29,147 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:36:30,117 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:36:30,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:36:31,085 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:36:31,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:36:31,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:36:31,168 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:36:31,249 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:36:31,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:36:31,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:36:31,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:36:31,530 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:32,721 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:32,721 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 01:36:34,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:35,005 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:35,005 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 01:36:37,005 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:37,582 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:37,582 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 01:36:39,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:40,052 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:40,053 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 01:36:42,058 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:42,323 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:36:42,325 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 13.1800000668. 2022-02-08 01:36:42,325 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:36:42,326 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:36:43,160 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:36:43,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:36:43,690 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:36:43,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:36:43,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:36:43,762 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:36:43,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:36:43,821 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:36:43,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:36:43,877 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:36:43,903 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:44,648 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:36:44,648 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.32399988174. 2022-02-08 01:36:44,650 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:36:44,651 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:36:45,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:36:45,466 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:36:45,947 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:36:45,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:36:46,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:36:46,160 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:36:46,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:36:46,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:36:46,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:36:46,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:36:46,739 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:47,798 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:47,798 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 5 queries submitted, 0 failed, 2344 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:36:49,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:50,209 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:50,210 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 01:36:52,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:52,973 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:36:52,973 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 01:36:54,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:55,203 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:36:55,204 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 10.5549998283. 2022-02-08 01:36:55,204 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:36:55,207 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:36:56,109 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:36:56,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:36:56,571 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:36:56,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:36:56,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:36:56,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:36:56,645 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:36:56,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:36:56,693 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:36:56,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:36:56,775 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:57,516 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:36:57,516 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.31100010872. 2022-02-08 01:36:57,516 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:36:57,519 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:36:58,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:36:58,459 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:36:58,578 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:36:58,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:36:58,599 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:36:58,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:36:58,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:36:58,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:36:58,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:36:58,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:36:58,752 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:36:59,483 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:36:59,483 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.96700000763. 2022-02-08 01:36:59,483 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:36:59,484 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:37:00,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:37:01,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:37:01,401 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:37:01,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:37:01,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:37:01,576 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:37:01,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:37:01,729 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:37:01,780 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:37:01,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:37:01,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:37:03,125 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:37:03,127 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.64400005341. 2022-02-08 01:37:03,127 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:37:03,128 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:37:03,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:37:04,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:37:04,204 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:37:04,204 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:37:04,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:37:04,237 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:37:04,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:37:04,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:37:04,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:37:04,316 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:37:04,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:37:04,887 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:37:04,888 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.76099991798. 2022-02-08 01:37:04,888 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:37:04,890 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:37:05,770 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:37:06,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:37:06,819 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:37:06,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:37:06,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:37:07,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:37:07,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:37:07,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:37:07,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:37:07,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:37:07,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:37:08,582 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:37:08,582 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:37:10,584 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:37:10,865 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:37:10,865 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:37:12,890 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:37:13,088 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:37:13,089 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 8.20199990273. ############################################################ # # RAM FootPrint: 4042.625 # ############################################################ 2022-02-08 01:37:13,444 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 01:37:13,444 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 109 seconds. Reason: Iteration:17 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 2369 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 2388 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:39:02,486 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 01:39:02,487 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 01:39:07,490 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:39:08,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:09,252 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:39:09,698 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:09,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:39:09,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:39:09,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:39:09,974 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:39:10,066 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:39:10,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:39:10,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:39:10,351 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:39:10,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:39:10,464 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['15227 memcached\n'], []) 2022-02-08 01:39:10,464 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 01:39:10,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:39:10,655 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['15227 memcached\n'], []) 2022-02-08 01:39:10,655 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:39:10,657 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:39:11,263 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:11,594 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:39:11,686 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:11,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:39:11,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:39:11,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:39:11,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:39:11,760 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:39:11,775 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:39:11,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:39:11,815 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:39:11,816 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:39:11,913 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['91310 memcached\n'], []) 2022-02-08 01:39:11,914 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 01:39:11,941 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:39:11,963 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['91310 memcached\n'], []) 2022-02-08 01:39:11,964 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:39:11,966 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:39:12,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:12,933 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:39:13,086 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:13,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:39:13,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:39:13,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:39:13,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:39:13,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:39:13,196 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:39:13,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:39:13,226 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:39:13,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:39:13,292 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['3137 memcached\n'], []) 2022-02-08 01:39:13,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 01:39:13,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:39:13,348 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['3137 memcached\n'], []) 2022-02-08 01:39:13,348 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:39:13,351 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:39:14,262 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:15,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:39:15,194 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:15,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:39:15,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:39:15,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:39:15,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:39:15,285 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:39:15,313 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:39:15,349 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:39:15,368 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:39:15,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:39:15,440 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['122471 memcached\n'], []) 2022-02-08 01:39:15,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 01:39:15,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:39:15,503 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['122471 memcached\n'], []) 2022-02-08 01:39:15,503 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:39:15,506 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:39:16,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:17,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:39:17,473 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:17,474 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:39:17,492 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:39:17,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:39:17,612 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:39:17,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:39:17,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:39:17,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:39:17,852 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:39:17,854 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:39:17,930 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['29955 memcached\n'], []) 2022-02-08 01:39:17,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 01:39:18,002 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:39:18,026 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['29955 memcached\n'], []) 2022-02-08 01:39:18,026 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:39:18,029 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:39:19,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:20,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:39:20,473 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:20,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:39:20,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:39:20,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:39:20,545 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:39:20,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:39:20,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:39:20,624 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:39:20,766 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:39:20,767 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:39:20,878 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['23635 memcached\n'], []) 2022-02-08 01:39:20,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 01:39:20,926 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:39:20,956 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['23635 memcached\n'], []) 2022-02-08 01:39:20,956 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:39:20,957 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:39:22,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:23,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:39:23,654 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:23,655 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:39:23,714 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:39:23,821 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:39:23,971 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:39:24,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:39:24,033 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:39:24,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:39:24,095 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:39:24,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:39:24,170 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['64694 memcached\n'], []) 2022-02-08 01:39:24,170 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 01:39:24,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:39:24,216 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['64694 memcached\n'], []) 2022-02-08 01:39:24,216 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:39:24,217 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:39:25,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:26,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:39:27,013 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:27,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:39:27,137 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:39:27,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:39:27,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:39:27,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:39:27,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:39:27,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:39:27,368 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:39:27,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:39:27,444 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['58676 memcached\n'], []) 2022-02-08 01:39:27,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 01:39:27,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:39:27,647 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['58676 memcached\n'], []) 2022-02-08 01:39:27,648 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:39:27,648 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:39:28,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:29,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:39:29,186 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:29,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:39:29,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:39:29,221 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:39:29,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:39:29,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:39:29,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:39:29,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:39:29,312 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:39:29,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:39:29,355 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['122717 memcached\n'], []) 2022-02-08 01:39:29,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 01:39:29,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:39:29,405 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['122717 memcached\n'], []) 2022-02-08 01:39:29,405 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:39:29,407 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:39:30,433 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:30,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:39:31,177 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:31,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:39:31,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:39:31,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:39:31,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:39:31,466 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:39:31,482 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:39:31,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:39:31,525 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:39:31,525 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:39:31,604 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['70797 memcached\n'], []) 2022-02-08 01:39:31,604 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 01:39:31,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:39:31,663 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['70797 memcached\n'], []) 2022-02-08 01:39:31,663 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:39:31,664 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:39:31,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:31,982 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:39:32,049 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:32,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:39:32,068 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:39:32,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:39:32,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:39:32,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:39:32,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:39:32,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:39:32,165 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:39:32,183 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 01:39:32,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:39:32,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:34,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:39:34,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:34,357 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:34,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:39:34,473 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 01:39:34,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:39:34,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:34,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:39:34,543 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 01:39:34,543 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:39:34,546 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:39:34,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:34,844 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:39:34,953 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:34,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:39:34,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:39:34,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:39:35,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:39:35,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:39:35,035 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:39:35,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:39:35,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:39:35,078 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 01:39:35,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:39:35,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:37,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:39:37,336 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:37,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:37,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:39:37,607 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 01:39:37,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:39:37,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:37,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:39:37,711 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 01:39:37,713 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:39:37,714 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:39:37,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:38,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:39:38,253 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:38,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:39:38,276 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:39:38,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:39:38,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:39:38,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:39:38,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:39:38,349 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:39:38,364 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:39:38,377 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 01:39:38,403 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:39:38,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:40,352 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:39:40,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:40,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:40,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:39:40,845 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 01:39:40,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:39:40,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:40,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:39:40,943 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 01:39:40,943 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:39:40,947 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 01:39:41,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:41,263 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 01:39:41,329 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:41,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 01:39:41,344 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 01:39:41,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 01:39:41,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 01:39:41,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 01:39:41,395 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 01:39:41,408 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 01:39:41,421 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:39:41,434 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 01:39:41,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:39:41,482 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:41,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:39:41,510 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:41,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:41,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:39:41,549 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 01:39:41,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:39:41,563 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:41,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:39:41,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:41,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:39:41,624 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:41,640 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:41,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:41,673 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:41,688 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:41,704 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:41,721 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:41,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:41,755 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:41,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:39:41,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:41,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:41,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:41,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:41,864 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:41,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:41,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:41,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:41,961 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 01:39:41,964 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:39:42,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:42,275 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:39:42,355 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:42,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:39:42,369 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:39:42,382 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:39:42,400 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:39:42,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:39:42,438 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:39:42,454 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:39:42,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:39:42,493 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 01:39:42,528 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:39:42,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:43,431 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:39:43,532 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:43,588 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:43,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:39:43,707 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 01:39:43,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:39:43,724 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:43,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:39:43,822 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 01:39:43,822 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:39:43,825 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 01:39:43,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:44,135 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 01:39:44,198 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:44,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 01:39:44,213 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 01:39:44,224 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 01:39:44,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 01:39:44,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 01:39:44,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 01:39:44,278 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 01:39:44,290 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:39:44,305 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 01:39:44,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:39:44,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:44,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:39:44,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:44,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:44,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:39:44,414 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 01:39:44,414 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:39:44,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:44,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:39:44,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:44,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:39:44,484 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:44,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:44,517 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:44,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:44,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:44,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:44,584 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:44,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:44,614 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:44,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:39:44,647 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:44,663 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:44,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:44,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:44,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:44,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:44,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:44,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:44,773 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 01:39:44,778 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:39:44,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:45,091 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:39:45,201 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:45,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:39:45,214 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:39:45,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:39:45,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:39:45,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:39:45,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:39:45,290 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:39:45,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:39:45,325 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 01:39:45,354 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:39:45,367 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:47,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:39:47,790 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:47,931 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:48,056 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:39:48,075 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 01:39:48,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:39:48,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:48,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:39:48,144 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 01:39:48,144 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:39:48,148 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:39:48,227 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:48,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:39:48,516 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:48,517 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:39:48,533 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:39:48,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:39:48,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:39:48,584 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:39:48,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:39:48,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:39:48,632 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:39:48,653 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 01:39:48,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:39:48,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 2414 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:39:50,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:39:50,617 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:50,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:50,852 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:39:50,871 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 01:39:50,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:39:50,890 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:50,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:39:50,967 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 01:39:50,967 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:39:50,969 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 01:39:51,059 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:51,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 01:39:51,382 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:51,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 01:39:51,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 01:39:51,420 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 01:39:51,434 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 01:39:51,451 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 01:39:51,466 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 01:39:51,477 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 01:39:51,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:39:51,506 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 01:39:51,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:39:51,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:51,563 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:39:51,576 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:51,589 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:51,602 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:39:51,617 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 01:39:51,617 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:39:51,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:51,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:39:51,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:51,683 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:39:51,704 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:51,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:51,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:51,765 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:51,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:51,806 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:51,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:51,846 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:51,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:51,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:39:51,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:51,926 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:51,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:51,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:51,984 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:52,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:52,025 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:52,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:52,065 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 01:39:52,066 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 01:39:52,176 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:52,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 01:39:52,454 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:52,456 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 01:39:52,469 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 01:39:52,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 01:39:52,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 01:39:52,510 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 01:39:52,523 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 01:39:52,538 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 01:39:52,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:39:52,566 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 01:39:52,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:39:52,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:52,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:39:52,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:52,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:52,655 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:39:52,667 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 01:39:52,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:39:52,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:52,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:39:52,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:52,721 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:39:52,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:52,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:52,783 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:52,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:52,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:52,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:52,854 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:52,872 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:39:52,905 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:52,923 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:39:52,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:52,966 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:52,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:53,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:53,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:53,043 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:53,059 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:53,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:39:53,092 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 01:39:53,094 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:39:53,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:53,421 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:39:53,499 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:53,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:39:53,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:39:53,532 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:39:53,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:39:53,571 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:39:53,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:39:53,609 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:39:53,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:39:53,654 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 01:39:53,686 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:39:53,752 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:55,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:39:56,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:56,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:56,477 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:39:56,553 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 01:39:56,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:39:56,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:56,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:39:56,625 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 01:39:56,625 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:39:56,628 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:39:56,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:39:56,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:39:57,029 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:39:57,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:39:57,045 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:39:57,058 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:39:57,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:39:57,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:39:57,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:39:57,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:39:57,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:39:57,163 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 01:39:57,188 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:39:57,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:59,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:39:59,263 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:59,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:39:59,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:39:59,558 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 01:39:59,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:39:59,576 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:39:59,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:39:59,632 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 01:39:59,632 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:39:59,634 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:40:00,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:40:01,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:40:01,555 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:40:01,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:40:01,571 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:40:01,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:40:01,665 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:40:01,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:40:01,709 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:40:01,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:40:01,762 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:40:01,779 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 01:40:01,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:40:01,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:40:03,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:40:03,298 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:40:03,385 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:40:03,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:40:03,494 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 01:40:03,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:40:03,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:40:03,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:40:03,566 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 01:40:03,566 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:40:03,569 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:40:03,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:40:03,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:40:04,068 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:40:04,068 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:40:04,086 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:40:04,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:40:04,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:40:04,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:40:04,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:40:04,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:40:04,209 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:40:04,233 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 01:40:04,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:40:04,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:40:06,109 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:40:06,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:40:06,507 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:40:06,663 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:40:06,684 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 01:40:06,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:40:06,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:40:06,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:40:06,766 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 01:40:06,766 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:40:06,769 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 01:40:06,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:40:07,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 01:40:07,147 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:40:07,147 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 01:40:07,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 01:40:07,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 01:40:07,191 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 01:40:07,214 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 01:40:07,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 01:40:07,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 01:40:07,275 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:40:07,292 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 01:40:07,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:40:07,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:40:07,357 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:40:07,372 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:40:07,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:40:07,403 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:40:07,417 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 01:40:07,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:40:07,433 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:40:07,448 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:40:07,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:40:07,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:40:07,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:40:07,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:40:07,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:40:07,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:40:07,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:40:07,602 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:40:07,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:40:07,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:40:07,660 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:40:07,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:40:07,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:40:07,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:40:07,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:40:07,749 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:40:07,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:40:07,785 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:40:07,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:40:07,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:40:07,839 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 01:40:07,841 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:40:07,842 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:40:07,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:40:08,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:40:08,394 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:40:08,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:40:08,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:40:08,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:40:08,443 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:40:08,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:40:08,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:40:08,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:40:08,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:08,788 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:08,789 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:40:10,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:11,154 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:11,154 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:40:13,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:13,361 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:13,361 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:40:15,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:15,783 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:15,783 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:40:17,783 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:18,239 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:18,239 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:40:20,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:20,559 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:20,559 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:40:22,561 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:23,154 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:23,155 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:40:25,155 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:25,670 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:25,670 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:40:27,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:28,141 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:28,141 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:40:30,142 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:30,427 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:30,427 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:40:32,428 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:32,828 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:32,828 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:40:34,829 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:35,048 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:35,049 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:40:37,053 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:37,417 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:37,418 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:40:39,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:39,950 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:39,950 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:40:41,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:42,319 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:42,319 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:40:44,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:44,565 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:40:44,566 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 36.7259998322. 2022-02-08 01:40:44,566 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:40:44,569 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:40:44,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:40:44,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:40:44,963 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:40:44,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:40:44,977 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:40:44,999 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:40:45,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:40:45,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:40:45,053 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:40:45,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:40:45,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:45,575 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:40:45,575 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.00800013542. 2022-02-08 01:40:45,575 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:40:45,578 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:40:46,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:40:47,035 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:40:47,693 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:40:47,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:40:47,788 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:40:47,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:40:47,936 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:40:48,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:40:48,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:40:48,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:40:48,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 2440 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:40:49,421 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:49,421 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 01:40:51,421 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:51,648 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:51,648 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 01:40:53,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:53,901 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:40:53,901 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 8.3259999752. 2022-02-08 01:40:53,901 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:40:53,904 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:40:54,076 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:40:54,598 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:40:54,948 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:40:54,948 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:40:55,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:40:55,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:40:55,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:40:55,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:40:55,459 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:40:55,553 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:40:55,693 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:57,217 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:57,217 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 01:40:59,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:40:59,562 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:40:59,562 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 01:41:01,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:01,917 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:01,917 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 01:41:03,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:04,211 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:04,213 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 01:41:06,213 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:06,450 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:41:06,450 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 12.5490000248. 2022-02-08 01:41:06,450 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:41:06,453 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:41:07,405 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:41:07,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:41:08,071 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:41:08,071 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:41:08,094 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:41:08,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:41:08,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:41:08,170 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:41:08,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:41:08,227 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:41:08,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:08,948 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:41:08,948 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.49799990654. 2022-02-08 01:41:08,950 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:41:08,951 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:41:09,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:41:09,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:41:09,756 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:41:09,757 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:41:09,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:41:09,818 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:41:09,857 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:41:09,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:41:09,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:41:09,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:41:10,010 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:10,717 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:41:10,717 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.76699995995. 2022-02-08 01:41:10,717 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:41:10,719 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:41:11,122 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:41:11,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:41:11,552 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:41:11,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:41:11,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:41:11,602 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:41:11,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:41:11,681 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:41:11,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:41:11,732 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:41:11,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:12,578 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:12,579 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 01:41:14,581 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:14,802 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:14,803 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 01:41:16,803 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:16,989 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:16,989 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 01:41:18,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:19,232 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:19,233 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 01:41:21,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:21,539 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:21,539 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 01:41:23,543 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:23,747 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:23,747 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 01:41:25,749 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:26,046 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:26,046 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 01:41:28,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:28,352 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:28,352 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 01:41:30,354 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:30,592 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:30,592 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 01:41:32,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:32,828 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:32,829 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 01:41:34,829 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:35,036 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:35,036 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 01:41:37,038 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:37,309 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:37,309 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 01:41:39,311 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:39,668 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:39,668 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 01:41:41,670 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:41,858 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:41,858 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 01:41:43,859 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:44,160 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:44,160 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 01:41:46,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:46,398 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:46,398 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 01:41:48,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:48,614 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:41:48,615 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 37.8980000019. 2022-02-08 01:41:48,615 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:41:48,617 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 5 queries submitted, 0 failed, 2463 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:41:49,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:41:49,510 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:41:49,644 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:41:49,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:41:49,665 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:41:49,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:41:49,706 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:41:49,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:41:49,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:41:49,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:41:49,803 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:50,503 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:41:50,503 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.88899993896. 2022-02-08 01:41:50,503 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:41:50,506 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:41:51,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:41:51,471 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:41:51,578 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:41:51,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:41:51,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:41:51,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:41:51,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:41:51,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:41:51,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:41:51,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:41:51,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:52,296 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:41:52,329 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.8259999752. 2022-02-08 01:41:52,329 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:41:52,332 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:41:52,507 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:41:52,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:41:52,973 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:41:52,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:41:52,999 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:41:53,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:41:53,043 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:41:53,079 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:41:53,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:41:53,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:41:53,168 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:53,901 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:53,901 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:41:55,903 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:56,099 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:56,099 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:41:58,099 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:41:58,286 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:41:58,286 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:00,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:00,496 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:00,496 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:02,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:02,723 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:02,723 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:04,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:04,950 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:04,950 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:06,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:07,170 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:07,171 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:09,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:09,401 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:09,401 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:11,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:11,638 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:11,640 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:13,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:13,859 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:13,859 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:15,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:16,066 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:16,066 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:18,066 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:18,328 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:18,329 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:20,331 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:20,505 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:20,506 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:22,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:22,690 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:22,690 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:24,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:24,878 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:24,878 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:26,880 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:27,219 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:27,219 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:29,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:29,394 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:29,394 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:31,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:31,619 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:31,621 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:33,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:33,934 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:33,934 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:35,936 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:36,237 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:36,237 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:38,237 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:38,602 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:38,604 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:40,604 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:40,798 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:40,798 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:42,798 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:43,168 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:43,168 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:45,170 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:45,451 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:45,451 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:47,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:47,726 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:47,726 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 5 queries submitted, 0 failed, 2489 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:42:49,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:49,920 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:49,921 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:51,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:52,107 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:52,108 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:54,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:54,302 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:54,302 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:56,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:56,525 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:56,525 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:42:58,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:42:58,828 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:42:58,828 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:43:00,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:43:01,055 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:43:01,055 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:43:03,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:43:03,227 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:43:03,227 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:43:05,229 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:43:05,460 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:43:05,461 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:43:07,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:43:07,707 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:43:07,709 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:43:09,709 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:43:09,970 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:43:09,970 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:43:11,971 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:43:12,220 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:43:12,220 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:43:14,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:43:14,509 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:43:14,509 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:43:16,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:43:16,812 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:43:16,812 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:43:18,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:43:19,059 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:43:19,059 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:43:21,061 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:43:21,292 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:43:21,293 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:43:23,293 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:43:23,730 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:43:23,730 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:43:25,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:43:26,022 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:43:26,022 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:43:28,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:43:28,315 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:43:28,316 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:43:30,316 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:43:30,608 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:43:30,609 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 01:43:32,609 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:43:32,980 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:43:32,980 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 100.650000095. ############################################################ # # RAM FootPrint: 4074.30078125 # ############################################################ 2022-02-08 01:43:33,312 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 01:43:33,313 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 79 seconds. Reason: Iteration:18 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 2511 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 2535 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:44:52,318 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 01:44:52,319 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 01:44:57,322 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:44:57,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:44:58,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:44:58,400 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:44:58,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:44:58,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:44:58,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:44:58,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:44:58,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:44:58,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:44:58,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:44:58,520 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:44:58,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:44:58,578 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['16161 memcached\n'], []) 2022-02-08 01:44:58,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 01:44:58,602 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:44:58,625 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['16161 memcached\n'], []) 2022-02-08 01:44:58,625 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:44:58,627 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:44:58,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:44:58,914 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:44:58,970 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:44:58,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:44:58,984 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:44:58,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:44:59,007 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:44:59,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:44:59,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:44:59,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:44:59,061 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:44:59,061 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:44:59,117 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['92054 memcached\n'], []) 2022-02-08 01:44:59,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 01:44:59,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:44:59,161 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['92054 memcached\n'], []) 2022-02-08 01:44:59,161 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:44:59,164 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:45:00,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:00,507 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:45:00,674 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:00,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:45:00,688 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:45:00,704 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:45:00,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:45:00,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:45:00,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:45:00,778 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:45:00,792 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:45:00,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:45:00,851 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['3928 memcached\n'], []) 2022-02-08 01:45:00,852 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 01:45:00,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:45:00,900 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['3928 memcached\n'], []) 2022-02-08 01:45:00,900 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:45:00,903 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:45:01,803 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:02,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:45:02,244 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:02,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:45:02,262 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:45:02,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:45:02,298 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:45:02,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:45:02,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:45:02,367 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:45:02,382 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:45:02,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:45:02,444 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['123307 memcached\n'], []) 2022-02-08 01:45:02,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 01:45:02,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:45:02,496 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['123307 memcached\n'], []) 2022-02-08 01:45:02,496 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:45:02,499 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:45:03,553 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:03,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:45:03,967 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:03,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:45:03,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:45:03,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:45:04,010 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:45:04,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:45:04,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:45:04,084 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:45:04,096 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:45:04,096 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:45:04,153 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['30731 memcached\n'], []) 2022-02-08 01:45:04,153 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 01:45:04,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:45:04,196 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['30731 memcached\n'], []) 2022-02-08 01:45:04,196 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:45:04,197 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:45:05,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:05,421 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:45:05,571 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:05,571 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:45:05,586 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:45:05,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:45:05,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:45:05,640 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:45:05,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:45:05,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:45:05,703 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:45:05,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:45:05,766 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['24376 memcached\n'], []) 2022-02-08 01:45:05,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 01:45:05,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:45:05,812 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['24376 memcached\n'], []) 2022-02-08 01:45:05,812 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:45:05,815 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:45:06,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:07,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:45:07,447 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:07,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:45:07,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:45:07,479 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:45:07,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:45:07,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:45:07,528 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:45:07,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:45:07,572 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:45:07,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:45:07,631 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['65433 memcached\n'], []) 2022-02-08 01:45:07,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 01:45:07,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:45:07,676 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['65433 memcached\n'], []) 2022-02-08 01:45:07,676 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:45:07,677 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:45:08,586 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:08,951 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:45:09,032 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:09,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:45:09,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:45:09,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:45:09,076 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:45:09,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:45:09,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:45:09,135 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:45:09,148 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:45:09,150 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:45:09,203 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['59412 memcached\n'], []) 2022-02-08 01:45:09,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 01:45:09,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:45:09,249 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['59412 memcached\n'], []) 2022-02-08 01:45:09,249 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:45:09,250 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:45:09,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:09,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:45:09,594 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:09,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:45:09,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:45:09,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:45:09,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:45:09,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:45:09,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:45:09,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:45:09,687 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:45:09,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:45:09,740 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['123461 memcached\n'], []) 2022-02-08 01:45:09,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 01:45:09,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:45:09,786 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['123461 memcached\n'], []) 2022-02-08 01:45:09,786 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:45:09,789 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:45:10,834 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:11,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:45:11,378 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:11,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:45:11,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:45:11,408 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:45:11,421 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:45:11,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:45:11,456 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:45:11,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:45:11,510 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:45:11,512 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:45:11,572 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['71600 memcached\n'], []) 2022-02-08 01:45:11,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 01:45:11,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:45:11,631 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['71600 memcached\n'], []) 2022-02-08 01:45:11,631 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:45:11,634 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:45:11,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:11,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:45:12,013 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:12,013 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:45:12,028 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:45:12,040 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:45:12,053 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:45:12,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:45:12,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:45:12,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:45:12,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:45:12,127 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 01:45:12,153 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:45:12,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:13,525 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:45:13,711 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:13,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:13,943 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:45:13,957 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 01:45:13,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:45:13,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:14,025 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:45:14,040 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 01:45:14,040 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:45:14,042 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:45:14,132 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:14,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:45:14,430 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:14,431 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:45:14,446 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:45:14,459 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:45:14,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:45:14,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:45:14,505 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:45:14,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:45:14,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:45:14,551 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 01:45:14,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:45:14,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:16,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:45:16,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:16,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:17,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:45:17,099 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 01:45:17,099 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:45:17,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:17,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:45:17,178 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 01:45:17,178 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:45:17,180 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:45:17,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:17,528 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:45:17,589 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:17,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:45:17,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:45:17,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:45:17,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:45:17,645 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:45:17,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:45:17,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:45:17,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:45:17,706 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 01:45:17,732 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:45:17,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:19,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:45:19,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:20,104 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:20,227 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:45:20,240 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 01:45:20,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:45:20,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:20,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:45:20,322 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 01:45:20,322 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:45:20,325 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 01:45:20,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:20,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 01:45:20,688 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:20,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 01:45:20,703 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 01:45:20,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 01:45:20,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 01:45:20,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 01:45:20,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 01:45:20,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 01:45:20,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:45:20,811 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 01:45:20,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:45:20,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:20,867 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:45:20,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:20,892 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:20,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:45:20,918 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 01:45:20,918 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:45:20,931 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:20,946 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:45:20,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:20,977 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:45:20,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:21,015 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:21,033 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:21,051 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:21,071 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:21,089 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:21,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:21,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:21,144 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:21,160 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:45:21,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:21,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:21,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:21,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:21,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:21,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:21,272 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:21,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:21,299 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 01:45:21,302 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:45:21,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:21,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:45:21,690 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:21,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:45:21,709 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:45:21,729 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:45:21,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:45:21,760 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:45:21,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:45:21,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:45:21,803 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:45:21,822 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 01:45:21,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:45:21,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:21,959 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:45:22,043 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:22,094 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:22,142 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:45:22,157 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 01:45:22,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:45:22,170 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:22,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:45:22,226 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 01:45:22,226 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:45:22,227 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 01:45:22,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:22,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 01:45:22,601 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:22,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 01:45:22,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 01:45:22,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 01:45:22,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 01:45:22,683 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 01:45:22,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 01:45:22,709 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 01:45:22,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:45:22,736 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 01:45:22,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:45:22,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:22,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:45:22,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:22,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:22,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:45:22,857 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 01:45:22,857 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:45:22,869 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:22,882 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:45:22,895 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:22,913 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:45:22,933 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:22,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:22,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:22,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:23,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:23,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:23,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:23,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:23,068 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:23,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:45:23,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:23,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:23,127 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:23,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:23,155 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:23,170 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:23,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:23,198 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:23,211 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 01:45:23,214 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:45:23,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:23,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:45:23,595 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:23,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:45:23,609 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:45:23,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:45:23,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:45:23,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:45:23,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:45:23,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:45:23,698 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:45:23,717 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 01:45:23,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:45:23,755 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:25,714 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:45:25,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:26,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:26,213 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:45:26,230 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 01:45:26,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:45:26,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:26,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:45:26,299 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 01:45:26,301 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:45:26,302 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:45:26,400 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:26,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:45:26,723 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:26,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:45:26,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:45:26,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:45:26,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:45:26,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:45:26,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:45:26,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:45:26,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:45:26,869 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 01:45:26,900 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:45:26,918 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:28,803 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:45:28,999 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:29,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:29,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:45:29,255 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 01:45:29,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:45:29,276 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:29,323 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:45:29,345 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 01:45:29,345 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:45:29,346 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 01:45:29,436 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:29,663 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 01:45:29,743 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:29,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 01:45:29,762 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 01:45:29,778 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 01:45:29,793 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 01:45:29,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 01:45:29,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 01:45:29,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 01:45:29,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:45:29,869 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 01:45:29,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:45:29,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:29,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:45:29,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:29,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:29,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:45:29,974 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 01:45:29,974 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:45:29,987 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:30,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:45:30,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:30,035 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:45:30,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:30,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:30,096 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:30,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:30,137 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:30,155 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:30,176 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:30,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:30,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:30,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:45:30,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:30,275 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:30,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:30,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:30,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:30,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:30,354 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:30,369 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:30,390 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 01:45:30,392 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 01:45:30,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:30,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 01:45:30,739 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:30,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 01:45:30,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 01:45:30,765 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 01:45:30,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 01:45:30,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 01:45:30,808 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 01:45:30,821 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 01:45:30,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:45:30,852 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 01:45:30,882 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:45:30,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:30,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:45:30,926 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:30,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:30,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:45:30,963 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 01:45:30,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:45:30,974 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:30,989 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:45:31,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:31,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:45:31,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:31,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:31,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:31,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:31,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:31,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:31,142 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:31,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:31,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:31,191 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:45:31,209 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:31,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:31,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:31,255 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:31,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:31,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:31,303 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:31,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:31,335 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 01:45:31,336 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:45:31,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:32,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:45:32,355 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:32,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:45:32,372 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:45:32,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:45:32,420 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:45:32,441 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:45:32,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:45:32,482 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:45:32,499 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:45:32,516 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 01:45:32,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:45:32,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:34,706 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:45:34,961 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:35,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:35,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:45:35,267 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 01:45:35,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:45:35,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:35,326 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:45:35,345 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 01:45:35,345 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:45:35,346 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:45:35,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:35,673 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:45:35,755 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:35,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:45:35,770 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:45:35,788 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:45:35,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:45:35,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:45:35,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:45:35,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:45:35,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:45:35,881 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 01:45:35,915 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:45:35,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:37,979 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:45:38,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:38,313 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:38,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:45:38,444 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 01:45:38,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:45:38,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:38,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:45:38,516 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 01:45:38,516 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:45:38,519 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:45:38,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:38,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:45:38,887 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:38,887 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:45:38,900 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:45:38,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:45:38,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:45:38,940 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:45:38,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:45:38,966 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:45:38,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:45:38,993 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 01:45:39,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:45:39,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:39,155 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:45:39,285 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:39,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:39,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:45:39,460 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 01:45:39,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:45:39,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:39,489 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:45:39,503 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 01:45:39,503 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:45:39,505 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:45:39,584 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:39,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:45:39,894 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:39,895 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:45:39,910 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:45:39,923 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:45:39,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:45:39,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:45:39,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:45:39,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:45:40,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:45:40,026 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 01:45:40,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:45:40,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:41,872 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:45:42,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:42,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:42,367 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:45:42,387 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 01:45:42,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:45:42,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:42,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:45:42,456 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 01:45:42,457 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:45:42,459 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 01:45:42,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:42,836 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 01:45:42,898 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:42,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 01:45:42,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 01:45:42,923 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 01:45:42,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 01:45:42,951 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 01:45:42,966 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 01:45:42,977 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 01:45:42,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:45:43,005 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 01:45:43,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:45:43,042 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:43,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:45:43,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:43,081 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:45:43,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:45:43,104 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 01:45:43,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:45:43,117 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:45:43,130 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:45:43,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:43,158 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:45:43,178 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:43,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:43,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:43,262 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:43,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:43,301 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:43,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:43,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:45:43,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:43,369 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:45:43,385 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:43,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:43,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:43,431 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:43,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:43,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:43,477 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:43,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:45:43,507 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 01:45:43,507 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:45:43,509 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:45:43,582 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:43,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:45:43,891 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:43,892 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:45:43,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:45:43,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:45:43,936 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:45:43,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:45:43,969 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:45:43,982 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:45:43,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:45:44,436 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:45:44,437 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.929000139236. 2022-02-08 01:45:44,437 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:45:44,438 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:45:44,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:45:44,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:45:44,786 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:45:44,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:45:44,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:45:44,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:45:44,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:45:44,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:45:44,859 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:45:44,872 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:45:44,885 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:45:45,013 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:45:45,017 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:45:45,017 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:45:47,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:45:47,125 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:45:47,130 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:45:47,130 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 5 queries submitted, 0 failed, 2560 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:45:49,130 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:45:49,242 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:45:49,247 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:45:49,247 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:45:51,247 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:45:51,358 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:45:51,362 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:45:51,362 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:45:53,364 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:45:53,483 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:45:53,490 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:45:53,490 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:45:55,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:45:55,608 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:45:55,614 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:45:55,614 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:45:57,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:45:57,740 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:45:57,744 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:45:57,744 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:45:59,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:45:59,855 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:45:59,861 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:45:59,861 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:01,862 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:01,971 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:01,976 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:01,976 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:03,979 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:04,089 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:04,095 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:04,095 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:06,096 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:06,203 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:06,210 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:06,210 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:08,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:08,321 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:08,328 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:08,328 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:10,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:10,438 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:10,444 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:10,446 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:12,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:12,559 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:12,565 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:12,565 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:14,566 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:14,676 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:14,680 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:14,680 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:16,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:16,793 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:16,799 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:16,799 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:18,811 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:18,920 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:18,926 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:18,927 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:20,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:21,039 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:21,045 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:21,045 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:23,045 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:23,151 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:23,157 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:23,157 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:25,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:25,265 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:25,270 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:25,270 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:27,270 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:27,381 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:27,385 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:27,385 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:29,385 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:29,496 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:29,502 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:29,503 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:31,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:31,611 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:31,617 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:31,617 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:33,617 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:33,733 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:33,739 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:33,739 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:35,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:35,848 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:35,855 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:35,855 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:37,857 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:37,967 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:37,974 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:37,974 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:39,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:40,085 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:40,091 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:40,091 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:42,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:42,204 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:42,210 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:42,210 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:44,211 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:44,322 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:44,328 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:44,328 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:46,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:46,434 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:46,440 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:46,441 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:48,443 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:48,549 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:48,555 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:48,555 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 5 queries submitted, 0 failed, 2586 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:46:50,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:50,664 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:50,670 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:50,670 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:52,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:52,779 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:52,785 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:52,785 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:54,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:54,894 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:54,898 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:54,900 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:56,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:57,013 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:57,019 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:57,019 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:46:59,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:46:59,131 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:46:59,138 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:46:59,138 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:01,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:01,250 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:01,255 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:01,255 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:03,255 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:03,362 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:03,368 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:03,368 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:05,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:05,476 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:05,483 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:05,483 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:07,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:07,594 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:07,598 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:07,598 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:09,599 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:09,707 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:09,713 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:09,713 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:11,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:11,819 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:11,825 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:11,825 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:13,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:13,940 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:13,944 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:13,944 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:15,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:16,062 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:16,068 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:16,069 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:18,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:18,177 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:18,184 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:18,184 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:20,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:20,290 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:20,296 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:20,296 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:22,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:22,427 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:22,433 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:22,433 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:24,433 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:24,540 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:24,546 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:24,546 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:26,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:26,655 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:26,661 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:26,661 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:28,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:28,770 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:28,778 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:28,778 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:30,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:30,888 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:30,894 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:30,894 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:32,895 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:33,003 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:33,007 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:33,007 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:35,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:35,124 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:35,128 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:35,128 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:37,130 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:37,237 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:37,243 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:37,243 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:39,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:39,354 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:39,358 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:39,358 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:41,359 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:41,470 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:41,476 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:41,476 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:43,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:43,586 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:43,592 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:43,592 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:45,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:45,707 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:45,713 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:45,713 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:47,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:47,823 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:47,828 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:47,829 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 5 queries submitted, 0 failed, 2609 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:47:49,829 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:49,941 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:49,947 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:49,947 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:51,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:52,062 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:52,069 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:52,069 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:54,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:54,177 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:54,183 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:54,183 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:56,183 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:56,292 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:56,298 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:56,299 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:47:58,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:47:58,411 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:47:58,418 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:47:58,418 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:00,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:00,529 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:00,535 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:00,535 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:02,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:02,644 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:02,648 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:02,650 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:04,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:04,767 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:04,773 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:04,773 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:06,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:06,898 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:06,904 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:06,904 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:08,905 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:09,020 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:09,026 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:09,026 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:11,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:11,151 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:11,155 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:11,155 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:13,155 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:13,275 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:13,280 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:13,280 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:15,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:15,407 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:15,414 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:15,414 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:17,415 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:17,536 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:17,542 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:17,542 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:19,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:19,657 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:19,663 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:19,663 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:21,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:21,790 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:21,796 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:21,796 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:23,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:23,917 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:23,924 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:23,924 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:25,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:26,053 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:26,059 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:26,059 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:28,061 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:28,188 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:28,194 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:28,194 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:30,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:30,321 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:30,325 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:30,325 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:32,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:32,446 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:32,450 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:32,450 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:34,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:34,568 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:34,572 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:34,572 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:36,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:36,680 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:36,684 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:36,684 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:38,686 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:38,801 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:38,805 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:38,805 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:40,806 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:40,927 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:40,934 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:40,934 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:42,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:43,052 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:43,059 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:43,059 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:45,059 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:45,171 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:45,177 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:45,177 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:47,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:47,286 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:47,292 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:47,292 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 5 queries submitted, 0 failed, 2633 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:48:49,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:49,421 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:49,427 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:49,427 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:51,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:51,543 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:51,551 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:51,551 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:53,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:53,674 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:53,680 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:53,680 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:55,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:55,792 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:55,799 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:55,799 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:57,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:48:57,913 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:48:57,918 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:48:57,918 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:48:59,920 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:00,039 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:00,046 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:00,046 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:02,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:02,165 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:02,171 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:02,171 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:04,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:04,289 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:04,295 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:04,295 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:06,295 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:06,411 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:06,414 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:06,415 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:08,415 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:08,542 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:08,546 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:08,546 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:10,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:10,664 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:10,671 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:10,671 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:12,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:12,786 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:12,790 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:12,792 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:14,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:14,907 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:14,913 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:14,913 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:16,913 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:17,022 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:17,026 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:17,026 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:19,028 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:19,140 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:19,147 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:19,147 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:21,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:21,262 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:21,266 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:21,266 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:23,267 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:23,382 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:23,388 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:23,388 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:25,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:25,497 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:25,503 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:25,503 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:27,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:27,627 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:27,631 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:27,632 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:29,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:29,747 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:29,753 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:29,753 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:31,755 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:31,874 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:31,878 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:31,880 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:33,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:33,994 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:34,000 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:34,000 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:36,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:36,115 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:36,121 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:36,121 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:38,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:38,236 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:38,240 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:38,240 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:40,242 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:40,378 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:40,382 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:40,382 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:42,382 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:42,496 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:42,502 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:42,502 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:44,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:44,622 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:44,628 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:44,628 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:46,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:46,757 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:46,763 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:46,763 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:48,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:48,877 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:48,882 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:48,882 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 5 queries submitted, 0 failed, 2657 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:49:50,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:51,000 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:51,006 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:51,006 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:53,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:53,130 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:53,134 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:53,135 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:55,135 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:55,252 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:55,256 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:55,256 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:57,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:57,372 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:57,378 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:57,378 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:49:59,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:49:59,493 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:49:59,500 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:49:59,500 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:01,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:01,608 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:01,615 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:01,615 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:03,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:03,733 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:03,739 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:03,739 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:05,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:05,857 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:05,862 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:05,864 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:07,864 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:07,980 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:07,984 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:07,984 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:09,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:10,108 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:10,114 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:10,114 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:12,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:12,233 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:12,237 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:12,239 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:14,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:14,348 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:14,354 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:14,355 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:16,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:16,474 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:16,480 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:16,480 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:18,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:18,608 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:18,614 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:18,614 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:20,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:20,726 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:20,730 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:20,730 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:22,732 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:22,845 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:22,851 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:22,851 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:24,852 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:24,971 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:24,976 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:24,976 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:26,979 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:27,105 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:27,111 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:27,111 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:29,112 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:29,223 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:29,230 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:29,230 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:31,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:31,345 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:31,351 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:31,351 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:33,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:33,483 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:33,487 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:33,487 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:35,487 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:35,601 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:35,605 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:35,605 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:37,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:37,714 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:37,723 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:37,723 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:39,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:39,851 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:39,855 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:39,855 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:41,857 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:41,973 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:41,979 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:41,979 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:43,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:44,096 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:44,101 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:44,101 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:46,102 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:46,214 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 01:50:46,220 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:46,220 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:48,221 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:48,358 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:48,358 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 5 queries submitted, 0 failed, 2681 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:50:50,359 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:50,519 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:50,519 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:52,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:52,673 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:52,673 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:54,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:54,826 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:54,828 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:56,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:56,970 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:56,970 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:50:58,971 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:50:59,134 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:50:59,134 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:01,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:01,296 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:01,296 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:03,296 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:03,460 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:03,460 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:05,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:05,611 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:05,612 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:07,614 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:07,772 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:07,772 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:09,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:09,930 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:09,930 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:11,933 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:12,081 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:12,081 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:14,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:14,240 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:14,240 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:16,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:16,401 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:16,401 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:18,403 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:18,569 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:18,569 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:20,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:20,713 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:20,713 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:22,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:22,871 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:22,871 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:24,872 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:25,020 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:25,020 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:27,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:27,164 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:27,164 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:29,164 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:29,318 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:29,318 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:31,318 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:31,487 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:31,487 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:33,489 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:33,657 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:33,657 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:35,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:35,868 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:35,868 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:37,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:38,046 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:38,046 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:40,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:40,303 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:40,303 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:42,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:42,471 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:42,471 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:44,471 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:44,622 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:44,622 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:46,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:46,825 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:46,825 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:48,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:49,019 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:49,020 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 5 queries submitted, 0 failed, 2704 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:51:51,020 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:51,223 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:51,224 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:53,224 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:53,384 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:53,385 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:55,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:55,611 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:55,612 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:57,612 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:51:57,783 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:51:57,783 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:51:59,783 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:00,023 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:00,023 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:02,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:02,269 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:02,269 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:04,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:04,437 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:04,437 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:06,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:06,729 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:06,729 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:08,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:08,970 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:08,970 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:10,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:11,131 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:11,131 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:13,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:13,299 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:13,299 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:15,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:15,502 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:15,502 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:17,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:17,657 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:17,658 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:19,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:19,839 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:19,841 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:21,841 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:22,012 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:22,012 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:24,012 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:24,219 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:24,219 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:26,219 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:26,497 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:26,499 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:28,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:28,770 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:28,772 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:30,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:30,963 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:30,963 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:32,966 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:33,223 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:33,223 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:35,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:35,375 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:35,375 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:37,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:37,542 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:37,543 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:39,543 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:39,726 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:39,726 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:41,727 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:41,881 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:41,881 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:43,882 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:44,065 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:44,065 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:46,066 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:46,221 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:46,223 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:48,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:48,377 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:48,378 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 5 queries submitted, 0 failed, 2732 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:52:50,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:50,533 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:50,535 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:52,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:52,690 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:52,690 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:54,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:54,851 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:54,851 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:56,852 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:57,026 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:57,026 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:52:59,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:52:59,263 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:52:59,263 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:01,263 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:01,441 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:01,443 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:03,443 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:03,611 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:03,612 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:05,612 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:05,798 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:05,799 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:07,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:07,977 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:07,979 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:09,979 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:10,157 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:10,157 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:12,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:12,306 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:12,306 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:14,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:14,480 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:14,480 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:16,482 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:16,660 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:16,661 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:18,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:18,818 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:18,818 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:20,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:20,970 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:20,970 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:22,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:23,125 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:23,127 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:25,127 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:25,286 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:25,286 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:27,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:27,438 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:27,440 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:29,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:29,584 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:29,585 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:31,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:31,733 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:31,733 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:33,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:33,881 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:33,881 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:35,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:36,028 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:36,028 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:38,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:38,165 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:38,167 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:40,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:40,322 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:40,322 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:42,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:42,490 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:42,490 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:44,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:44,683 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:44,683 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:46,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:46,845 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:46,846 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:48,854 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:49,029 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:49,029 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 5 queries submitted, 0 failed, 2753 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:53:51,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:51,437 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:51,437 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:53,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:53,592 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:53,592 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:55,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:55,780 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:55,780 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:57,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:53:57,927 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:53:57,927 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:53:59,928 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:00,122 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:00,122 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:54:02,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:02,302 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:02,302 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:54:04,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:04,572 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:04,572 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:54:06,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:06,836 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:06,836 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:54:08,836 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:09,016 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:54:09,032 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 504.595000029. 2022-02-08 01:54:09,032 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:54:09,105 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:54:10,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:54:10,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:54:11,180 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:54:11,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:54:11,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:54:11,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:54:11,260 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:54:11,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:54:11,335 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:54:11,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:54:11,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:12,223 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:54:12,223 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.19199991226. 2022-02-08 01:54:12,223 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:54:12,226 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:54:13,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:54:13,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:54:14,023 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:54:14,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:54:14,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:54:14,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:54:14,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:54:14,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:54:14,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:54:14,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:54:14,263 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:15,105 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:54:15,108 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.88400006294. 2022-02-08 01:54:15,108 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:54:15,111 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:54:16,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:54:16,821 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:54:17,082 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:54:17,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:54:17,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:54:17,140 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:54:17,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:54:17,190 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:54:17,210 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:54:17,278 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:54:17,326 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:18,099 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:54:18,101 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.992000103. 2022-02-08 01:54:18,101 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:54:18,102 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:54:19,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:54:19,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:54:20,000 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:54:20,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:54:20,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:54:20,086 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:54:20,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:54:20,211 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:54:20,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:54:20,339 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:54:20,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:21,023 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:54:21,023 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.92299985886. 2022-02-08 01:54:21,023 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:54:21,026 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:54:22,038 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:54:22,584 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:54:22,759 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:54:22,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:54:22,780 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:54:22,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:54:22,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:54:22,880 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:54:23,020 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:54:23,056 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:54:23,076 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:23,839 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:54:23,839 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.81599998474. 2022-02-08 01:54:23,839 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:54:23,842 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:54:24,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:54:25,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:54:25,700 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:54:25,701 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:54:25,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:54:25,780 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:54:25,816 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:54:25,841 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:54:25,862 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:54:25,900 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:54:25,918 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:26,654 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:54:26,654 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.81500005722. 2022-02-08 01:54:26,654 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:54:26,657 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:54:27,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:54:27,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:54:27,663 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:54:27,663 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:54:27,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:54:27,698 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:54:27,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:54:27,729 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:54:27,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:54:27,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:54:27,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:28,290 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:28,292 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:54:30,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:30,464 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:30,464 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:54:32,466 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:32,611 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:32,611 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:54:34,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:34,755 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:34,756 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:54:36,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:37,028 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:37,029 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:54:39,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:39,174 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:39,174 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:54:41,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:41,325 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:41,325 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:54:43,326 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:43,513 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:43,513 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:54:45,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:45,799 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:45,799 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:54:47,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:47,950 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:47,950 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 5 queries submitted, 0 failed, 2774 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:54:49,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:50,128 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:50,128 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:54:52,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:52,285 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:52,286 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:54:54,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:54,573 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:54,575 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:54:56,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:56,713 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:56,714 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:54:58,714 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:54:58,881 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:54:58,881 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:55:00,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:55:01,032 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:55:01,032 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:55:03,033 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:55:03,177 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:55:03,177 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:55:05,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:55:05,328 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:55:05,329 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:55:07,331 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:55:07,657 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:55:07,657 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 01:55:09,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:55:09,881 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:55:09,881 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 43.2260000706. 2022-02-08 01:55:09,881 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:55:09,884 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:55:11,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:55:11,859 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:55:12,030 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:55:12,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:55:12,045 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:55:12,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:55:12,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:55:12,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:55:12,104 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:55:12,135 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:55:12,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:55:12,677 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:55:12,678 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.79699993134. ############################################################ # # RAM FootPrint: 4110.1640625 # ############################################################ 2022-02-08 01:55:12,982 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 01:55:12,982 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 103 seconds. Reason: Iteration:19 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 2795 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 2821 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:56:55,987 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 01:56:55,987 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 01:57:01,012 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:57:01,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:02,117 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:57:02,256 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:02,257 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:57:02,269 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:57:02,288 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:57:02,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:57:02,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:57:02,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:57:02,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:57:02,368 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:57:02,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:57:02,418 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['17205 memcached\n'], []) 2022-02-08 01:57:02,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 01:57:02,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 01:57:02,461 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['17205 memcached\n'], []) 2022-02-08 01:57:02,461 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:57:02,463 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:57:02,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:03,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:57:03,299 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:03,299 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:57:03,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:57:03,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:57:03,344 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:57:03,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:57:03,369 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:57:03,397 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:57:03,418 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:57:03,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:57:03,482 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['92950 memcached\n'], []) 2022-02-08 01:57:03,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 01:57:03,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 01:57:03,526 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['92950 memcached\n'], []) 2022-02-08 01:57:03,526 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:57:03,528 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:57:03,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:03,829 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:57:03,887 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:03,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:57:03,900 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:57:03,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:57:03,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:57:03,940 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:57:03,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:57:03,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:57:03,979 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:57:03,979 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:57:04,026 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['4847 memcached\n'], []) 2022-02-08 01:57:04,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 01:57:04,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 01:57:04,075 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['4847 memcached\n'], []) 2022-02-08 01:57:04,075 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:57:04,076 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:57:04,155 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:04,380 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:57:04,440 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:04,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:57:04,456 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:57:04,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:57:04,484 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:57:04,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:57:04,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:57:04,530 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:57:04,548 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:57:04,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:57:04,594 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['124359 memcached\n'], []) 2022-02-08 01:57:04,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 01:57:04,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 01:57:04,644 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['124359 memcached\n'], []) 2022-02-08 01:57:04,644 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:57:04,647 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:57:04,737 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:04,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:57:05,033 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:05,033 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:57:05,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:57:05,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:57:05,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:57:05,125 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:57:05,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:57:05,155 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:57:05,171 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:57:05,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:57:05,315 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['31625 memcached\n'], []) 2022-02-08 01:57:05,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 01:57:05,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 01:57:05,362 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['31625 memcached\n'], []) 2022-02-08 01:57:05,362 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:57:05,365 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:57:05,454 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:05,654 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:57:05,710 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:05,711 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:57:05,724 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:57:05,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:57:05,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:57:05,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:57:05,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:57:05,795 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:57:05,806 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:57:05,808 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:57:05,874 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['25270 memcached\n'], []) 2022-02-08 01:57:05,874 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 01:57:05,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 01:57:05,920 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['25270 memcached\n'], []) 2022-02-08 01:57:05,920 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:57:05,921 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:57:06,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:06,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:57:06,282 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:06,282 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:57:06,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:57:06,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:57:06,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:57:06,349 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:57:06,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:57:06,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:57:06,392 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:57:06,392 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:57:06,507 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['66517 memcached\n'], []) 2022-02-08 01:57:06,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 01:57:06,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 01:57:06,559 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['66517 memcached\n'], []) 2022-02-08 01:57:06,559 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:57:06,561 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:57:06,647 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:06,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:57:06,931 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:06,931 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:57:06,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:57:06,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:57:06,989 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:57:07,005 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:57:07,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:57:07,030 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:57:07,043 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:57:07,043 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:57:07,114 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['60300 memcached\n'], []) 2022-02-08 01:57:07,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 01:57:07,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 01:57:07,167 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['60300 memcached\n'], []) 2022-02-08 01:57:07,167 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:57:07,170 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:57:07,263 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:07,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:57:07,532 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:07,532 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:57:07,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:57:07,558 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:57:07,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:57:07,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:57:07,596 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:57:07,609 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:57:07,622 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:57:07,622 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:57:07,667 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['124353 memcached\n'], []) 2022-02-08 01:57:07,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 01:57:07,690 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 01:57:07,710 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['124353 memcached\n'], []) 2022-02-08 01:57:07,710 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:57:07,713 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:57:07,793 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:08,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:57:08,079 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:08,079 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:57:08,094 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:57:08,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:57:08,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:57:08,141 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:57:08,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:57:08,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:57:08,180 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 01:57:08,181 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:57:08,243 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['72978 memcached\n'], []) 2022-02-08 01:57:08,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 01:57:08,270 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 01:57:08,296 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['72978 memcached\n'], []) 2022-02-08 01:57:08,298 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:57:08,299 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 01:57:08,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:08,627 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 01:57:08,691 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:08,693 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 01:57:08,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 01:57:08,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 01:57:08,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 01:57:08,750 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 01:57:08,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 01:57:08,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 01:57:08,790 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:57:08,805 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 01:57:08,835 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:57:08,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:10,444 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:57:10,641 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:10,757 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:10,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:57:10,877 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 01:57:10,877 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:57:10,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:10,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:57:11,000 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 01:57:11,000 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 01:57:11,003 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 01:57:11,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:11,301 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 01:57:11,361 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:11,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 01:57:11,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 01:57:11,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 01:57:11,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 01:57:11,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 01:57:11,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 01:57:11,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 01:57:11,464 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:57:11,480 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 01:57:11,510 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:57:11,525 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:13,413 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:57:13,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:13,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:13,918 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:57:13,936 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 01:57:13,936 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:57:13,948 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:13,994 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:57:14,009 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 01:57:14,010 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 01:57:14,013 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 01:57:14,102 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:14,308 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 01:57:14,369 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:14,369 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 01:57:14,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 01:57:14,397 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 01:57:14,408 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 01:57:14,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 01:57:14,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 01:57:14,448 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 01:57:14,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:57:14,476 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 01:57:14,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:57:14,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:16,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:57:16,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:16,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:16,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:57:16,944 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 01:57:16,944 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:57:16,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:17,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:57:17,101 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 01:57:17,101 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 01:57:17,105 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 01:57:17,193 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:17,403 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 01:57:17,457 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:17,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 01:57:17,469 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 01:57:17,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 01:57:17,492 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 01:57:17,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 01:57:17,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 01:57:17,528 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 01:57:17,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:57:17,552 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 01:57:17,576 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:57:17,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:17,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:57:17,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:17,640 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:17,653 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:57:17,664 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 01:57:17,665 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:57:17,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:17,693 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:57:17,704 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:17,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:57:17,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:17,763 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:17,783 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:17,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:17,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:17,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:17,864 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:17,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:17,903 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:17,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:57:17,933 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:17,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:17,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:17,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:18,007 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:18,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:18,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:18,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:18,085 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 01:57:18,086 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 01:57:18,160 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:18,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 01:57:18,426 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:18,426 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 01:57:18,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 01:57:18,450 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 01:57:18,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 01:57:18,477 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 01:57:18,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 01:57:18,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 01:57:18,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:57:18,529 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 01:57:18,553 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:57:18,566 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:18,697 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:57:18,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:18,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:18,892 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:57:18,907 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 01:57:18,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:57:18,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:18,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:57:18,974 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 01:57:18,974 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 01:57:18,976 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 01:57:19,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:19,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 01:57:19,308 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:19,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 01:57:19,321 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 01:57:19,334 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 01:57:19,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 01:57:19,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 01:57:19,372 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 01:57:19,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 01:57:19,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:57:19,411 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 01:57:19,436 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:57:19,448 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:19,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:57:19,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:19,489 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:19,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:57:19,513 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 01:57:19,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:57:19,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:19,543 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:57:19,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:19,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:57:19,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:19,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:19,650 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:19,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:19,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:19,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:19,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:19,757 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:19,778 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:19,793 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:57:19,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:19,823 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:19,839 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:19,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:19,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:19,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:19,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:19,918 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:19,934 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 01:57:19,936 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 01:57:20,032 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:20,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 01:57:20,313 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:20,313 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 01:57:20,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 01:57:20,339 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 01:57:20,352 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 01:57:20,368 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 01:57:20,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 01:57:20,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 01:57:20,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:57:20,427 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 01:57:20,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:57:20,469 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:22,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:57:22,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:22,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:22,924 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:57:22,937 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 01:57:22,937 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:57:22,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:22,992 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:57:23,006 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 01:57:23,006 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 01:57:23,009 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 01:57:23,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:23,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 01:57:23,392 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:23,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 01:57:23,411 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 01:57:23,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 01:57:23,446 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 01:57:23,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 01:57:23,484 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 01:57:23,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 01:57:23,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:57:23,545 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 01:57:23,573 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:57:23,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:25,479 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:57:25,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:25,818 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:25,959 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:57:25,976 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 01:57:25,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:57:25,992 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:26,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:57:26,061 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 01:57:26,062 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 01:57:26,065 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 01:57:26,150 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:26,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 01:57:26,451 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:26,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 01:57:26,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 01:57:26,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 01:57:26,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 01:57:26,510 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 01:57:26,525 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 01:57:26,539 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 01:57:26,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:57:26,569 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 01:57:26,594 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:57:26,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:26,625 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:57:26,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:26,651 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:26,665 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:57:26,678 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 01:57:26,678 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:57:26,693 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:26,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:57:26,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:26,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:57:26,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:26,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:26,822 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:26,844 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:26,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:26,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:26,911 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:26,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:26,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:26,973 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:57:26,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:27,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:27,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:27,038 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:27,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:27,068 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:27,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:27,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:27,117 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 01:57:27,118 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 01:57:27,197 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:27,410 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 01:57:27,507 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:27,507 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 01:57:27,522 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 01:57:27,533 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 01:57:27,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 01:57:27,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 01:57:27,575 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 01:57:27,588 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 01:57:27,601 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:57:27,615 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 01:57:27,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:57:27,658 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:27,673 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:57:27,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:27,700 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:27,713 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:57:27,727 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 01:57:27,727 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:57:27,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:27,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:57:27,766 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:27,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:57:27,809 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:27,829 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:27,854 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:27,872 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:27,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:27,914 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:27,936 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:27,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:27,974 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:27,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:57:28,010 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:28,040 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:28,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:28,072 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:28,085 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:28,102 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:28,119 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:28,142 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:28,161 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 01:57:28,168 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:57:28,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:28,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:57:28,586 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:28,586 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:57:28,601 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:57:28,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:57:28,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:57:28,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:57:28,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:57:28,670 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:57:28,683 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:57:28,698 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 01:57:28,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:57:28,742 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:30,874 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:57:31,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:31,280 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:31,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:57:31,456 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 01:57:31,456 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:57:31,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:31,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:57:31,523 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 01:57:31,523 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:57:31,525 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 01:57:31,604 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:31,844 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 01:57:31,905 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:31,905 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 01:57:31,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 01:57:31,934 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 01:57:31,947 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 01:57:31,966 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 01:57:31,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 01:57:31,994 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 01:57:32,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:57:32,025 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 01:57:32,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:57:32,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:34,007 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:57:34,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:34,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:34,500 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:57:34,516 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 01:57:34,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:57:34,530 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:34,568 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:57:34,584 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 01:57:34,584 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 01:57:34,585 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:57:34,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:34,882 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:57:34,948 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:34,948 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:57:34,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:57:34,989 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:57:35,003 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:57:35,019 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:57:35,033 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:57:35,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:57:35,063 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:57:35,081 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 01:57:35,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:57:35,124 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:36,193 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:57:36,355 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:36,446 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:36,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:57:36,543 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 01:57:36,545 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:57:36,556 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:36,584 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:57:36,596 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 01:57:36,596 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 01:57:36,598 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 01:57:36,739 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:36,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 01:57:37,049 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:37,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 01:57:37,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 01:57:37,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 01:57:37,099 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 01:57:37,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 01:57:37,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 01:57:37,153 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 01:57:37,168 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:57:37,186 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 01:57:37,213 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:57:37,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:39,214 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:57:39,448 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:39,596 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:39,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:57:39,739 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 01:57:39,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:57:39,752 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:39,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:57:39,799 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 01:57:39,799 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 01:57:39,802 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 01:57:39,877 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:40,089 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 01:57:40,148 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:40,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 01:57:40,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 01:57:40,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 01:57:40,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 01:57:40,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 01:57:40,216 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 01:57:40,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 01:57:40,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 01:57:40,259 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 01:57:40,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 01:57:40,305 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:40,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 01:57:40,334 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:40,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 01:57:40,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 01:57:40,374 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 01:57:40,374 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 01:57:40,388 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 01:57:40,404 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 01:57:40,417 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:40,438 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 01:57:40,461 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:40,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:40,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:40,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:40,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:40,569 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:40,591 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:40,612 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 01:57:40,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:40,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 01:57:40,665 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:40,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:40,696 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:40,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:40,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:40,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:40,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:40,770 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 01:57:40,785 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 01:57:40,785 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:57:40,786 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 01:57:40,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:57:41,112 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 01:57:41,184 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:57:41,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 01:57:41,286 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 01:57:41,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 01:57:41,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 01:57:41,332 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 01:57:41,346 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 01:57:41,364 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 01:57:41,378 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:57:42,211 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:57:42,211 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:57:44,213 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:57:44,408 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:57:44,410 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:57:46,410 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:57:46,664 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:57:46,664 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:57:48,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:57:48,825 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:57:48,825 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 2832 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:57:50,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:57:51,063 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:57:51,063 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:57:53,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:57:53,328 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:57:53,328 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:57:55,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:57:55,526 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:57:55,526 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:57:57,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:57:57,862 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:57:57,862 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:57:59,864 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:00,381 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:00,381 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:58:02,381 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:02,779 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:02,780 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:58:04,780 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:05,059 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:05,059 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:58:07,061 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:07,322 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:07,322 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:58:09,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:09,831 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:09,831 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:58:11,831 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:12,118 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:12,119 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:58:14,119 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:14,447 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:14,447 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:58:16,448 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:16,811 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:16,811 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:58:18,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:19,259 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:19,259 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:58:21,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:21,505 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:21,505 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 01:58:23,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:23,733 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 01:58:23,734 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 42.9499998093. 2022-02-08 01:58:23,734 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 01:58:23,737 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 01:58:23,826 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 01:58:24,145 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 01:58:24,223 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 01:58:24,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 01:58:24,267 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 01:58:24,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 01:58:24,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 01:58:24,336 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 01:58:24,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 01:58:24,382 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 01:58:24,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:24,625 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:24,627 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:58:26,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:26,875 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:26,875 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:58:28,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:29,055 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:29,055 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:58:31,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:31,289 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:31,289 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:58:33,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:33,638 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:33,638 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:58:35,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:36,253 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:36,255 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:58:38,255 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:38,502 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:38,502 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:58:40,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:40,823 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:40,825 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:58:42,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:43,101 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:43,102 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:58:45,102 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:45,394 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:45,394 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:58:47,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:47,684 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:47,684 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 5 queries submitted, 0 failed, 2843 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:58:49,686 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:49,984 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:49,986 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:58:51,986 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:52,293 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:52,293 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:58:54,295 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:54,601 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:54,601 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:58:56,601 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:56,766 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:56,766 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:58:58,767 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:58:58,926 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:58:58,926 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:00,926 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:01,183 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:01,183 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:03,184 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:03,530 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:03,530 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:05,530 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:05,694 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:05,694 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:07,696 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:07,868 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:07,868 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:09,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:10,105 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:10,105 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:12,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:12,299 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:12,299 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:14,301 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:14,476 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:14,476 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:16,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:16,638 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:16,638 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:18,638 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:18,823 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:18,825 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:20,825 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:21,059 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:21,059 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:23,061 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:23,253 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:23,253 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:25,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:25,427 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:25,427 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:27,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:27,608 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:27,608 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:29,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:29,773 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:29,773 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:31,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:32,023 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:32,025 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:34,025 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:34,216 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:34,216 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:36,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:36,397 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:36,398 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:38,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:38,558 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:38,558 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:40,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:40,744 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:40,744 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:42,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:42,908 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:42,908 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:44,910 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:45,095 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:45,095 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:47,096 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:47,276 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:47,276 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 5 queries submitted, 0 failed, 2868 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 01:59:49,276 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:49,460 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:49,460 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:51,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:51,617 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:51,617 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:53,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:53,790 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:53,790 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:55,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:55,950 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:55,950 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 01:59:57,951 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 01:59:58,121 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 01:59:58,122 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:00:00,122 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:00,407 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:00,407 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:00:02,408 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:02,607 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:02,608 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:00:04,608 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:04,786 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:04,786 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:00:06,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:06,963 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:06,963 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:00:08,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:09,118 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:00:09,121 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 105.386000156. 2022-02-08 02:00:09,121 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:00:09,122 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:00:10,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:00:10,785 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:00:10,963 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:00:10,964 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:00:10,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:00:11,028 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:00:11,053 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:00:11,089 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:00:11,138 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:00:11,219 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:00:11,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:11,933 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:00:11,933 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.81200003624. 2022-02-08 02:00:11,934 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:00:11,936 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:00:12,631 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:00:13,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:00:13,371 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:00:13,372 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:00:13,434 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:00:13,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:00:13,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:00:13,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:00:13,556 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:00:13,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:00:13,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:14,417 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:00:14,417 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.48300004005. 2022-02-08 02:00:14,417 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:00:14,418 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:00:15,502 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:00:16,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:00:16,237 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:00:16,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:00:16,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:00:16,280 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:00:16,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:00:16,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:00:16,346 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:00:16,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:00:16,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:17,217 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:00:17,217 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.7990000248. 2022-02-08 02:00:17,217 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:00:17,220 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:00:18,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:00:18,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:00:18,913 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:00:18,913 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:00:18,938 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:00:19,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:00:19,033 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:00:19,062 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:00:19,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:00:19,140 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:00:19,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:19,888 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:00:19,888 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.67000007629. 2022-02-08 02:00:19,888 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:00:19,890 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:00:20,650 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:00:21,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:00:21,428 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:00:21,431 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:00:21,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:00:21,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:00:21,502 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:00:21,532 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:00:21,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:00:21,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:00:21,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:22,266 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:00:22,276 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.38800001144. 2022-02-08 02:00:22,276 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:00:22,279 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:00:23,318 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:00:24,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:00:24,283 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:00:24,283 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:00:24,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:00:24,334 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:00:24,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:00:24,391 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:00:24,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:00:24,467 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:00:24,492 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:25,246 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:00:25,246 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.96900010109. 2022-02-08 02:00:25,246 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:00:25,249 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:00:25,520 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:00:25,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:00:25,872 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:00:25,874 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:00:25,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:00:25,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:00:25,920 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:00:25,938 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:00:25,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:00:25,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:00:25,989 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:26,177 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:26,177 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:00:28,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:28,341 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:28,342 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:00:30,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:30,502 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:30,502 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:00:32,502 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:32,667 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:32,667 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:00:34,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:34,835 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:34,835 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:00:36,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:36,990 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:36,990 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:00:38,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:39,243 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:39,243 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:00:41,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:41,594 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:41,595 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:00:43,595 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:43,744 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:43,746 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:00:45,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:45,901 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:45,901 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:00:47,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:48,069 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:48,069 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 5 queries submitted, 0 failed, 2892 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:00:50,071 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:50,226 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:50,226 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:00:52,227 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:52,398 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:52,398 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:00:54,400 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:54,558 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:54,558 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:00:56,558 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:56,737 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:56,739 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:00:58,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:00:58,914 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:00:58,914 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:01:00,915 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:01:01,170 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:01:01,171 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:01:03,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:01:03,398 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:01:03,400 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:01:05,400 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:01:05,558 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:01:05,559 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:01:07,559 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:01:07,706 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:01:07,707 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:01:09,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:01:09,888 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:01:09,888 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:01:11,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:01:12,042 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:01:12,042 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.135 2022-02-08 02:01:14,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:01:14,196 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:01:14,196 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 48.9490001202. 2022-02-08 02:01:14,196 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:01:14,198 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:01:14,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:01:15,533 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:01:16,453 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:01:16,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:01:16,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:01:16,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:01:16,543 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:01:16,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:01:16,599 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:01:16,645 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:01:16,677 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:01:17,578 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:01:17,578 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.382999897. ############################################################ # # RAM FootPrint: 4138.7578125 # ############################################################ 2022-02-08 02:01:17,749 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 02:01:17,749 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 114 seconds. Reason: Iteration:20 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 2916 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 2942 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:03:11,755 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 02:03:11,755 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 02:03:16,806 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 02:03:17,377 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:17,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 02:03:18,009 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:18,009 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 02:03:18,049 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 02:03:18,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 02:03:18,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 02:03:18,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 02:03:18,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 02:03:18,232 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 02:03:18,250 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:03:18,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 02:03:18,315 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['18646 memcached\n'], []) 2022-02-08 02:03:18,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 02:03:18,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 02:03:18,368 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['18646 memcached\n'], []) 2022-02-08 02:03:18,368 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 02:03:18,369 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:03:18,943 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:19,250 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:03:19,398 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:19,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:03:19,414 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:03:19,434 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:03:19,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:03:19,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:03:19,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:03:19,510 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:03:19,526 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:03:19,526 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 02:03:19,581 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['97213 memcached\n'], []) 2022-02-08 02:03:19,581 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 02:03:19,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 02:03:19,631 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['97213 memcached\n'], []) 2022-02-08 02:03:19,631 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:03:19,634 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:03:19,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:19,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:03:20,015 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:20,015 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:03:20,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:03:20,040 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:03:20,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:03:20,068 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:03:20,081 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:03:20,094 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:03:20,107 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:03:20,107 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 02:03:20,147 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['6288 memcached\n'], []) 2022-02-08 02:03:20,147 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 02:03:20,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 02:03:20,193 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['6288 memcached\n'], []) 2022-02-08 02:03:20,194 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:03:20,196 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:03:20,272 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:20,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:03:20,563 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:20,563 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:03:20,579 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:03:20,596 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:03:20,611 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:03:20,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:03:20,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:03:20,660 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:03:20,674 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:03:20,674 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 02:03:20,723 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['125904 memcached\n'], []) 2022-02-08 02:03:20,723 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 02:03:20,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 02:03:20,782 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['125904 memcached\n'], []) 2022-02-08 02:03:20,782 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:03:20,783 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:03:20,862 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:21,095 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:03:21,161 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:21,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:03:21,176 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:03:21,187 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:03:21,201 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:03:21,219 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:03:21,230 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:03:21,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:03:21,259 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:03:21,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 02:03:21,319 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['33067 memcached\n'], []) 2022-02-08 02:03:21,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 02:03:21,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 02:03:21,375 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['33067 memcached\n'], []) 2022-02-08 02:03:21,375 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:03:21,378 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:03:21,469 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:21,670 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:03:21,724 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:21,724 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:03:21,737 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:03:21,749 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:03:21,760 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:03:21,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:03:21,789 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:03:21,802 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:03:21,815 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:03:21,815 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 02:03:21,857 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['26712 memcached\n'], []) 2022-02-08 02:03:21,858 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 02:03:21,887 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 02:03:21,907 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['26712 memcached\n'], []) 2022-02-08 02:03:21,907 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:03:21,910 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:03:21,979 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:22,183 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:03:22,243 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:22,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:03:22,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:03:22,272 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:03:22,285 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:03:22,301 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:03:22,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:03:22,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:03:22,342 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:03:22,342 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 02:03:22,414 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['67960 memcached\n'], []) 2022-02-08 02:03:22,414 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 02:03:22,438 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 02:03:22,460 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['67960 memcached\n'], []) 2022-02-08 02:03:22,461 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:03:22,463 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:03:22,553 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:22,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:03:22,838 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:22,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:03:22,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:03:22,862 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:03:22,881 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:03:22,897 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:03:22,908 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:03:22,920 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:03:22,933 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:03:22,933 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 02:03:22,974 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['61736 memcached\n'], []) 2022-02-08 02:03:22,974 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 02:03:23,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 02:03:23,030 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['61736 memcached\n'], []) 2022-02-08 02:03:23,030 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:03:23,032 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:03:23,494 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:23,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:03:23,839 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:23,839 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:03:23,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:03:23,885 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:03:23,901 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:03:23,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:03:23,936 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:03:23,963 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:03:23,977 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:03:23,977 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 02:03:24,052 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['126025 memcached\n'], []) 2022-02-08 02:03:24,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 02:03:24,099 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 02:03:24,121 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['126025 memcached\n'], []) 2022-02-08 02:03:24,122 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:03:24,125 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:03:24,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:24,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:03:24,720 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:24,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:03:24,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:03:24,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:03:24,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:03:24,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:03:24,788 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:03:24,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:03:24,812 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:03:24,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:03:24,877 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['74423 memcached\n'], []) 2022-02-08 02:03:24,877 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 02:03:24,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:03:24,924 | infra | DEBUG | Thread-1075 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['74423 memcached\n'], []) 2022-02-08 02:03:24,924 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:03:24,926 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:03:24,997 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:25,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:03:25,288 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:25,288 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:03:25,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:03:25,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:03:25,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:03:25,345 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:03:25,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:03:25,375 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:03:25,394 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:03:25,413 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 02:03:25,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:03:25,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:26,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:03:26,976 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:27,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:27,213 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:03:27,230 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 02:03:27,232 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:03:27,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:27,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:03:27,305 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 02:03:27,305 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:03:27,309 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:03:27,395 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:27,618 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:03:27,680 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:27,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:03:27,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:03:27,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:03:27,720 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:03:27,736 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:03:27,749 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:03:27,760 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:03:27,773 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:03:27,788 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 02:03:27,813 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:03:27,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:29,868 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:03:30,088 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:30,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:30,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:03:30,384 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 02:03:30,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:03:30,398 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:30,453 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:03:30,466 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 02:03:30,466 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:03:30,467 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:03:30,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:30,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:03:30,803 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:30,803 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:03:30,818 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:03:30,829 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:03:30,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:03:30,859 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:03:30,871 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:03:30,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:03:30,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:03:30,911 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 02:03:30,940 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:03:30,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:32,996 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:03:33,209 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:33,346 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:33,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:03:33,483 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 02:03:33,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:03:33,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:33,552 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:03:33,563 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 02:03:33,565 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:03:33,566 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 02:03:33,645 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:33,859 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 02:03:33,914 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:33,914 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 02:03:33,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 02:03:33,938 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 02:03:33,950 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 02:03:33,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 02:03:33,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 02:03:33,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 02:03:34,007 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:03:34,025 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 02:03:34,051 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:03:34,065 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:34,078 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:03:34,091 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:34,104 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:34,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:03:34,128 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 02:03:34,128 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:03:34,140 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:34,155 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:03:34,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:34,194 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:03:34,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:34,243 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:34,267 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:34,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:34,313 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:34,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:34,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:34,392 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:34,414 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:34,431 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:03:34,448 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:34,464 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:34,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:34,496 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:34,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:34,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:34,548 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:34,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:34,582 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 02:03:34,584 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:03:34,678 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:34,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:03:34,957 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:34,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:03:34,970 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:03:34,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:03:34,994 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:03:35,010 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:03:35,023 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:03:35,038 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:03:35,051 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:03:35,065 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 02:03:35,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:03:35,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:35,880 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:03:35,983 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:36,040 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:36,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:03:36,117 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 02:03:36,117 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:03:36,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:36,170 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:03:36,183 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 02:03:36,183 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:03:36,188 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 02:03:36,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:36,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 02:03:36,572 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:36,572 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 02:03:36,588 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 02:03:36,601 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 02:03:36,614 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 02:03:36,630 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 02:03:36,642 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 02:03:36,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 02:03:36,670 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:03:36,684 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 02:03:36,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:03:36,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:36,744 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:03:36,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:36,772 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:36,785 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:03:36,798 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 02:03:36,798 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:03:36,811 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:36,823 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:03:36,836 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:36,858 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:03:36,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:36,907 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:36,931 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:36,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:36,977 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:37,002 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:37,025 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:37,048 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:37,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:37,086 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:03:37,102 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:37,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:37,132 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:37,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:37,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:37,177 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:37,191 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:37,207 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:37,221 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 02:03:37,223 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:03:37,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:37,525 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:03:37,585 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:37,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:03:37,601 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:03:37,615 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:03:37,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:03:37,645 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:03:37,660 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:03:37,673 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:03:37,687 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:03:37,701 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 02:03:37,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:03:37,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:39,780 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:03:40,012 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:40,161 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:40,293 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:03:40,309 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 02:03:40,309 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:03:40,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:40,361 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:03:40,375 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 02:03:40,375 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:03:40,377 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:03:40,454 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:40,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:03:40,752 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:40,752 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:03:40,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:03:40,785 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:03:40,799 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:03:40,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:03:40,832 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:03:40,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:03:40,862 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:03:40,880 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 02:03:40,903 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:03:40,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:42,746 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:03:42,946 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:43,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:43,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:03:43,216 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 02:03:43,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:03:43,232 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:43,273 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:03:43,288 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 02:03:43,289 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:03:43,290 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 02:03:43,365 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:43,585 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 02:03:43,644 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:43,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 02:03:43,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 02:03:43,671 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 02:03:43,686 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 02:03:43,701 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 02:03:43,717 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 02:03:43,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 02:03:43,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:03:43,759 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 02:03:43,786 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:03:43,821 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:43,836 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:03:43,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:43,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:43,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:03:43,891 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 02:03:43,891 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:03:43,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:43,918 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:03:43,933 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:43,954 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:03:43,982 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:44,005 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:44,033 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:44,056 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:44,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:44,105 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:44,127 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:44,150 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:44,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:44,186 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:03:44,203 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:44,217 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:44,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:44,249 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:44,266 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:44,288 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:44,302 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:44,319 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:44,334 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 02:03:44,338 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 02:03:44,421 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:44,645 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 02:03:44,713 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:44,714 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 02:03:44,729 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 02:03:44,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 02:03:44,753 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 02:03:44,769 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 02:03:44,782 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 02:03:44,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 02:03:44,808 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:03:44,823 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 02:03:44,851 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:03:44,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:44,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:03:44,892 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:44,904 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:44,917 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:03:44,930 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 02:03:44,930 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:03:44,943 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:44,956 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:03:44,967 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:44,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:03:45,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:45,036 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:45,059 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:45,082 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:45,108 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:45,131 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:45,154 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:45,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:45,200 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:45,216 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:03:45,232 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:45,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:45,262 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:45,276 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:45,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:45,306 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:45,322 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:45,338 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:45,352 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 02:03:45,355 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 02:03:45,440 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:45,667 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 02:03:45,732 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:45,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 02:03:45,747 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 02:03:45,760 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 02:03:45,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 02:03:45,792 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 02:03:45,805 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 02:03:45,819 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 02:03:45,834 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:03:45,848 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 02:03:45,875 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:03:45,890 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:48,101 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:03:48,344 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:48,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:48,645 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:03:48,661 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 02:03:48,661 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:03:48,676 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:48,710 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:03:48,723 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 02:03:48,724 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 02:03:48,726 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:03:48,796 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:49,017 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:03:49,075 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:49,075 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:03:49,089 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:03:49,102 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:03:49,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 5 queries submitted, 0 failed, 2964 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:03:49,132 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:03:49,145 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:03:49,157 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:03:49,171 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:03:49,187 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 02:03:49,213 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:03:49,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:51,167 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:03:51,390 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:51,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:51,664 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:03:51,678 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 02:03:51,680 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:03:51,693 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:51,726 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:03:51,740 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 02:03:51,740 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:03:51,742 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:03:51,828 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:52,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:03:52,108 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:52,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:03:52,132 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:03:52,148 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:03:52,170 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:03:52,193 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:03:52,209 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:03:52,223 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:03:52,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:03:52,253 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 02:03:52,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:03:52,301 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:53,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:03:53,644 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:53,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:53,829 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:03:53,844 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 02:03:53,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:03:53,857 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:53,888 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:03:53,901 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 02:03:53,901 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:03:53,903 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:03:53,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:54,196 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:03:54,302 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:54,303 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:03:54,318 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:03:54,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:03:54,341 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:03:54,358 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:03:54,374 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:03:54,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:03:54,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:03:54,417 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 02:03:54,443 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:03:54,457 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:56,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:03:56,660 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:56,812 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:56,946 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:03:56,960 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 02:03:56,960 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:03:56,971 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:57,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:03:57,019 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 02:03:57,020 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:03:57,023 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 02:03:57,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:57,329 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 02:03:57,391 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:57,392 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 02:03:57,405 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 02:03:57,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 02:03:57,431 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 02:03:57,456 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 02:03:57,469 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 02:03:57,480 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 02:03:57,493 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:03:57,509 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 02:03:57,533 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:03:57,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:57,565 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:03:57,581 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:57,594 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:03:57,607 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:03:57,619 | test | DEBUG | Thread-1075 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 02:03:57,621 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:03:57,634 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:03:57,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:03:57,660 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:57,681 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:03:57,709 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:57,732 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:57,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:57,779 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:57,811 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:57,842 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:57,865 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:57,890 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:03:57,910 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:57,926 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:03:57,941 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:57,957 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:57,974 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:57,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:58,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:58,022 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:58,038 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:58,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:03:58,069 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 02:03:58,069 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:03:58,071 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 02:03:58,163 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:03:58,380 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 02:03:58,444 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:03:58,446 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 02:03:58,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 02:03:58,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 02:03:58,489 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 02:03:58,505 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 02:03:58,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 02:03:58,532 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 02:03:58,549 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:03:58,693 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:03:58,693 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:00,694 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:00,838 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:00,838 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:02,838 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:03,006 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:03,006 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:05,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:05,244 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:05,244 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:07,246 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:07,756 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:07,756 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:09,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:10,260 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:10,262 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:12,262 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:12,750 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:12,750 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:14,752 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:14,973 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:14,973 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:16,974 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:17,407 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:17,408 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:19,410 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:19,859 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:19,859 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:21,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:22,092 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:22,092 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:24,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:24,279 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:24,279 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:26,289 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:26,533 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:26,535 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:28,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:28,798 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:28,798 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:30,798 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:31,072 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:31,072 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:33,073 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:33,328 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:33,328 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:35,328 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:35,723 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:35,723 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:37,724 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:37,927 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:37,927 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:39,927 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:40,259 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:40,259 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:42,259 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:42,539 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:42,539 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 02:04:44,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:44,767 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 02:04:44,767 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 46.6989998817. 2022-02-08 02:04:44,769 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:04:44,770 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:04:44,848 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:04:45,069 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:04:45,131 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:04:45,132 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:04:45,145 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:04:45,160 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:04:45,174 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:04:45,191 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:04:45,206 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:04:45,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:04:45,234 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:45,513 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:04:45,517 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:45,517 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:04:47,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:47,641 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:04:47,647 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:47,647 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 5 queries submitted, 0 failed, 2987 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:04:49,648 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:49,770 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:04:49,776 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:49,776 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:04:51,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:51,891 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:04:51,897 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:51,897 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:04:53,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:54,010 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:04:54,016 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:54,016 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:04:56,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:56,128 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:04:56,132 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:56,134 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:04:58,134 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:04:58,246 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:04:58,250 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:04:58,252 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:00,252 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:00,378 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:00,384 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:00,384 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:02,384 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:02,502 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:02,506 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:02,506 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:04,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:04,622 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:04,628 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:04,628 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:06,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:06,747 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:06,755 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:06,755 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:08,759 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:08,877 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:08,881 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:08,881 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:10,882 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:11,000 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:11,006 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:11,006 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:13,006 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:13,130 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:13,135 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:13,135 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:15,137 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:15,257 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:15,263 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:15,265 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:17,265 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:17,380 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:17,385 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:17,387 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:19,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:19,503 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:19,507 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:19,507 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:21,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:21,621 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:21,628 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:21,628 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:23,628 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:23,750 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:23,755 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:23,756 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:25,756 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:25,872 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:25,878 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:25,878 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:27,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:27,994 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:28,000 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:28,000 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:30,000 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:30,115 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:30,121 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:30,121 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:32,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:32,237 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:32,243 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:32,243 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:34,244 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:34,365 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:34,369 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:34,369 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:36,374 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:36,483 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:36,489 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:36,489 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:38,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:38,615 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:38,619 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:38,619 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:40,619 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:40,737 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:40,743 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:40,743 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:42,743 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:42,855 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:42,861 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:42,861 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:44,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:44,983 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:44,987 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:44,987 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:46,989 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:47,108 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:47,114 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:47,115 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:49,115 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 3010 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:05:49,234 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:49,240 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:49,240 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:51,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:51,357 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:51,361 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:51,361 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:53,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:53,476 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:53,480 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:53,480 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:55,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:55,607 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:55,611 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:55,611 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:57,612 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:57,724 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:57,730 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:57,730 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:05:59,730 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:05:59,851 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:05:59,855 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:05:59,855 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:01,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:01,973 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:01,979 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:01,979 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:03,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:04,105 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:04,109 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:04,109 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:06,111 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:06,230 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:06,236 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:06,236 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:08,236 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:08,362 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:08,369 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:08,369 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:10,371 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:10,483 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:10,489 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:10,489 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:12,490 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:12,605 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:12,609 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:12,609 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:14,609 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:14,733 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:14,737 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:14,739 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:16,740 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:16,852 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:16,858 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:16,858 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:18,859 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:18,986 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:18,990 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:18,990 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:20,990 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:21,115 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:21,119 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:21,119 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:23,121 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:23,233 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:23,237 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:23,237 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:25,239 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:25,358 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:25,364 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:25,364 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:27,364 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:27,479 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:27,483 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:27,483 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:29,483 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:29,604 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:29,608 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:29,609 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:31,609 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:31,726 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:31,733 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:31,733 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:33,733 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:33,845 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:33,849 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:33,849 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:35,852 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:35,973 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:35,980 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:35,980 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:37,982 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:38,108 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:38,112 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:38,112 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:40,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:40,236 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:40,282 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:40,282 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:42,283 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:42,401 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:42,407 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:42,407 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:44,407 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:44,532 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:44,538 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:44,538 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:46,538 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:46,651 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:46,657 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:46,657 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:48,657 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:48,778 | test | WARNING | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 02:06:48,782 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:48,782 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 5 queries submitted, 0 failed, 3037 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:06:50,783 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:50,937 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:50,938 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:52,940 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:53,091 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:53,091 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:55,092 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:55,240 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:55,240 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:57,240 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:57,385 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:57,387 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:06:59,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:06:59,535 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:06:59,535 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:01,536 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:01,694 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:01,694 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:03,696 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:03,845 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:03,845 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:05,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:05,997 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:05,999 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:07,999 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:08,150 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:08,150 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:10,151 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:10,312 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:10,312 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:12,312 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:12,480 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:12,482 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:14,482 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:14,622 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:14,622 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:16,624 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:16,799 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:16,799 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:18,801 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:19,028 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:19,029 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:21,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:21,180 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:21,180 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:23,180 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:23,351 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:23,351 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:25,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:25,513 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:25,513 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:27,513 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:27,683 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:27,683 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:29,684 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:29,859 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:29,861 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:31,861 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:32,114 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:32,114 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:34,114 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:34,329 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:34,329 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:36,331 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:36,503 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:36,503 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:38,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:38,651 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:38,651 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:40,653 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:40,832 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:40,832 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:42,869 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:43,052 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:43,052 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:45,052 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:45,224 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:45,224 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:47,226 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:47,380 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:47,380 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 5 queries submitted, 0 failed, 3059 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:07:49,380 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:49,528 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:49,528 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:51,529 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:51,878 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:51,878 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:53,878 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:54,095 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:54,096 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:56,098 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:56,256 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:56,256 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:07:58,256 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:07:58,424 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:07:58,424 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:08:00,424 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:00,822 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:08:00,822 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:08:02,823 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:02,973 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:08:02,973 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:08:04,974 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:05,191 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:08:05,193 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:08:07,193 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:07,348 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:08:07,348 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:08:09,348 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:09,517 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:08:09,517 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:08:11,517 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:11,732 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:08:11,733 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:08:13,734 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:13,946 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:08:13,946 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:08:15,946 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:16,101 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:08:16,101 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:08:18,104 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:18,293 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:08:18,295 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:08:20,295 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:20,447 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:08:20,447 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:08:22,448 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:22,605 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:08:22,605 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:08:24,605 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:24,775 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:08:24,775 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:08:26,776 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:26,921 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:08:26,921 | test | INFO | Thread-1075 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 02:08:28,923 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:29,075 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:08:29,075 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 224.306999922. 2022-02-08 02:08:29,075 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:08:29,078 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:08:29,894 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:08:30,295 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:08:30,444 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:08:30,446 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:08:30,459 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:08:30,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:08:30,487 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:08:30,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:08:30,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:08:30,542 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:08:30,555 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:31,052 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:08:31,065 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.98900008202. 2022-02-08 02:08:31,065 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:08:31,068 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:08:31,845 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:08:32,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:08:32,437 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:08:32,437 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:08:32,454 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:08:32,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:08:32,487 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:08:32,506 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:08:32,519 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:08:32,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:08:32,562 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:33,032 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:08:33,032 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.96799993515. 2022-02-08 02:08:33,032 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:08:33,035 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:08:34,220 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:08:34,707 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:08:34,884 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:08:34,884 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:08:34,898 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:08:34,921 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:08:34,936 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:08:34,953 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:08:34,966 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:08:35,026 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:08:35,039 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:35,671 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:08:35,673 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.6400001049. 2022-02-08 02:08:35,673 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:08:35,674 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:08:36,775 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:08:37,233 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:08:37,413 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:08:37,414 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:08:37,427 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:08:37,447 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:08:37,460 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:08:37,476 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:08:37,489 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:08:37,528 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:08:37,540 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:38,586 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:08:38,586 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.91399979591. 2022-02-08 02:08:38,588 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:08:38,588 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:08:39,362 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:08:39,783 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:08:39,979 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:08:39,980 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:08:39,993 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:08:40,016 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:08:40,029 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:08:40,046 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:08:40,059 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:08:40,102 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:08:40,118 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:42,934 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:08:42,934 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.34699988365. 2022-02-08 02:08:42,934 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:08:42,937 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:08:43,855 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:08:44,315 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:08:44,473 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:08:44,473 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:08:44,486 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:08:44,503 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:08:44,516 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:08:44,535 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:08:44,546 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:08:44,578 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:08:44,592 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:45,618 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:08:45,618 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.68400001526. 2022-02-08 02:08:45,618 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:08:45,621 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:08:46,509 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:08:47,055 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:08:47,279 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:08:47,279 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:08:47,292 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:08:47,311 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:08:47,325 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:08:47,351 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:08:47,367 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:08:47,423 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:08:47,470 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:48,068 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:08:48,068 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.44900012016. 2022-02-08 02:08:48,069 | test | DEBUG | Thread-1075 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:08:48,071 | infra | DEBUG | Thread-1075 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:08:48,852 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 3081 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:08:49,253 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:08:49,387 | infra | DEBUG | Thread-1075 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:08:49,387 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:08:49,401 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:08:49,418 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:08:49,430 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:08:49,448 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:08:49,463 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:08:49,484 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:08:49,497 | infra | DEBUG | Thread-1075 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:08:50,865 | infra | DEBUG | Thread-1075 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:08:50,865 | test | INFO | Thread-1075 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.79600000381. 2022-02-08 02:08:50,868 | test | DEBUG | Thread-1075 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L836 2022-02-08 02:08:50,868 | test | DEBUG | Thread-1075 | [common_lib:sleep:24] Sleep 300 seconds. Reason: None 5 queries submitted, 0 failed, 3109 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 3129 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 3156 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 3177 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 3203 passed, 0 rejected, 0 cancelled, 0 timeout ############################################################ # # Step 6: Rebalance Out with Loading of docs # ############################################################ KV nodes in cluster: ['172.23.121.115', '172.23.121.116', '172.23.121.123', '172.23.121.124', '172.23.121.126', '172.23.121.127', '172.23.121.128', '172.23.121.129', '172.23.121.135'] CBAS nodes in cluster: [] INDEX nodes in cluster: ['172.23.121.130', '172.23.121.131', '172.23.121.132', '172.23.121.133', '172.23.121.134'] FTS nodes in cluster: [] QUERY nodes in cluster: ['172.23.121.130', '172.23.121.131', '172.23.121.132', '172.23.121.133', '172.23.121.134'] EVENTING nodes in cluster: [] AVAILABLE nodes for cluster: ['172.23.121.136'] Servers coming in : [] with services: ['kv'] Servers going out : ['172.23.121.115'] NEW AVAILABLE nodes for cluster: ['172.23.121.136', '172.23.121.115'] 2022-02-08 02:13:50,937 | infra | DEBUG | MainThread | [task:__init__:249] Last known rebalance status_id: 42a2376640eb9e8896d1add42fac2734 2022-02-08 02:13:50,944 | infra | INFO | pool-3-thread-9 | [task:start_task:91] Thread 'Rebalance_task_IN=[]_OUT=[172.23.121.115]_1644315230.87' started 2022-02-08 02:13:50,944 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.121.115]_1644315230.87 2022-02-08 02:13:50,971 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/Hospital.py -> SystemTestMagma():L516 2022-02-08 02:13:50,973 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep for 60s for rebalance to start 2022-02-08 02:13:51,016 | test | DEBUG | pool-3-thread-9 | [task:start_rebalance:384] Removing node 172.23.121.115:8091 from cluster 2022-02-08 02:13:51,016 | test | DEBUG | pool-3-thread-9 | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.128,ns_1@172.23.121.115,ns_1@172.23.121.132,ns_1@172.23.121.135,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.134,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123,ns_1@172.23.121.131,ns_1@172.23.121.130,ns_1@172.23.121.133', 'ejectedNodes': u'ns_1@172.23.121.115', 'user': 'Administrator'} 2022-02-08 02:13:51,457 | test | DEBUG | pool-3-thread-9 | [rest_client:rebalance:1459] Rebalance operation started 2022-02-08 02:13:51,467 | test | INFO | pool-3-thread-9 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.121.128 | kv | 7.1.0-2255-enterprise | 67.4176325042 | Cluster node | | 172.23.121.115 | kv | 7.1.0-2255-enterprise | 67.8120525836 | --- OUT ---> | | 172.23.121.132 | index, n1ql | 7.1.0-2255-enterprise | 12.5553529565 | Cluster node | | 172.23.121.135 | kv | 7.1.0-2255-enterprise | 65.7011191573 | Cluster node | | 172.23.120.170 | kv | 7.1.0-2255-enterprise | 66.2364750358 | Cluster node | | 172.23.121.129 | kv | 7.1.0-2255-enterprise | 66.1118335501 | Cluster node | | 172.23.121.124 | kv | 7.1.0-2255-enterprise | 67.4015131751 | Cluster node | | 172.23.121.126 | kv | 7.1.0-2255-enterprise | 66.8360036473 | Cluster node | | 172.23.121.134 | index, n1ql | 7.1.0-2255-enterprise | 13.7537695031 | Cluster node | | 172.23.121.116 | kv | 7.1.0-2255-enterprise | 67.0694864048 | Cluster node | | 172.23.121.127 | kv | 7.1.0-2255-enterprise | 67.484423676 | Cluster node | | 172.23.121.123 | kv | 7.1.0-2255-enterprise | 68.6873943021 | Cluster node | | 172.23.121.131 | index, n1ql | 7.1.0-2255-enterprise | 13.1319678507 | Cluster node | | 172.23.121.130 | index, n1ql | 7.1.0-2255-enterprise | 19.5414415602 | Cluster node | | 172.23.121.133 | index, n1ql | 7.1.0-2255-enterprise | 13.2493768857 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2022-02-08 02:13:51,480 | infra | DEBUG | pool-3-thread-9 | [task:call:316] New rebalance status_id: 22a1f4a5058a699653aab68d1f557cce 2022-02-08 02:13:51,493 | test | DEBUG | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 02:13:51,493 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 02:13:51,563 | infra | DEBUG | pool-3-thread-9 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:13:51,565 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:13:51,566 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:14:01,592 | test | DEBUG | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.76 % 2022-02-08 02:14:01,592 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance - status: running, progress: 0.76 2022-02-08 02:14:01,637 | infra | DEBUG | pool-3-thread-9 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:14:01,640 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:14:01,640 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:14:11,657 | test | DEBUG | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.14 % 2022-02-08 02:14:11,657 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance - status: running, progress: 1.14 2022-02-08 02:14:11,700 | infra | DEBUG | pool-3-thread-9 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:14:11,703 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:14:11,703 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:14:21,733 | test | DEBUG | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.06 % 2022-02-08 02:14:21,733 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance - status: running, progress: 2.06 2022-02-08 02:14:21,779 | infra | DEBUG | pool-3-thread-9 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:14:21,782 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:14:21,782 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:14:31,815 | test | DEBUG | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.86 % 2022-02-08 02:14:31,815 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance - status: running, progress: 3.86 2022-02-08 02:14:31,862 | infra | DEBUG | pool-3-thread-9 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:14:31,865 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:14:31,867 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:14:41,885 | test | DEBUG | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.65 % 2022-02-08 02:14:41,885 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance - status: running, progress: 5.65 2022-02-08 02:14:41,930 | infra | DEBUG | pool-3-thread-9 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:14:41,931 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:14:41,931 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 3228 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:14:50,984 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L766 2022-02-08 02:14:50,984 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 30 seconds. Reason: Let the rebalance begin! 2022-02-08 02:14:51,980 | test | DEBUG | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.29 % 2022-02-08 02:14:51,980 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance - status: running, progress: 6.29 2022-02-08 02:14:52,036 | infra | DEBUG | pool-3-thread-9 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:14:52,039 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:14:52,039 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:15:02,065 | test | DEBUG | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.30 % 2022-02-08 02:15:02,065 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance - status: running, progress: 7.3 2022-02-08 02:15:02,109 | infra | DEBUG | pool-3-thread-9 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:15:02,112 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:15:02,112 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:15:12,140 | test | DEBUG | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.37 % 2022-02-08 02:15:12,141 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance - status: running, progress: 8.37 2022-02-08 02:15:12,188 | infra | DEBUG | pool-3-thread-9 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:15:12,191 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:15:12,193 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:15:21,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.01 % 2022-02-08 02:15:21,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:15:21,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:15:22,211 | test | DEBUG | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.01 % 2022-02-08 02:15:22,211 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance - status: running, progress: 9.01 2022-02-08 02:15:22,240 | infra | DEBUG | pool-3-thread-9 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:15:22,243 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:15:22,243 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:15:31,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.79 % 2022-02-08 02:15:31,042 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:15:31,043 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:15:32,256 | test | DEBUG | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.79 % 2022-02-08 02:15:32,257 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance - status: running, progress: 9.79 2022-02-08 02:15:32,292 | infra | DEBUG | pool-3-thread-9 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:15:32,295 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:15:32,296 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:15:41,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.32 % 2022-02-08 02:15:41,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:15:41,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:15:42,312 | test | DEBUG | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.57 % 2022-02-08 02:15:42,312 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance - status: running, progress: 15.57 2022-02-08 02:15:42,342 | infra | DEBUG | pool-3-thread-9 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:15:42,345 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:15:42,345 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 3250 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:15:51,102 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.22 % 2022-02-08 02:15:51,109 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:15:51,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:15:52,358 | test | DEBUG | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.22 % 2022-02-08 02:15:52,359 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance - status: running, progress: 16.22 2022-02-08 02:15:52,387 | infra | DEBUG | pool-3-thread-9 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:15:52,388 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:15:52,390 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:16:01,127 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.92 % 2022-02-08 02:16:01,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:16:01,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:16:02,404 | test | DEBUG | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.92 % 2022-02-08 02:16:02,404 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance - status: running, progress: 18.92 2022-02-08 02:16:02,434 | infra | DEBUG | pool-3-thread-9 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:16:02,437 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:16:02,437 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:16:11,158 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.92 % 2022-02-08 02:16:11,165 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:16:11,165 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:16:12,448 | test | DEBUG | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.92 % 2022-02-08 02:16:12,448 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance - status: running, progress: 18.92 2022-02-08 02:16:12,479 | infra | DEBUG | pool-3-thread-9 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:16:12,480 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:16:12,482 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:16:21,181 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.24 % 2022-02-08 02:16:21,188 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:16:21,188 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:16:22,494 | test | DEBUG | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.15 % 2022-02-08 02:16:22,494 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance - status: running, progress: 22.15 2022-02-08 02:16:22,523 | infra | DEBUG | pool-3-thread-9 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:16:22,526 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:16:22,526 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:16:31,188 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >21.24% in 70.1839997768 seconds 2022-02-08 02:16:31,200 | test | DEBUG | MainThread | [rest_client:diag_eval:1476] /diag/eval status on 172.23.120.170:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2022-02-08 02:16:31,201 | test | INFO | MainThread | [opd:abort_rebalance:780] Abort rebalance 2022-02-08 02:16:31,203 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:16:31,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:16:31,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:16:31,828 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:16:31,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:16:31,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:16:31,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:16:31,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:16:31,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:16:32,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:16:32,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:16:32,082 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:16:32,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:16:32,173 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['75373 memcached\n'], []) 2022-02-08 02:16:32,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 02:16:32,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:16:32,253 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['75373 memcached\n'], []) 2022-02-08 02:16:32,255 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:16:32,257 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:16:32,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:16:32,545 | test | DEBUG | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 24.34 % 2022-02-08 02:16:32,545 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance - status: running, progress: 24.34 2022-02-08 02:16:32,575 | infra | DEBUG | pool-3-thread-9 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:16:32,578 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:16:32,578 | infra | DEBUG | pool-3-thread-9 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:16:32,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:16:32,713 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:16:32,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:16:32,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:16:32,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:16:32,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:16:32,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:16:32,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:16:32,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:16:32,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:16:32,835 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 02:16:32,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:16:32,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:34,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:16:35,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:35,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:35,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:16:35,391 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 02:16:35,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:16:35,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:35,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:16:35,470 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 02:16:35,471 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:16:35,473 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:16:35,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:16:35,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:16:35,832 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:16:35,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:16:35,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:16:35,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:16:35,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:16:35,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:16:35,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:16:35,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:16:35,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:16:35,940 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 02:16:35,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:16:35,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:38,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:16:38,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:38,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:38,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:16:38,628 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 02:16:38,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:16:38,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:38,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:16:38,703 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 02:16:38,703 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:16:38,706 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:16:38,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:16:38,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:16:39,036 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:16:39,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:16:39,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:16:39,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:16:39,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:16:39,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:16:39,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:16:39,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:16:39,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:16:39,144 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 02:16:39,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:16:39,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:41,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:16:41,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:41,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:41,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:16:41,760 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 02:16:41,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:16:41,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:41,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:16:41,849 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 02:16:41,849 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:16:41,851 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 02:16:41,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:16:42,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 02:16:42,213 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:16:42,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 02:16:42,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 02:16:42,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 02:16:42,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 02:16:42,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 02:16:42,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 02:16:42,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 02:16:42,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:16:42,318 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 02:16:42,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:16:42,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:42,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:16:42,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:42,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:42,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:16:42,451 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 02:16:42,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:16:42,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:42,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:16:42,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:42,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:16:42,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:42,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:42,588 | test | ERROR | pool-3-thread-9 | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'c5d97e16c66bbac4a2e859b4652399d5', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=a19fd2bef52f8bddbe2aa297b8c22641', u'status': u'notRunning'} - rebalance failed 2022-02-08 02:16:42,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:42,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:42,615 | test | INFO | pool-3-thread-9 | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.120.170: 2022-02-08 02:16:42,617 | test | ERROR | pool-3-thread-9 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315393355L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:16:33.355Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW0208 02:04:20.056339 75454 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2022-02-08 02:16:42,617 | test | ERROR | pool-3-thread-9 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315393242L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:16:33.242Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.16937.61>,\n {{wait_seqno_persisted_failed,\n "GleamBookUsers0",254,244786,\n [{\'ns_1@172.23.120.170\',\n {\'EXIT\',\n {socket_closed,\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.120.170\'},\n {if_rebalance,<0.30855.60>,\n {wait_seqno_persisted,254,244786}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,474}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,211}]}]}}}}.\nRebalance Operation Id = abb9a7fad38db963edefb71d8b44ca42'} 2022-02-08 02:16:42,617 | test | ERROR | pool-3-thread-9 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315393191L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:16:33.191Z', u'text': u"Control connection to memcached on 'ns_1@172.23.120.170' disconnected. Check logs for details."} 2022-02-08 02:16:42,617 | test | ERROR | pool-3-thread-9 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315393188L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:16:33.188Z', u'text': u'Worker <0.15541.61> (for action {move,{254,\n [\'ns_1@172.23.121.115\',\n \'ns_1@172.23.121.135\'],\n [\'ns_1@172.23.121.135\',\n \'ns_1@172.23.120.170\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.16937.61>,\n {{wait_seqno_persisted_failed,\n "GleamBookUsers0",\n 254,\n 244786,\n [{\'ns_1@172.23.120.170\',\n {\'EXIT\',\n {socket_closed,\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.120.170\'},\n {if_rebalance,\n <0.30855.60>,\n {wait_seqno_persisted,\n 254,\n 244786}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 474}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 211}]}]}}}'} 2022-02-08 02:16:42,618 | test | ERROR | pool-3-thread-9 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315232020L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:13:52.020Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2022-02-08 02:16:42,618 | test | ERROR | pool-3-thread-9 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315231658L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:13:51.658Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2022-02-08 02:16:42,618 | test | ERROR | pool-3-thread-9 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315231031L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:13:51.031Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.121.128','ns_1@172.23.121.132',\n 'ns_1@172.23.121.135','ns_1@172.23.120.170',\n 'ns_1@172.23.121.129','ns_1@172.23.121.124',\n 'ns_1@172.23.121.126','ns_1@172.23.121.134',\n 'ns_1@172.23.121.116','ns_1@172.23.121.127',\n 'ns_1@172.23.121.123','ns_1@172.23.121.131',\n 'ns_1@172.23.121.130','ns_1@172.23.121.133'], EjectNodes = ['ns_1@172.23.121.115'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = abb9a7fad38db963edefb71d8b44ca42"} 2022-02-08 02:16:42,618 | test | ERROR | pool-3-thread-9 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.135', u'tstamp': 1644314913553L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:08:33.553Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.121.135\' in 126 seconds.'} 2022-02-08 02:16:42,618 | test | ERROR | pool-3-thread-9 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.116', u'tstamp': 1644314908584L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:08:28.584Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.121.116\' in 98 seconds.'} 2022-02-08 02:16:42,618 | test | ERROR | pool-3-thread-9 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.128', u'tstamp': 1644314726312L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:05:26.312Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.121.128\' in 65 seconds.'} 2022-02-08 02:16:42,618 | test | ERROR | pool-3-thread-9 | [task:call:325] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'c5d97e16c66bbac4a2e859b4652399d5', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=a19fd2bef52f8bddbe2aa297b8c22641', u'status': u'notRunning'} - rebalance failed 2022-02-08 02:16:42,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:42,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:42,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:42,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:42,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:42,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:16:42,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:42,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:42,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:42,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:42,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:42,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:42,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:42,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:42,897 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 02:16:42,901 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:16:43,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:16:43,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:16:43,319 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:16:43,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:16:43,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:16:43,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:16:43,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:16:43,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:16:43,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:16:43,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:16:43,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:16:43,423 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 02:16:43,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:16:43,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:44,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:16:44,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:44,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:44,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:16:44,473 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 02:16:44,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:16:44,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:44,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:16:44,542 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 02:16:44,542 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:16:44,543 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 02:16:44,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:16:44,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 02:16:44,878 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:16:44,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 02:16:44,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 02:16:44,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 02:16:44,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 02:16:44,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 02:16:44,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 02:16:44,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 02:16:44,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:16:44,977 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 02:16:45,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:16:45,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:45,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:16:45,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:45,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:45,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:16:45,078 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 02:16:45,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:16:45,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:45,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:16:45,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:45,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:16:45,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:45,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:45,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:45,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:45,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:45,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:45,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:45,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:45,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:45,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:16:45,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:45,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:45,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:45,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:45,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:45,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:45,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:45,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:45,517 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 02:16:45,519 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:16:45,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:16:45,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:16:45,872 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:16:45,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:16:45,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:16:45,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:16:45,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:16:45,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:16:45,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:16:45,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:16:45,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:16:45,983 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 02:16:46,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:16:46,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:48,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:16:48,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:48,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:48,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:16:48,566 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 02:16:48,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:16:48,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:48,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:16:48,625 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 02:16:48,625 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:16:48,628 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:16:48,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:16:48,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:16:48,977 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:16:48,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:16:48,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:16:49,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:16:49,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:16:49,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:16:49,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:16:49,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:16:49,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:16:49,105 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 02:16:49,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 5 queries submitted, 0 failed, 3271 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:16:49,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:51,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:16:51,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:51,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:51,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:16:51,605 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 02:16:51,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:16:51,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:51,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:16:51,687 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 02:16:51,688 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:16:51,690 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 02:16:51,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:16:51,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 02:16:52,043 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:16:52,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 02:16:52,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 02:16:52,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 02:16:52,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 02:16:52,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 02:16:52,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 02:16:52,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 02:16:52,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:16:52,160 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 02:16:52,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:16:52,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:52,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:16:52,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:52,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:52,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:16:52,273 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 02:16:52,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:16:52,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:52,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:16:52,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:52,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:16:52,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:52,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:52,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:52,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:52,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:52,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:52,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:52,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:52,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:52,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:16:52,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:52,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:52,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:52,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:52,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:52,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:52,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:52,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:52,730 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 02:16:52,732 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 02:16:52,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:16:53,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 02:16:53,084 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:16:53,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 02:16:53,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 02:16:53,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 02:16:53,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 02:16:53,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 02:16:53,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 02:16:53,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 02:16:53,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:16:53,196 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 02:16:53,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:16:53,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:53,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:16:53,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:53,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:53,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:16:53,302 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 02:16:53,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:16:53,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:53,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:16:53,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:53,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:16:53,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:53,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:53,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:53,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:53,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:53,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:53,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:53,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:16:53,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:53,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:16:53,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:53,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:53,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:53,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:53,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:53,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:53,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:53,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:16:53,756 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 02:16:53,759 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:16:53,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:16:54,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:16:54,148 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:16:54,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:16:54,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:16:54,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:16:54,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:16:54,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:16:54,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:16:54,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:16:54,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:16:54,255 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 02:16:54,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:16:54,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:56,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:16:56,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:56,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:56,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:16:56,829 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 02:16:56,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:16:56,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:56,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:16:56,894 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 02:16:56,894 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:16:56,897 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:16:57,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:16:57,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:16:58,105 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:16:58,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:16:58,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:16:58,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:16:58,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:16:58,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:16:58,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:16:58,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:16:58,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:16:58,246 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 02:16:58,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:16:58,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:59,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:16:59,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:59,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:16:59,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:16:59,766 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 02:16:59,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:16:59,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:16:59,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:16:59,823 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 02:16:59,823 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:16:59,825 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:16:59,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:17:00,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:17:00,186 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:17:00,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:17:00,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:17:00,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:17:00,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:17:00,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:17:00,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:17:00,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:17:00,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:17:00,301 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 02:17:00,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:17:00,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:17:02,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:17:02,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:17:02,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:17:02,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:17:02,736 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 02:17:02,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:17:02,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:17:02,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:17:02,796 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 02:17:02,796 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:17:02,799 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 02:17:02,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:17:03,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 02:17:03,240 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:17:03,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 02:17:03,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 02:17:03,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 02:17:03,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 02:17:03,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 02:17:03,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 02:17:03,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 02:17:03,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:17:03,351 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 02:17:03,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:17:03,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:17:03,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:17:03,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:17:03,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:17:03,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:17:03,466 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 02:17:03,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:17:03,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:17:03,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:17:03,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:17:03,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:17:03,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:17:03,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:17:03,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:17:03,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:17:03,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:17:03,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:17:03,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:17:03,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:17:03,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:17:03,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:17:03,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:17:03,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:17:03,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:17:03,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:17:03,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:17:03,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:17:03,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:17:03,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:17:03,926 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 02:17:03,931 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L789 2022-02-08 02:17:03,931 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 5 queries submitted, 0 failed, 3295 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:18:03,959 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.120.170:8091 2022-02-08 02:18:03,960 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:18:03,973 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:03,982 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:03,982 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:06,000 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:06,007 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:06,007 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:08,033 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:08,045 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:08,045 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:10,066 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:10,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:10,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:12,092 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:12,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:12,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:14,117 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:14,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:14,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:16,148 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:16,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:16,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:18,174 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:18,181 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:18,181 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:20,200 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:20,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:20,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:22,223 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:22,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:22,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:24,259 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:24,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:24,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:26,288 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:26,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:26,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:28,309 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:28,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:28,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:30,335 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:30,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:30,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:32,358 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:32,367 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:32,367 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:34,381 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:34,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:34,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:36,404 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:36,413 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:36,413 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:38,430 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:38,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:38,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:40,453 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:18:40,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:18:40,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:18:42,474 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.120.170:8091 is running 2022-02-08 02:18:42,487 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.126:8091 2022-02-08 02:18:42,487 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:18:42,502 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.126:8091 is running 2022-02-08 02:18:42,515 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.128:8091 2022-02-08 02:18:42,515 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:18:42,528 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.128:8091 is running 2022-02-08 02:18:42,539 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.133:8091 2022-02-08 02:18:42,539 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:18:42,551 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.133:8091 is running 2022-02-08 02:18:42,565 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.135:8091 2022-02-08 02:18:42,565 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:18:42,578 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.135:8091 is running 2022-02-08 02:18:42,589 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.134:8091 2022-02-08 02:18:42,591 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:18:42,602 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.134:8091 is running 2022-02-08 02:18:42,615 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.127:8091 2022-02-08 02:18:42,615 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:18:42,628 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.127:8091 is running 2022-02-08 02:18:42,648 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.124:8091 2022-02-08 02:18:42,648 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:18:42,661 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.124:8091 is running 2022-02-08 02:18:42,677 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.130:8091 2022-02-08 02:18:42,677 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:18:42,690 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.130:8091 is running 2022-02-08 02:18:42,703 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.132:8091 2022-02-08 02:18:42,706 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:18:42,717 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.132:8091 is running 2022-02-08 02:18:42,743 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.123:8091 2022-02-08 02:18:42,743 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:18:42,762 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.123:8091 is running 2022-02-08 02:18:42,785 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.116:8091 2022-02-08 02:18:42,786 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:18:42,799 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.116:8091 is running 2022-02-08 02:18:42,813 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.129:8091 2022-02-08 02:18:42,813 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:18:42,825 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.129:8091 is running 2022-02-08 02:18:42,841 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.131:8091 2022-02-08 02:18:42,841 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:18:42,858 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.131:8091 is running 2022-02-08 02:18:42,859 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:18:43,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:18:43,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:18:43,862 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:18:43,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:18:43,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:18:43,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:18:43,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:18:43,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:18:43,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:18:43,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:18:43,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:18:44,006 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 02:18:44,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:18:44,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:44,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:18:44,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:44,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:44,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:18:44,680 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 02:18:44,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:18:44,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:44,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:18:44,757 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 02:18:44,759 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:18:44,760 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:18:44,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:18:45,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:18:45,119 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:18:45,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:18:45,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:18:45,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:18:45,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:18:45,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:18:45,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:18:45,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:18:45,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:18:45,232 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 02:18:45,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:18:45,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:45,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:18:45,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:45,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:45,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:18:45,960 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 02:18:45,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:18:45,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:45,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:18:46,005 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 02:18:46,005 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:18:46,007 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:18:46,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:18:46,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:18:46,341 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:18:46,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:18:46,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:18:46,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:18:46,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:18:46,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:18:46,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:18:46,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:18:46,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:18:46,451 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 02:18:46,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:18:46,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:46,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:18:46,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:47,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:47,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:18:47,187 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 02:18:47,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:18:47,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:47,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:18:47,232 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 02:18:47,232 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:18:47,234 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 02:18:47,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:18:47,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 02:18:47,571 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:18:47,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 02:18:47,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 02:18:47,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 02:18:47,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 02:18:47,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 02:18:47,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 02:18:47,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 02:18:47,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:18:47,680 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 02:18:47,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:18:47,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:47,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:18:47,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:47,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:47,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:18:47,782 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 02:18:47,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:18:47,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:47,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:18:47,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:47,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:18:47,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:47,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:47,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:47,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:47,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:47,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:48,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:48,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:48,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:48,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:18:48,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:48,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:48,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:48,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:48,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:48,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:48,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:48,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:48,240 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 02:18:48,243 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:18:48,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:18:48,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:18:48,614 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:18:48,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:18:48,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:18:48,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:18:48,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:18:48,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:18:48,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:18:48,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:18:48,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:18:48,720 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 02:18:48,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:18:48,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:48,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:18:49,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:49,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 3317 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:18:49,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:18:49,184 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 02:18:49,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:18:49,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:49,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:18:49,226 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 02:18:49,226 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:18:49,230 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 02:18:49,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:18:49,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 02:18:49,591 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:18:49,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 02:18:49,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 02:18:49,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 02:18:49,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 02:18:49,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 02:18:49,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 02:18:49,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 02:18:49,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:18:49,691 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 02:18:49,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:18:49,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:49,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:18:49,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:49,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:49,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:18:49,815 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 02:18:49,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:18:49,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:49,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:18:49,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:49,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:18:49,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:49,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:49,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:49,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:50,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:50,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:50,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:50,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:50,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:50,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:18:50,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:50,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:50,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:50,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:50,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:50,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:50,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:50,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:50,273 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 02:18:50,276 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:18:50,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:18:50,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:18:50,614 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:18:50,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:18:50,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:18:50,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:18:50,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:18:50,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:18:50,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:18:50,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:18:50,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:18:50,727 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 02:18:50,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:18:50,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:50,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:18:51,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:51,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:51,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:18:51,522 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 02:18:51,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:18:51,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:51,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:18:51,562 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 02:18:51,562 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:18:51,565 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:18:51,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:18:51,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:18:51,921 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:18:51,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:18:51,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:18:51,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:18:51,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:18:51,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:18:52,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:18:52,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:18:52,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:18:52,066 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 02:18:52,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:18:52,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:52,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:18:52,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:52,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:52,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:18:52,769 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 02:18:52,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:18:52,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:52,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:18:52,819 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 02:18:52,819 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:18:52,821 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 02:18:52,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:18:53,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 02:18:53,181 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:18:53,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 02:18:53,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 02:18:53,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 02:18:53,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 02:18:53,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 02:18:53,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 02:18:53,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 02:18:53,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:18:53,306 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 02:18:53,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:18:53,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:53,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:18:53,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:53,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:53,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:18:53,410 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 02:18:53,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:18:53,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:53,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:18:53,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:53,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:18:53,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:53,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:53,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:53,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:53,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:53,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:53,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:53,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:53,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:53,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:18:53,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:53,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:53,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:53,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:53,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:53,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:53,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:53,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:53,884 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 02:18:53,885 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 02:18:53,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:18:54,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 02:18:54,311 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:18:54,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 02:18:54,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 02:18:54,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 02:18:54,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 02:18:54,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 02:18:54,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 02:18:54,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 02:18:54,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:18:54,421 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 02:18:54,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:18:54,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:54,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:18:54,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:54,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:54,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:18:54,532 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 02:18:54,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:18:54,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:54,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:18:54,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:54,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:18:54,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:54,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:54,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:54,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:54,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:54,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:54,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:54,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:54,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:54,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:18:54,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:54,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:54,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:54,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:54,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:54,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:55,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:55,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:55,033 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 02:18:55,038 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:18:55,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:18:55,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:18:55,391 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:18:55,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:18:55,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:18:55,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:18:55,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:18:55,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:18:55,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:18:55,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:18:55,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:18:55,500 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 02:18:55,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:18:55,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:55,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:18:55,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:56,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:56,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:18:56,226 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 02:18:56,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:18:56,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:56,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:18:56,273 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 02:18:56,273 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:18:56,276 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:18:56,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:18:56,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:18:56,681 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:18:56,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:18:56,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:18:56,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:18:56,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:18:56,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:18:56,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:18:56,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:18:56,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:18:56,805 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 02:18:56,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:18:56,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:56,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:18:57,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:57,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:57,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:18:57,325 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 02:18:57,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:18:57,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:57,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:18:57,368 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 02:18:57,368 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:18:57,371 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:18:57,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:18:57,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:18:57,706 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:18:57,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:18:57,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:18:57,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:18:57,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:18:57,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:18:57,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:18:57,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:18:57,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:18:57,805 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 02:18:57,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:18:57,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:58,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:18:58,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:58,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:58,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:18:58,585 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 02:18:58,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:18:58,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:58,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:18:58,625 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 02:18:58,625 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:18:58,630 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 02:18:58,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:18:58,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 02:18:58,971 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:18:58,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 02:18:58,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 02:18:58,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 02:18:59,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 02:18:59,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 02:18:59,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 02:18:59,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 02:18:59,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:18:59,094 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 02:18:59,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:18:59,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:59,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:18:59,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:59,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:18:59,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:18:59,200 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 02:18:59,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:18:59,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:18:59,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:18:59,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:59,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:18:59,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:59,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:59,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:59,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:59,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:59,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:59,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:59,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:18:59,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:59,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:18:59,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:59,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:59,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:59,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:59,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:59,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:59,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:59,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:18:59,716 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 02:18:59,716 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.121.115]_1644315230.87 2022-02-08 02:18:59,717 | test | INFO | MainThread | [opd:abort_rebalance:808] Restarting Rebalance after killing at 20 2022-02-08 02:18:59,796 | infra | DEBUG | MainThread | [task:__init__:249] Last known rebalance status_id: c5d97e16c66bbac4a2e859b4652399d5 2022-02-08 02:18:59,796 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.121.115]_1644315539.72 2022-02-08 02:18:59,796 | infra | INFO | pool-3-thread-5 | [task:start_task:91] Thread 'Rebalance_task_IN=[]_OUT=[172.23.121.115]_1644315539.72' started 2022-02-08 02:18:59,806 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L813 2022-02-08 02:18:59,806 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2022-02-08 02:18:59,854 | test | DEBUG | pool-3-thread-5 | [task:start_rebalance:384] Removing node 172.23.121.115:8091 from cluster 2022-02-08 02:18:59,854 | test | DEBUG | pool-3-thread-5 | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.128,ns_1@172.23.121.115,ns_1@172.23.121.132,ns_1@172.23.121.135,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.134,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123,ns_1@172.23.121.131,ns_1@172.23.121.130,ns_1@172.23.121.133', 'ejectedNodes': u'ns_1@172.23.121.115', 'user': 'Administrator'} 2022-02-08 02:18:59,930 | test | DEBUG | pool-3-thread-5 | [rest_client:rebalance:1459] Rebalance operation started 2022-02-08 02:18:59,961 | test | INFO | pool-3-thread-5 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.121.128 | kv | 7.1.0-2255-enterprise | 9.0449082859 | Cluster node | | 172.23.121.115 | kv | 7.1.0-2255-enterprise | 1.96201735631 | --- OUT ---> | | 172.23.121.132 | index, n1ql | 7.1.0-2255-enterprise | 2.32499368208 | Cluster node | | 172.23.121.135 | kv | 7.1.0-2255-enterprise | 12.975734355 | Cluster node | | 172.23.120.170 | kv | 7.1.0-2255-enterprise | 17.8630838132 | Cluster node | | 172.23.121.129 | kv | 7.1.0-2255-enterprise | 2.14483976785 | Cluster node | | 172.23.121.124 | kv | 7.1.0-2255-enterprise | 4.04397826362 | Cluster node | | 172.23.121.126 | kv | 7.1.0-2255-enterprise | 4.3697903511 | Cluster node | | 172.23.121.134 | index, n1ql | 7.1.0-2255-enterprise | 3.02992046459 | Cluster node | | 172.23.121.116 | kv | 7.1.0-2255-enterprise | 11.4424676821 | Cluster node | | 172.23.121.127 | kv | 7.1.0-2255-enterprise | 8.51983723296 | Cluster node | | 172.23.121.123 | kv | 7.1.0-2255-enterprise | 1.85395384033 | Cluster node | | 172.23.121.131 | index, n1ql | 7.1.0-2255-enterprise | 1.99318783903 | Cluster node | | 172.23.121.130 | index, n1ql | 7.1.0-2255-enterprise | 3.77334679455 | Cluster node | | 172.23.121.133 | index, n1ql | 7.1.0-2255-enterprise | 3.07517084282 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2022-02-08 02:18:59,977 | infra | DEBUG | pool-3-thread-5 | [task:call:316] New rebalance status_id: 0c8c0b530d75c7529b11f54437eb7a3a 2022-02-08 02:18:59,993 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 02:18:59,993 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 02:19:00,048 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:19:00,051 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:19:00,051 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:19:10,072 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.30 % 2022-02-08 02:19:10,072 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 0.3 2022-02-08 02:19:10,128 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:19:10,134 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:19:10,134 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:19:20,148 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.18 % 2022-02-08 02:19:20,148 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 2.18 2022-02-08 02:19:20,177 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:19:20,180 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:19:20,181 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:19:30,197 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.30 % 2022-02-08 02:19:30,197 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 4.3 2022-02-08 02:19:30,224 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:19:30,226 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:19:30,227 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:19:40,243 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.66 % 2022-02-08 02:19:40,243 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 10.66 2022-02-08 02:19:40,279 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:19:40,282 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:19:40,282 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 3340 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:19:50,302 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.61 % 2022-02-08 02:19:50,302 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 12.61 2022-02-08 02:19:50,346 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:19:50,348 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:19:50,349 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:20:00,367 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.43 % 2022-02-08 02:20:00,367 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 18.43 2022-02-08 02:20:00,395 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:20:00,398 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:20:00,398 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:20:10,421 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.85 % 2022-02-08 02:20:10,421 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 18.85 2022-02-08 02:20:10,473 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:20:10,476 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:20:10,476 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:20:20,493 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.68 % 2022-02-08 02:20:20,493 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 19.68 2022-02-08 02:20:20,543 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:20:20,546 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:20:20,548 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:20:30,565 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.96 % 2022-02-08 02:20:30,565 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 23.96 2022-02-08 02:20:30,611 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:20:30,614 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:20:30,614 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:20:40,634 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 24.60 % 2022-02-08 02:20:40,634 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 24.6 2022-02-08 02:20:40,683 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:20:40,687 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:20:40,687 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 3363 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:20:50,704 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.18 % 2022-02-08 02:20:50,704 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 27.18 2022-02-08 02:20:50,752 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:20:50,755 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:20:50,756 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:20:59,826 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.11 % 2022-02-08 02:20:59,826 | test | INFO | MainThread | [opd:abort_rebalance:814] Rebalance % = 30.11 2022-02-08 02:20:59,839 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.11 % 2022-02-08 02:20:59,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:20:59,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:21:00,767 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.11 % 2022-02-08 02:21:00,769 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 30.11 2022-02-08 02:21:00,799 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:21:00,802 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:21:00,802 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:21:09,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 31.73 % 2022-02-08 02:21:09,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:21:09,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:21:10,818 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 31.73 % 2022-02-08 02:21:10,819 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 31.73 2022-02-08 02:21:10,848 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:21:10,851 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:21:10,851 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:21:19,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.76 % 2022-02-08 02:21:19,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:21:19,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:21:20,868 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.93 % 2022-02-08 02:21:20,868 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 34.93 2022-02-08 02:21:20,914 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:21:20,917 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:21:20,917 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:21:29,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.78 % 2022-02-08 02:21:29,911 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:21:29,911 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:21:30,944 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.24 % 2022-02-08 02:21:30,944 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 36.24 2022-02-08 02:21:30,993 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:21:30,996 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:21:30,996 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:21:39,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.70 % 2022-02-08 02:21:39,937 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:21:39,937 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:21:41,013 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.70 % 2022-02-08 02:21:41,013 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 37.7 2022-02-08 02:21:41,059 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:21:41,062 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:21:41,062 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 3387 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:21:49,982 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.59 % 2022-02-08 02:21:49,989 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:21:49,989 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:21:51,078 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.59 % 2022-02-08 02:21:51,078 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 38.59 2022-02-08 02:21:51,108 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:21:51,111 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:21:51,111 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:22:00,002 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.19 % 2022-02-08 02:22:00,009 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:22:00,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:22:01,124 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.19 % 2022-02-08 02:22:01,124 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 40.19 2022-02-08 02:22:01,154 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:22:01,157 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:22:01,157 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:22:10,010 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >40.19% in 70.1840000153 seconds 2022-02-08 02:22:10,023 | test | DEBUG | MainThread | [rest_client:diag_eval:1476] /diag/eval status on 172.23.120.170:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2022-02-08 02:22:10,023 | test | INFO | MainThread | [opd:abort_rebalance:780] Abort rebalance 2022-02-08 02:22:10,026 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:22:10,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:22:10,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:22:10,377 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:22:10,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:22:10,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:22:10,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:22:10,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:22:10,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:22:10,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:22:10,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:22:10,482 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:22:10,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:22:10,553 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['76934 memcached\n'], []) 2022-02-08 02:22:10,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 02:22:10,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:22:10,605 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['76934 memcached\n'], []) 2022-02-08 02:22:10,605 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:22:10,608 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:22:10,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:22:10,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:22:10,950 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:22:10,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:22:10,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:22:10,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:22:10,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:22:11,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:22:11,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:22:11,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:22:11,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:22:11,061 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 02:22:11,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:22:11,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:11,168 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.53 % 2022-02-08 02:22:11,168 | test | INFO | pool-3-thread-5 | [task:check:414] Rebalance - status: running, progress: 40.53 2022-02-08 02:22:11,198 | infra | DEBUG | pool-3-thread-5 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:22:11,201 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:22:11,201 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:22:11,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:22:11,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:11,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:11,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:22:11,752 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 02:22:11,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:22:11,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:11,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:22:11,796 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 02:22:11,796 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:22:11,799 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:22:11,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:22:12,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:22:12,174 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:22:12,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:22:12,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:22:12,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:22:12,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:22:12,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:22:12,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:22:12,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:22:12,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:22:12,325 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 02:22:12,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:22:12,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:12,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:22:12,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:12,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:13,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:22:13,098 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 02:22:13,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:22:13,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:13,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:22:13,138 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 02:22:13,138 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:22:13,141 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:22:13,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:22:13,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:22:13,483 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:22:13,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:22:13,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:22:13,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:22:13,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:22:13,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:22:13,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:22:13,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:22:13,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:22:13,592 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 02:22:13,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:22:13,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:13,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:22:14,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:14,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:14,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:22:14,331 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 02:22:14,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:22:14,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:14,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:22:14,371 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 02:22:14,371 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:22:14,375 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 02:22:14,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:22:14,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 02:22:14,743 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:22:14,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 02:22:14,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 02:22:14,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 02:22:14,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 02:22:14,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 02:22:14,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 02:22:14,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 02:22:14,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:22:14,842 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 02:22:14,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:22:14,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:14,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:22:14,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:14,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:14,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:22:14,944 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 02:22:14,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:22:14,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:14,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:22:14,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:15,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:22:15,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:15,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:15,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:15,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:15,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:15,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:15,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:15,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:15,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:15,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:22:15,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:15,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:15,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:15,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:15,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:15,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:15,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:15,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:15,430 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 02:22:15,433 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:22:16,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:22:16,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:22:16,763 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:22:16,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:22:16,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:22:16,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:22:16,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:22:16,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:22:16,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:22:16,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:22:16,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:22:16,894 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 02:22:16,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:22:16,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:17,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:22:17,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:17,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:17,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:22:17,326 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 02:22:17,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:22:17,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:17,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:22:17,372 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 02:22:17,374 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:22:17,378 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 02:22:17,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:22:17,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 02:22:17,742 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:22:17,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 02:22:17,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 02:22:17,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 02:22:17,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 02:22:17,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 02:22:17,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 02:22:17,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 02:22:17,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:22:17,846 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 02:22:17,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:22:17,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:17,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:22:17,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:17,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:17,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:22:17,948 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 02:22:17,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:22:17,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:17,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:22:17,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:18,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:22:18,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:18,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:18,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:18,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:18,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:18,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:18,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:18,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:18,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:18,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:22:18,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:18,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:18,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:18,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:18,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:18,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:18,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:18,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:18,443 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 02:22:18,446 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:22:18,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:22:18,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:22:18,785 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:22:18,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:22:18,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:22:18,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:22:18,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:22:18,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:22:18,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:22:18,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:22:18,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:22:18,895 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 02:22:18,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:22:18,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:19,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:22:19,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:19,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:19,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:22:19,716 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 02:22:19,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:22:19,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:19,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:22:19,759 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 02:22:19,759 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:22:19,760 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:22:19,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:22:20,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:22:20,161 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:22:20,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:22:20,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:22:20,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:22:20,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:22:20,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:22:20,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:22:20,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:22:20,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:22:20,295 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 02:22:20,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:22:20,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:20,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:22:20,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:20,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:20,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:22:21,006 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 02:22:21,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:22:21,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:21,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:22:21,053 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 02:22:21,053 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:22:21,056 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 02:22:21,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:22:21,216 | test | ERROR | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'fd78d67188e80820dc2637168daf43f1', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=7a3ada08ba10c5590ba113f988260d9c', u'status': u'notRunning'} - rebalance failed 2022-02-08 02:22:21,246 | test | INFO | pool-3-thread-5 | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.120.170: 2022-02-08 02:22:21,247 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315731407L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:22:11.407Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.20868.62>,\n {{wait_seqno_persisted_failed,\n "GleamBookUsers0",13,252862,\n [{\'ns_1@172.23.120.170\',\n {\'EXIT\',\n {socket_closed,\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.120.170\'},\n {if_rebalance,<0.371.62>,\n {wait_seqno_persisted,13,252862}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,474}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,211}]}]}}}}.\nRebalance Operation Id = 02e8c47b1391d6eeb7d345278f2dbf1f'} 2022-02-08 02:22:21,247 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315731388L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:22:11.388Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW0208 02:17:31.001719 77001 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2022-02-08 02:22:21,247 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315731381L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:22:11.381Z', u'text': u"Control connection to memcached on 'ns_1@172.23.120.170' disconnected. Check logs for details."} 2022-02-08 02:22:21,247 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315731377L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:22:11.377Z', u'text': u'Worker <0.21356.62> (for action {move,{13,\n [\'ns_1@172.23.121.115\',\n \'ns_1@172.23.121.135\'],\n [\'ns_1@172.23.121.135\',\n \'ns_1@172.23.120.170\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.20868.62>,\n {{wait_seqno_persisted_failed,\n "GleamBookUsers0",\n 13,252862,\n [{\'ns_1@172.23.120.170\',\n {\'EXIT\',\n {socket_closed,\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.120.170\'},\n {if_rebalance,\n <0.371.62>,\n {wait_seqno_persisted,\n 13,\n 252862}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 474}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 211}]}]}}}'} 2022-02-08 02:22:21,249 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315540605L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:19:00.605Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2022-02-08 02:22:21,249 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315540290L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:19:00.290Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2022-02-08 02:22:21,249 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315539860L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:18:59.860Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.121.128','ns_1@172.23.121.132',\n 'ns_1@172.23.121.135','ns_1@172.23.120.170',\n 'ns_1@172.23.121.129','ns_1@172.23.121.124',\n 'ns_1@172.23.121.126','ns_1@172.23.121.134',\n 'ns_1@172.23.121.116','ns_1@172.23.121.127',\n 'ns_1@172.23.121.123','ns_1@172.23.121.131',\n 'ns_1@172.23.121.130','ns_1@172.23.121.133'], EjectNodes = ['ns_1@172.23.121.115'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 02e8c47b1391d6eeb7d345278f2dbf1f"} 2022-02-08 02:22:21,249 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315512389L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:18:32.389Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.120.170\' in 19 seconds.'} 2022-02-08 02:22:21,250 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315393355L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:16:33.355Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW0208 02:04:20.056339 75454 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2022-02-08 02:22:21,250 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315393242L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:16:33.242Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.16937.61>,\n {{wait_seqno_persisted_failed,\n "GleamBookUsers0",254,244786,\n [{\'ns_1@172.23.120.170\',\n {\'EXIT\',\n {socket_closed,\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.120.170\'},\n {if_rebalance,<0.30855.60>,\n {wait_seqno_persisted,254,244786}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,474}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,211}]}]}}}}.\nRebalance Operation Id = abb9a7fad38db963edefb71d8b44ca42'} 2022-02-08 02:22:21,250 | test | ERROR | pool-3-thread-5 | [task:call:325] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'fd78d67188e80820dc2637168daf43f1', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=7a3ada08ba10c5590ba113f988260d9c', u'status': u'notRunning'} - rebalance failed 2022-02-08 02:22:21,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 02:22:21,421 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:22:21,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 02:22:21,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 02:22:21,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 02:22:21,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 02:22:21,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 02:22:21,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 02:22:21,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 02:22:21,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:22:21,539 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 02:22:21,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:22:21,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:21,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:22:21,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:21,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:21,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:22:21,654 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 02:22:21,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:22:21,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:21,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:22:21,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:21,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:22:21,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:21,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:21,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:21,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:21,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:21,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:21,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:21,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:21,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:22,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:22:22,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:22,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:22,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:22,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:22,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:22,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:22,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:22,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:22,163 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 02:22:22,164 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 02:22:22,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:22:22,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 02:22:22,539 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:22:22,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 02:22:22,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 02:22:22,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 02:22:22,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 02:22:22,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 02:22:22,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 02:22:22,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 02:22:22,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:22:22,711 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 02:22:22,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:22:22,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:22,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:22:22,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:22,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:22,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:22:22,816 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 02:22:22,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:22:22,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:22,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:22:22,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:22,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:22:22,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:22,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:22,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:22,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:23,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:23,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:23,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:23,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:23,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:23,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:22:23,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:23,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:23,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:23,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:23,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:23,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:23,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:23,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:23,289 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 02:22:23,292 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:22:23,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:22:23,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:22:23,684 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:22:23,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:22:23,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:22:23,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:22:23,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:22:23,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:22:23,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:22:23,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:22:23,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:22:23,798 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 02:22:23,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:22:23,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:24,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:22:24,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:24,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:24,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:22:24,566 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 02:22:24,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:22:24,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:24,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:22:24,615 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 02:22:24,615 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:22:24,618 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:22:25,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:22:25,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:22:25,819 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:22:25,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:22:25,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:22:25,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:22:25,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:22:25,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:22:25,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:22:25,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:22:25,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:22:25,976 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 02:22:26,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:22:26,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:27,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:22:27,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:27,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:27,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:22:27,628 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 02:22:27,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:22:27,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:27,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:22:27,694 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 02:22:27,694 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:22:27,696 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:22:27,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:22:28,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:22:28,072 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:22:28,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:22:28,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:22:28,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:22:28,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:22:28,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:22:28,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:22:28,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:22:28,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:22:28,187 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 02:22:28,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:22:28,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:28,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:22:28,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:28,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:28,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:22:29,007 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 02:22:29,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:22:29,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:29,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:22:29,049 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 02:22:29,049 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:22:29,051 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 02:22:29,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:22:29,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 02:22:29,411 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:22:29,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 02:22:29,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 02:22:29,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 02:22:29,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 02:22:29,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 02:22:29,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 02:22:29,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 02:22:29,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:22:29,546 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 02:22:29,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:22:29,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:29,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:22:29,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:29,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:22:29,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:22:29,661 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 02:22:29,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:22:29,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:22:29,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:22:29,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:29,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:22:29,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:29,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:29,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:29,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:29,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:29,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:29,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:29,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:22:29,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:30,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:22:30,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:30,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:30,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:30,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:30,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:30,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:30,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:30,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:22:30,164 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 02:22:30,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L789 2022-02-08 02:22:30,171 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 5 queries submitted, 0 failed, 3395 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:23:30,210 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.120.170:8091 2022-02-08 02:23:30,210 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:23:30,239 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:23:30,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:23:30,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:23:32,272 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:23:32,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:23:32,280 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:23:34,303 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:23:34,312 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:23:34,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:23:36,328 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:23:36,336 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:23:36,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:23:38,368 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:23:38,378 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:23:38,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:23:40,410 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:23:40,418 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:23:40,418 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:23:42,459 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:23:42,469 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:23:42,469 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:23:44,500 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:23:44,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:23:44,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:23:46,542 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:23:46,549 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:23:46,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:23:48,585 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:23:48,594 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:23:48,594 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 5 queries submitted, 0 failed, 3413 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:23:50,628 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:23:50,635 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:23:50,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:23:52,654 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:23:52,663 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:23:52,663 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:23:54,678 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:23:54,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:23:54,687 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:23:56,717 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:23:56,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:23:56,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:23:58,752 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:23:58,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:23:58,760 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:24:00,778 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:24:00,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:24:00,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:24:02,808 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 02:24:02,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 02:24:02,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 02:24:04,848 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.120.170:8091 is running 2022-02-08 02:24:04,877 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.126:8091 2022-02-08 02:24:04,877 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:24:04,900 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.126:8091 is running 2022-02-08 02:24:04,917 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.128:8091 2022-02-08 02:24:04,918 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:24:04,937 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.128:8091 is running 2022-02-08 02:24:04,950 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.133:8091 2022-02-08 02:24:04,950 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:24:04,961 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.133:8091 is running 2022-02-08 02:24:04,980 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.135:8091 2022-02-08 02:24:04,980 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:24:04,993 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.135:8091 is running 2022-02-08 02:24:05,010 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.134:8091 2022-02-08 02:24:05,010 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:24:05,023 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.134:8091 is running 2022-02-08 02:24:05,040 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.127:8091 2022-02-08 02:24:05,040 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:24:05,055 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.127:8091 is running 2022-02-08 02:24:05,075 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.124:8091 2022-02-08 02:24:05,076 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:24:05,092 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.124:8091 is running 2022-02-08 02:24:05,111 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.130:8091 2022-02-08 02:24:05,111 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:24:05,128 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.130:8091 is running 2022-02-08 02:24:05,144 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.132:8091 2022-02-08 02:24:05,144 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:24:05,158 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.132:8091 is running 2022-02-08 02:24:05,180 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.123:8091 2022-02-08 02:24:05,180 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:24:05,196 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.123:8091 is running 2022-02-08 02:24:05,210 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.116:8091 2022-02-08 02:24:05,210 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:24:05,233 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.116:8091 is running 2022-02-08 02:24:05,249 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.129:8091 2022-02-08 02:24:05,249 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:24:05,265 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.129:8091 is running 2022-02-08 02:24:05,279 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.131:8091 2022-02-08 02:24:05,280 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:24:05,293 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.131:8091 is running 2022-02-08 02:24:05,296 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:24:06,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:24:06,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:24:06,809 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:24:06,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:24:06,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:24:06,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:24:06,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:24:06,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:24:07,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:24:07,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:24:07,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:24:07,255 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 02:24:07,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:24:07,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:07,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:24:07,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:08,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:08,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:24:08,263 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 02:24:08,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:24:08,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:08,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:24:08,367 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 02:24:08,368 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:24:08,369 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:24:09,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:24:09,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:24:09,621 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:24:09,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:24:09,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:24:09,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:24:09,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:24:09,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:24:09,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:24:09,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:24:09,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:24:09,927 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 02:24:09,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:24:09,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:10,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:24:10,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:10,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:10,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:24:10,950 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 02:24:10,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:24:10,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:11,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:24:11,029 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 02:24:11,029 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:24:11,032 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:24:11,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:24:12,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:24:12,194 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:24:12,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:24:12,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:24:12,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:24:12,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:24:12,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:24:12,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:24:12,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:24:12,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:24:12,502 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 02:24:12,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:24:12,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:12,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:24:13,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:13,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:13,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:24:13,453 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 02:24:13,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:24:13,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:13,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:24:13,513 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 02:24:13,513 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:24:13,516 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 02:24:13,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:24:13,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 02:24:13,920 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:24:13,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 02:24:13,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 02:24:13,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 02:24:13,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 02:24:13,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 02:24:14,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 02:24:14,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 02:24:14,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:24:14,059 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 02:24:14,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:24:14,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:14,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:24:14,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:14,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:14,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:24:14,174 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 02:24:14,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:24:14,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:14,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:24:14,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:14,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:24:14,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:14,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:14,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:14,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:14,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:14,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:14,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:14,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:14,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:14,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:24:14,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:14,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:14,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:14,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:14,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:14,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:14,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:14,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:14,723 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 02:24:14,724 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:24:15,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:24:16,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:24:16,553 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:24:16,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:24:16,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:24:16,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:24:16,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:24:16,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:24:16,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:24:16,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:24:16,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:24:16,740 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 02:24:16,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:24:16,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:17,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:24:17,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:17,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:18,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:24:18,036 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 02:24:18,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:24:18,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:18,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:24:18,200 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 02:24:18,200 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:24:18,203 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 02:24:18,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:24:18,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 02:24:18,592 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:24:18,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 02:24:18,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 02:24:18,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 02:24:18,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 02:24:18,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 02:24:18,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 02:24:18,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 02:24:18,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:24:18,736 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 02:24:18,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:24:18,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:18,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:24:18,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:18,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:18,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:24:18,878 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 02:24:18,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:24:18,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:18,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:24:18,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:18,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:24:19,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:19,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:19,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:19,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:19,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:19,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:19,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:19,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:19,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:19,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:24:19,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:19,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:19,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:19,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:19,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:19,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:19,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:19,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:19,415 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 02:24:19,417 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:24:20,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:24:20,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:24:20,601 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:24:20,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:24:20,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:24:20,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:24:20,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:24:20,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:24:20,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:24:20,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:24:20,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:24:20,815 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 02:24:20,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:24:20,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:21,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:24:21,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:21,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:21,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:24:21,924 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 02:24:21,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:24:21,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:22,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:24:22,157 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 02:24:22,157 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:24:22,160 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:24:22,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:24:23,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:24:23,687 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:24:23,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:24:23,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:24:23,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:24:23,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:24:24,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:24:24,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:24:24,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:24:24,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:24:24,467 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 02:24:24,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:24:24,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:24,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:24:25,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:25,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:25,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:24:25,801 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 02:24:25,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:24:25,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:25,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:24:25,999 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 02:24:25,999 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:24:26,002 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 02:24:26,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:24:26,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 02:24:26,614 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:24:26,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 02:24:26,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 02:24:26,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 02:24:26,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 02:24:26,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 02:24:26,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 02:24:26,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 02:24:26,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:24:26,815 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 02:24:26,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:24:26,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:26,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:24:26,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:26,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:26,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:24:26,993 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 02:24:26,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:24:27,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:27,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:24:27,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:27,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:24:27,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:27,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:27,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:27,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:27,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:27,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:27,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:27,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:27,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:27,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:24:27,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:27,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:27,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:27,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:27,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:27,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:27,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:27,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:27,796 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 02:24:27,799 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 02:24:27,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:24:28,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 02:24:28,183 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:24:28,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 02:24:28,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 02:24:28,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 02:24:28,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 02:24:28,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 02:24:28,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 02:24:28,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 02:24:28,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:24:28,322 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 02:24:28,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:24:28,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:28,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:24:28,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:28,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:28,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:24:28,444 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 02:24:28,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:24:28,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:28,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:24:28,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:28,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:24:28,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:28,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:28,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:28,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:28,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:28,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:28,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:28,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:28,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:28,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:24:28,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:28,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:28,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:28,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:28,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:28,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:28,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:28,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:29,007 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 02:24:29,009 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:24:29,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:24:30,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:24:30,756 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:24:30,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:24:30,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:24:30,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:24:31,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:24:31,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:24:31,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:24:31,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:24:31,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:24:31,398 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 02:24:31,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:24:31,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:31,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:24:32,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:32,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:32,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:24:32,780 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 02:24:32,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:24:32,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:32,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:24:33,003 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 02:24:33,003 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:24:33,006 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:24:33,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:24:34,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:24:34,655 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:24:34,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:24:34,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:24:34,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:24:34,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:24:34,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:24:35,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:24:35,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:24:35,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:24:35,148 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 02:24:35,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:24:35,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:36,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:24:36,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:36,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:36,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:24:36,944 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 02:24:36,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:24:36,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:37,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:24:37,201 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 02:24:37,201 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:24:37,204 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:24:37,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:24:38,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:24:38,861 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:24:38,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:24:38,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:24:38,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:24:38,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:24:39,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:24:39,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:24:39,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:24:39,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:24:39,371 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 02:24:39,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:24:39,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:39,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:24:40,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:40,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:40,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:24:40,852 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 02:24:40,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:24:40,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:40,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:24:41,020 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 02:24:41,022 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:24:41,025 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 02:24:41,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:24:41,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 02:24:41,431 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:24:41,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 02:24:41,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 02:24:41,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 02:24:41,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 02:24:41,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 02:24:41,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 02:24:41,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 02:24:41,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:24:41,556 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 02:24:41,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:24:41,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:41,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:24:41,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:41,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:24:41,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:24:41,674 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 02:24:41,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:24:41,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:24:41,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:24:41,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:41,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:24:41,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:41,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:41,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:41,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:41,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:41,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:42,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:42,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:24:42,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:42,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:24:42,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:42,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:42,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:42,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:42,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:42,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:42,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:42,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:24:42,269 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 02:24:42,270 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.121.115]_1644315539.72 2022-02-08 02:24:42,270 | test | INFO | MainThread | [opd:abort_rebalance:808] Restarting Rebalance after killing at 40 2022-02-08 02:24:42,365 | infra | DEBUG | MainThread | [task:__init__:249] Last known rebalance status_id: fd78d67188e80820dc2637168daf43f1 2022-02-08 02:24:42,365 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.121.115]_1644315882.27 2022-02-08 02:24:42,365 | infra | INFO | pool-3-thread-1 | [task:start_task:91] Thread 'Rebalance_task_IN=[]_OUT=[172.23.121.115]_1644315882.27' started 2022-02-08 02:24:42,374 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L813 2022-02-08 02:24:42,375 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2022-02-08 02:24:42,431 | test | DEBUG | pool-3-thread-1 | [task:start_rebalance:384] Removing node 172.23.121.115:8091 from cluster 2022-02-08 02:24:42,431 | test | DEBUG | pool-3-thread-1 | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.128,ns_1@172.23.121.115,ns_1@172.23.121.132,ns_1@172.23.121.135,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.134,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123,ns_1@172.23.121.131,ns_1@172.23.121.130,ns_1@172.23.121.133', 'ejectedNodes': u'ns_1@172.23.121.115', 'user': 'Administrator'} 2022-02-08 02:24:42,582 | test | DEBUG | pool-3-thread-1 | [rest_client:rebalance:1459] Rebalance operation started 2022-02-08 02:24:42,585 | test | INFO | pool-3-thread-1 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.121.128 | kv | 7.1.0-2255-enterprise | 92.4278998857 | Cluster node | | 172.23.121.115 | kv | 7.1.0-2255-enterprise | 48.5218334689 | --- OUT ---> | | 172.23.121.132 | index, n1ql | 7.1.0-2255-enterprise | 13.5292594511 | Cluster node | | 172.23.121.135 | kv | 7.1.0-2255-enterprise | 91.917651544 | Cluster node | | 172.23.120.170 | kv | 7.1.0-2255-enterprise | 92.1279837481 | Cluster node | | 172.23.121.129 | kv | 7.1.0-2255-enterprise | 92.6448170732 | Cluster node | | 172.23.121.124 | kv | 7.1.0-2255-enterprise | 94.8513598988 | Cluster node | | 172.23.121.126 | kv | 7.1.0-2255-enterprise | 93.2285062135 | Cluster node | | 172.23.121.134 | index, n1ql | 7.1.0-2255-enterprise | 14.8701298701 | Cluster node | | 172.23.121.116 | kv | 7.1.0-2255-enterprise | 92.7530723426 | Cluster node | | 172.23.121.127 | kv | 7.1.0-2255-enterprise | 93.7919675662 | Cluster node | | 172.23.121.123 | kv | 7.1.0-2255-enterprise | 95.0050581689 | Cluster node | | 172.23.121.131 | index, n1ql | 7.1.0-2255-enterprise | 12.2291829809 | Cluster node | | 172.23.121.130 | index, n1ql | 7.1.0-2255-enterprise | 33.6502148717 | Cluster node | | 172.23.121.133 | index, n1ql | 7.1.0-2255-enterprise | 12.5256937307 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2022-02-08 02:24:42,631 | infra | DEBUG | pool-3-thread-1 | [task:call:316] New rebalance status_id: 8d63eba732dd0af3d52511e3f356589f 2022-02-08 02:24:42,645 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 02:24:42,647 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 02:24:42,694 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:24:42,697 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:24:42,697 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 3434 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:24:52,717 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.57 % 2022-02-08 02:24:52,717 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 0.57 2022-02-08 02:24:52,767 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:24:52,770 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:24:52,770 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:25:02,809 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.65 % 2022-02-08 02:25:02,809 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 1.65 2022-02-08 02:25:02,855 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:25:02,858 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:25:02,858 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:25:12,891 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.81 % 2022-02-08 02:25:12,891 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 4.81 2022-02-08 02:25:12,941 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:25:12,944 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:25:12,944 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:25:22,967 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.81 % 2022-02-08 02:25:22,967 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 4.81 2022-02-08 02:25:23,016 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:25:23,019 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:25:23,019 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:25:33,036 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.39 % 2022-02-08 02:25:33,036 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 7.39 2022-02-08 02:25:33,066 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:25:33,069 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:25:33,069 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:25:43,088 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.74 % 2022-02-08 02:25:43,088 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 9.74 2022-02-08 02:25:43,140 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:25:43,141 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:25:43,142 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 3461 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:25:53,171 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.14 % 2022-02-08 02:25:53,171 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 15.14 2022-02-08 02:25:53,210 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:25:53,213 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:25:53,213 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:26:03,230 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.80 % 2022-02-08 02:26:03,230 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 16.8 2022-02-08 02:26:03,266 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:26:03,267 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:26:03,267 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:26:13,298 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.44 % 2022-02-08 02:26:13,299 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 17.44 2022-02-08 02:26:13,338 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:26:13,341 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:26:13,342 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:26:23,357 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.91 % 2022-02-08 02:26:23,358 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 21.91 2022-02-08 02:26:23,401 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:26:23,404 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:26:23,404 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:26:33,459 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.78 % 2022-02-08 02:26:33,459 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 22.78 2022-02-08 02:26:33,506 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:26:33,507 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:26:33,509 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:26:42,392 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.92 % 2022-02-08 02:26:42,392 | test | INFO | MainThread | [opd:abort_rebalance:814] Rebalance % = 25.92 2022-02-08 02:26:42,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.92 % 2022-02-08 02:26:42,423 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:26:42,423 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:26:43,535 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.78 % 2022-02-08 02:26:43,536 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 26.78 2022-02-08 02:26:43,576 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:26:43,579 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:26:43,579 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 3486 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:26:52,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.21 % 2022-02-08 02:26:52,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:26:52,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:26:53,595 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.29 % 2022-02-08 02:26:53,595 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 29.29 2022-02-08 02:26:53,631 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:26:53,634 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:26:53,634 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:27:02,471 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.95 % 2022-02-08 02:27:02,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:27:02,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:27:03,651 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.95 % 2022-02-08 02:27:03,653 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 33.95 2022-02-08 02:27:03,701 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:27:03,704 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:27:03,706 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:27:12,500 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.48 % 2022-02-08 02:27:12,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:27:12,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:27:13,721 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.48 % 2022-02-08 02:27:13,723 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 35.48 2022-02-08 02:27:13,753 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:27:13,756 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:27:13,756 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:27:22,525 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.83 % 2022-02-08 02:27:22,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:27:22,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:27:23,769 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.83 % 2022-02-08 02:27:23,769 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 37.83 2022-02-08 02:27:23,802 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:27:23,805 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:27:23,805 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:27:32,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.62 % 2022-02-08 02:27:32,556 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:27:32,556 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:27:33,821 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.62 % 2022-02-08 02:27:33,822 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 39.62 2022-02-08 02:27:33,852 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:27:33,855 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:27:33,855 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:27:42,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.70 % 2022-02-08 02:27:42,576 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:27:42,576 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:27:43,868 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.70 % 2022-02-08 02:27:43,868 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 40.7 2022-02-08 02:27:43,901 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:27:43,904 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:27:43,904 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 3510 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:27:52,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.37 % 2022-02-08 02:27:52,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:27:52,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:27:53,914 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.37 % 2022-02-08 02:27:53,914 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 44.37 2022-02-08 02:27:53,944 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:27:53,947 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:27:53,947 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:28:02,614 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.00 % 2022-02-08 02:28:02,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:28:02,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:28:03,964 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.00 % 2022-02-08 02:28:03,964 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 45.0 2022-02-08 02:28:03,996 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:28:03,999 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:28:03,999 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:28:12,635 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.87 % 2022-02-08 02:28:12,642 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:28:12,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:28:14,013 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.87 % 2022-02-08 02:28:14,015 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 45.87 2022-02-08 02:28:14,046 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:28:14,049 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:28:14,049 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:28:22,658 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.23 % 2022-02-08 02:28:22,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:28:22,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:28:24,066 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.23 % 2022-02-08 02:28:24,068 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 47.23 2022-02-08 02:28:24,105 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:28:24,108 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:28:24,108 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:28:32,696 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.68 % 2022-02-08 02:28:32,703 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:28:32,703 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:28:34,132 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.68 % 2022-02-08 02:28:34,132 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 47.68 2022-02-08 02:28:34,173 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:28:34,174 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:28:34,176 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:28:42,721 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 53.42 % 2022-02-08 02:28:42,729 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:28:42,729 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:28:44,200 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 53.42 % 2022-02-08 02:28:44,200 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 53.42 2022-02-08 02:28:44,263 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:28:44,266 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:28:44,266 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 3535 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:28:52,757 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 55.73 % 2022-02-08 02:28:52,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:28:52,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:28:54,282 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 55.73 % 2022-02-08 02:28:54,282 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 55.73 2022-02-08 02:28:54,325 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:28:54,328 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:28:54,328 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:29:02,795 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 58.54 % 2022-02-08 02:29:02,802 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:29:02,802 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:29:04,339 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 58.54 % 2022-02-08 02:29:04,339 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 58.54 2022-02-08 02:29:04,368 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:29:04,371 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:29:04,371 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:29:12,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 58.98 % 2022-02-08 02:29:12,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:29:12,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:29:14,413 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 58.98 % 2022-02-08 02:29:14,413 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 58.98 2022-02-08 02:29:14,451 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:29:14,454 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:29:14,454 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:29:22,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 61.42 % 2022-02-08 02:29:22,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:29:22,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:29:24,470 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 61.42 % 2022-02-08 02:29:24,470 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 61.42 2022-02-08 02:29:24,502 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:29:24,505 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:29:24,505 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:29:32,869 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >61.42% in 170.476999998 seconds 2022-02-08 02:29:32,882 | test | DEBUG | MainThread | [rest_client:diag_eval:1476] /diag/eval status on 172.23.120.170:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2022-02-08 02:29:32,882 | test | INFO | MainThread | [opd:abort_rebalance:780] Abort rebalance 2022-02-08 02:29:32,884 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:29:33,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:29:34,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:29:34,290 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:29:34,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:29:34,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:29:34,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:29:34,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:29:34,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:29:34,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:29:34,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:29:34,471 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:29:34,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:29:34,517 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 64.00 % 2022-02-08 02:29:34,517 | test | INFO | pool-3-thread-1 | [task:check:414] Rebalance - status: running, progress: 64.0 2022-02-08 02:29:34,556 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['77862 memcached\n'], []) 2022-02-08 02:29:34,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 02:29:34,565 | infra | DEBUG | pool-3-thread-1 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:29:34,569 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:29:34,569 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:29:34,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:29:34,625 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['77862 memcached\n'], []) 2022-02-08 02:29:34,625 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:29:34,628 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:29:34,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:29:34,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:29:35,022 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:29:35,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:29:35,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:29:35,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:29:35,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:29:35,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:29:35,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:29:35,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:29:35,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:29:35,154 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 02:29:35,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:29:35,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:29:37,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:29:38,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:29:38,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:29:38,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:29:38,302 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 02:29:38,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:29:38,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:29:38,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:29:38,378 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 02:29:38,378 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:29:38,381 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:29:39,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:29:39,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:29:39,750 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:29:39,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:29:39,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:29:39,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:29:39,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:29:39,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:29:39,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:29:39,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:29:39,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:29:39,921 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 02:29:39,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:29:39,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:29:42,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:29:42,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:29:42,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:29:42,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:29:42,809 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 02:29:42,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:29:42,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:29:42,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:29:42,881 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 02:29:42,881 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:29:42,882 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:29:43,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:29:44,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:29:44,496 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:29:44,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:29:44,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:29:44,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:29:44,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:29:44,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:29:44,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:29:44,578 | test | ERROR | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'bb2502617484b163934e158347397aac', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=8e42ad53a29dc222c1366a60c438c8cb', u'status': u'notRunning'} - rebalance failed 2022-02-08 02:29:44,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:29:44,605 | test | INFO | pool-3-thread-1 | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.120.170: 2022-02-08 02:29:44,605 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644316176169L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:29:36.169Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW0208 02:23:00.797147 77938 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2022-02-08 02:29:44,605 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644316175665L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:29:35.665Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.6827.64>,\n {{wait_seqno_persisted_failed,\n "GleamBookUsers0",186,254978,\n [{\'ns_1@172.23.120.170\',\n {\'EXIT\',\n {{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,151}]},\n {mc_client_binary,\n wait_for_seqno_persistence,3,\n [{file,"src/mc_client_binary.erl"},\n {line,742}]},\n {ns_memcached,\n \'-wait_for_seqno_persistence/3-fun-0-\',\n 3,\n [{file,"src/ns_memcached.erl"},\n {line,1452}]},\n {ns_memcached,\n \'-perform_very_long_call/3-fun-0-\',2,\n [{file,"src/ns_memcached.erl"},\n {line,337}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,86}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},\n {line,191}]}]},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.120.170\'},\n {if_rebalance,<0.7116.63>,\n {wait_seqno_persisted,186,254978}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,474}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,211}]}]}}}}.\nRebalance Operation Id = 0030c52d09d886a440e6af61e63da67c'} 2022-02-08 02:29:44,607 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644316175642L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:29:35.642Z', u'text': u"Control connection to memcached on 'ns_1@172.23.120.170' disconnected. Check logs for details."} 2022-02-08 02:29:44,607 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644316175631L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:29:35.631Z', u'text': u'Worker <0.6436.64> (for action {move,{186,\n [\'ns_1@172.23.121.115\',\n \'ns_1@172.23.121.127\'],\n [\'ns_1@172.23.121.127\',\n \'ns_1@172.23.120.170\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.6827.64>,\n {{wait_seqno_persisted_failed,\n "GleamBookUsers0",\n 186,254978,\n [{\'ns_1@172.23.120.170\',\n {\'EXIT\',\n {{{badmatch,\n {error,\n closed}},\n [{mc_client_binary,\n cmd_vocal_recv,\n 5,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 151}]},\n {mc_client_binary,\n wait_for_seqno_persistence,\n 3,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 742}]},\n {ns_memcached,\n \'-wait_for_seqno_persistence/3-fun-0-\',\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 1452}]},\n {ns_memcached,\n \'-perform_very_long_call/3-fun-0-\',\n 2,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 337}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',\n 3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 86}]},\n {async,\n \'-async_init/4-fun-1-\',\n 3,\n [{file,\n "src/async.erl"},\n {line,\n 191}]}]},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.120.170\'},\n {if_rebalance,\n <0.7116.63>,\n {wait_seqno_persisted,\n 186,\n 254978}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 474}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 211}]}]}}}'} 2022-02-08 02:29:44,608 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315883598L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:24:43.598Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2022-02-08 02:29:44,608 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315883075L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:24:43.075Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2022-02-08 02:29:44,608 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315882439L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:24:42.439Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.121.128','ns_1@172.23.121.132',\n 'ns_1@172.23.121.135','ns_1@172.23.120.170',\n 'ns_1@172.23.121.129','ns_1@172.23.121.124',\n 'ns_1@172.23.121.126','ns_1@172.23.121.134',\n 'ns_1@172.23.121.116','ns_1@172.23.121.127',\n 'ns_1@172.23.121.123','ns_1@172.23.121.131',\n 'ns_1@172.23.121.130','ns_1@172.23.121.133'], EjectNodes = ['ns_1@172.23.121.115'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 0030c52d09d886a440e6af61e63da67c"} 2022-02-08 02:29:44,608 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315831916L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:23:51.916Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.120.170\' in 34 seconds.'} 2022-02-08 02:29:44,608 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315731407L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:22:11.407Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.20868.62>,\n {{wait_seqno_persisted_failed,\n "GleamBookUsers0",13,252862,\n [{\'ns_1@172.23.120.170\',\n {\'EXIT\',\n {socket_closed,\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.120.170\'},\n {if_rebalance,<0.371.62>,\n {wait_seqno_persisted,13,252862}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,474}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,211}]}]}}}}.\nRebalance Operation Id = 02e8c47b1391d6eeb7d345278f2dbf1f'} 2022-02-08 02:29:44,608 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644315731388L, u'shortText': u'message', u'serverTime': u'2022-02-08T02:22:11.388Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW0208 02:17:31.001719 77001 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2022-02-08 02:29:44,608 | test | ERROR | pool-3-thread-1 | [task:call:325] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'bb2502617484b163934e158347397aac', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=8e42ad53a29dc222c1366a60c438c8cb', u'status': u'notRunning'} - rebalance failed 2022-02-08 02:29:44,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:29:44,635 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 02:29:44,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:29:44,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:29:47,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:29:47,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:29:47,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:29:47,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:29:47,628 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 02:29:47,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:29:47,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:29:47,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:29:47,720 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 02:29:47,721 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:29:47,724 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 02:29:47,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:29:48,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 02:29:48,079 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:29:48,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 02:29:48,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 02:29:48,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 02:29:48,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 02:29:48,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 02:29:48,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 02:29:48,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 02:29:48,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:29:48,194 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 02:29:48,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:29:48,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:29:48,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:29:48,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:29:48,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:29:48,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:29:48,296 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 02:29:48,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:29:48,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:29:48,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:29:48,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:48,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:29:48,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:48,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:48,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:48,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:48,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:48,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:48,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:48,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:48,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:48,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:29:48,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:48,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:48,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:48,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:48,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:48,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:48,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:48,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:48,776 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 02:29:48,778 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 5 queries submitted, 0 failed, 3560 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:29:49,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:29:49,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:29:50,045 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:29:50,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:29:50,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:29:50,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:29:50,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:29:50,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:29:50,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:29:50,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:29:50,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:29:50,173 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 02:29:50,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:29:50,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:29:51,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:29:51,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:29:51,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:29:51,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:29:51,308 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 02:29:51,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:29:51,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:29:51,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:29:51,372 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 02:29:51,372 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:29:51,377 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 02:29:51,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:29:51,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 02:29:51,719 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:29:51,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 02:29:51,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 02:29:51,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 02:29:51,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 02:29:51,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 02:29:51,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 02:29:51,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 02:29:51,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:29:51,819 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 02:29:51,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:29:51,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:29:51,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:29:51,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:29:51,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:29:51,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:29:51,937 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 02:29:51,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:29:51,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:29:51,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:29:51,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:52,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:29:52,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:52,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:52,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:52,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:52,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:52,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:52,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:52,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:29:52,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:52,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:29:52,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:52,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:52,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:52,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:52,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:52,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:52,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:52,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:29:52,447 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 02:29:52,448 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:29:53,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:29:53,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:29:53,742 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:29:53,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:29:53,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:29:53,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:29:53,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:29:53,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:29:53,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:29:53,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:29:53,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:29:53,890 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 02:29:53,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:29:53,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:29:55,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:29:56,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:29:56,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:29:56,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:29:56,549 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 02:29:56,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:29:56,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:29:56,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:29:56,602 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 02:29:56,604 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:29:56,605 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:29:57,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:29:57,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:29:58,105 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:29:58,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:29:58,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:29:58,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:29:58,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:29:58,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:29:58,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:29:58,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:29:58,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:29:58,275 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 02:29:58,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:29:58,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:30:00,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:30:00,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:30:00,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:30:00,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:30:00,940 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 02:30:00,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:30:00,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:30:01,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:30:01,017 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 02:30:01,019 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:30:01,022 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 02:30:01,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:30:01,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 02:30:01,374 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:30:01,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 02:30:01,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 02:30:01,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 02:30:01,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 02:30:01,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 02:30:01,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 02:30:01,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 02:30:01,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:30:01,486 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 02:30:01,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:30:01,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:30:01,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:30:01,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:30:01,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:30:01,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:30:01,596 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 02:30:01,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:30:01,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:30:01,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:30:01,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:01,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:30:01,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:01,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:01,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:01,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:01,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:01,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:01,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:01,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:01,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:01,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:30:01,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:01,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:01,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:01,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:02,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:02,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:02,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:02,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:02,069 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 02:30:02,072 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 02:30:02,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:30:02,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 02:30:02,413 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:30:02,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 02:30:02,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 02:30:02,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 02:30:02,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 02:30:02,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 02:30:02,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 02:30:02,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 02:30:02,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:30:02,520 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 02:30:02,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:30:02,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:30:02,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:30:02,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:30:02,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:30:02,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:30:02,628 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 02:30:02,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:30:02,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:30:02,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:30:02,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:02,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:30:02,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:02,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:02,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:02,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:02,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:02,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:02,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:02,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:02,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:03,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:30:03,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:03,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:03,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:03,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:03,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:03,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:03,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:03,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:03,138 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 02:30:03,140 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:30:03,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:30:04,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:30:04,681 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:30:04,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:30:04,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:30:04,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:30:04,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:30:04,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:30:04,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:30:04,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:30:04,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:30:04,832 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 02:30:04,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:30:04,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:30:07,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:30:07,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:30:07,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:30:07,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:30:07,592 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 02:30:07,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:30:07,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:30:07,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:30:07,660 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 02:30:07,660 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:30:07,661 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:30:08,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:30:08,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:30:08,529 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:30:08,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:30:08,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:30:08,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:30:08,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:30:08,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:30:08,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:30:08,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:30:08,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:30:08,667 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 02:30:08,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:30:08,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:30:08,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:30:09,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:30:09,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:30:09,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:30:09,213 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 02:30:09,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:30:09,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:30:09,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:30:09,253 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 02:30:09,253 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:30:09,256 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:30:10,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:30:10,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:30:11,094 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:30:11,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:30:11,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:30:11,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:30:11,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:30:11,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:30:11,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:30:11,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:30:11,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:30:11,223 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 02:30:11,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:30:11,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:30:13,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:30:13,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:30:13,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:30:14,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:30:14,138 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 02:30:14,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:30:14,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:30:14,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:30:14,196 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 02:30:14,197 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:30:14,198 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 02:30:14,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:30:14,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 02:30:14,539 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:30:14,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 02:30:14,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 02:30:14,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 02:30:14,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 02:30:14,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 02:30:14,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 02:30:14,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 02:30:14,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:30:14,648 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 02:30:14,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:30:14,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:30:14,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:30:14,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:30:14,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:30:14,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:30:14,757 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 02:30:14,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:30:14,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:30:14,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:30:14,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:14,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:30:14,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:14,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:14,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:14,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:14,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:15,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:15,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:15,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:30:15,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:15,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:30:15,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:15,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:15,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:15,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:15,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:15,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:15,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:15,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:30:15,233 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 02:30:15,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L789 2022-02-08 02:30:15,240 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 5 queries submitted, 0 failed, 3576 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:31:15,263 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.120.170:8091 2022-02-08 02:31:15,263 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:31:15,279 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.120.170:8091 is running 2022-02-08 02:31:15,292 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.126:8091 2022-02-08 02:31:15,293 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:31:15,308 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.126:8091 is running 2022-02-08 02:31:15,325 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.128:8091 2022-02-08 02:31:15,325 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:31:15,341 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.128:8091 is running 2022-02-08 02:31:15,355 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.133:8091 2022-02-08 02:31:15,355 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:31:15,368 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.133:8091 is running 2022-02-08 02:31:15,391 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.135:8091 2022-02-08 02:31:15,391 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:31:15,405 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.135:8091 is running 2022-02-08 02:31:15,420 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.134:8091 2022-02-08 02:31:15,420 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:31:15,433 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.134:8091 is running 2022-02-08 02:31:15,448 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.127:8091 2022-02-08 02:31:15,448 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:31:15,463 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.127:8091 is running 2022-02-08 02:31:15,482 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.124:8091 2022-02-08 02:31:15,482 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:31:15,496 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.124:8091 is running 2022-02-08 02:31:15,513 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.130:8091 2022-02-08 02:31:15,513 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:31:15,529 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.130:8091 is running 2022-02-08 02:31:15,546 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.132:8091 2022-02-08 02:31:15,548 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:31:15,559 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.132:8091 is running 2022-02-08 02:31:15,575 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.123:8091 2022-02-08 02:31:15,575 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:31:15,588 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.123:8091 is running 2022-02-08 02:31:15,601 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.116:8091 2022-02-08 02:31:15,601 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:31:15,618 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.116:8091 is running 2022-02-08 02:31:15,634 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.129:8091 2022-02-08 02:31:15,634 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:31:15,653 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.129:8091 is running 2022-02-08 02:31:15,665 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.131:8091 2022-02-08 02:31:15,667 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 02:31:15,677 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.131:8091 is running 2022-02-08 02:31:15,680 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:31:16,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:31:16,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:31:16,644 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:31:16,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:31:16,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:31:16,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:31:16,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:31:16,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:31:16,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:31:16,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:31:16,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:31:16,821 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 02:31:16,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:31:16,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:17,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:31:17,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:17,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:17,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:31:17,631 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 02:31:17,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:31:17,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:17,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:31:17,707 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 02:31:17,709 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:31:17,710 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:31:17,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:31:18,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:31:18,164 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:31:18,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:31:18,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:31:18,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:31:18,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:31:18,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:31:18,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:31:18,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:31:18,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:31:18,299 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 02:31:18,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:31:18,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:18,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:31:18,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:18,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:19,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:31:19,141 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 02:31:19,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:31:19,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:19,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:31:19,193 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 02:31:19,193 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:31:19,196 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:31:19,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:31:19,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:31:19,578 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:31:19,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:31:19,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:31:19,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:31:19,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:31:19,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:31:19,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:31:19,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:31:19,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:31:19,694 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 02:31:19,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:31:19,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:19,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:31:20,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:20,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:20,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:31:20,565 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 02:31:20,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:31:20,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:20,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:31:20,609 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 02:31:20,609 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:31:20,611 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 02:31:20,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:31:20,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 02:31:20,990 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:31:20,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 02:31:21,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 02:31:21,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 02:31:21,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 02:31:21,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 02:31:21,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 02:31:21,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 02:31:21,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:31:21,111 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 02:31:21,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:31:21,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:21,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:31:21,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:21,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:21,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:31:21,224 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 02:31:21,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:31:21,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:21,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:31:21,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:21,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:31:21,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:21,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:21,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:21,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:21,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:21,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:21,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:21,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:21,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:21,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:31:21,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:21,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:21,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:21,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:21,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:21,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:21,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:21,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:21,744 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 02:31:21,746 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:31:22,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:31:22,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:31:23,121 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:31:23,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:31:23,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:31:23,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:31:23,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:31:23,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:31:23,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:31:23,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:31:23,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:31:23,493 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 02:31:23,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:31:23,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:23,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:31:23,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:24,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:24,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:31:24,134 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 02:31:24,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:31:24,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:24,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:31:24,256 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 02:31:24,257 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:31:24,259 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 02:31:24,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:31:24,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 02:31:24,640 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:31:24,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 02:31:24,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 02:31:24,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 02:31:24,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 02:31:24,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 02:31:24,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 02:31:24,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 02:31:24,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:31:24,766 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 02:31:24,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:31:24,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:24,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:31:24,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:24,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:24,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:31:24,888 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 02:31:24,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:31:24,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:24,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:31:24,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:24,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:31:24,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:25,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:25,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:25,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:25,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:25,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:25,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:25,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:25,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:25,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:31:25,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:25,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:25,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:25,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:25,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:25,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:25,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:25,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:25,415 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 02:31:25,417 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:31:25,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:31:25,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:31:25,825 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:31:25,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:31:25,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:31:25,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:31:25,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:31:25,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:31:25,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:31:25,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:31:25,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:31:25,997 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 02:31:26,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:31:26,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:26,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:31:26,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:26,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:26,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:31:26,956 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 02:31:26,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:31:26,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:26,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:31:27,019 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 02:31:27,019 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:31:27,023 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:31:27,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:31:27,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:31:27,480 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:31:27,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:31:27,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:31:27,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:31:27,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:31:27,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:31:27,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:31:27,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:31:27,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:31:27,676 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 02:31:27,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:31:27,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:27,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:31:28,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:28,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:28,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:31:28,525 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 02:31:28,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:31:28,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:28,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:31:28,572 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 02:31:28,572 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:31:28,575 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 02:31:28,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:31:28,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 02:31:29,032 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:31:29,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 02:31:29,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 02:31:29,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 02:31:29,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 02:31:29,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 02:31:29,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 02:31:29,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 02:31:29,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:31:29,171 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 02:31:29,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:31:29,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:29,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:31:29,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:29,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:29,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:31:29,298 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 02:31:29,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:31:29,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:29,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:31:29,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:29,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:31:29,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:29,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:29,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:29,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:29,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:29,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:29,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:29,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:29,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:29,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:31:29,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:29,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:29,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:29,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:29,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:29,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:29,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:29,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:29,894 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 02:31:29,898 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 02:31:29,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:31:30,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 02:31:30,278 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:31:30,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 02:31:30,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 02:31:30,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 02:31:30,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 02:31:30,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 02:31:30,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 02:31:30,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 02:31:30,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:31:30,390 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 02:31:30,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:31:30,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:30,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:31:30,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:30,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:30,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:31:30,509 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 02:31:30,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:31:30,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:30,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:31:30,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:30,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:31:30,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:30,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:30,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:30,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:30,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:30,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:30,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:30,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:30,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:30,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:31:30,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:30,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:30,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:30,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:30,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:30,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:30,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:31,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:31,036 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 02:31:31,038 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:31:31,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:31:31,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:31:31,503 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:31:31,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:31:31,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:31:31,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:31:31,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:31:31,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:31:31,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:31:31,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:31:31,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:31:31,653 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 02:31:31,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:31:31,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:31,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:31:32,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:32,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:32,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:31:32,503 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 02:31:32,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:31:32,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:32,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:31:32,555 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 02:31:32,555 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:31:32,558 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:31:32,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:31:33,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:31:33,309 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:31:33,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:31:33,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:31:33,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:31:33,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:31:33,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:31:33,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:31:33,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:31:33,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:31:33,690 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 02:31:33,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:31:33,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:33,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:31:34,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:34,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:34,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:31:34,459 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 02:31:34,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:31:34,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:34,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:31:34,700 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 02:31:34,700 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:31:34,703 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:31:34,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:31:35,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:31:35,125 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:31:35,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:31:35,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:31:35,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:31:35,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:31:35,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:31:35,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:31:35,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:31:35,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:31:35,276 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 02:31:35,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:31:35,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:35,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:31:35,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:36,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:36,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:31:36,221 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 02:31:36,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:31:36,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:36,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:31:36,278 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 02:31:36,278 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:31:36,280 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 02:31:36,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:31:36,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 02:31:36,674 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:31:36,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 02:31:36,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 02:31:36,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 02:31:36,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 02:31:36,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 02:31:36,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 02:31:36,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 02:31:36,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:31:36,811 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 02:31:36,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:31:36,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:36,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:31:36,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:36,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:31:36,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:31:36,973 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 02:31:36,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:31:36,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:31:37,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:31:37,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:37,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:31:37,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:37,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:37,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:37,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:37,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:37,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:37,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:37,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:31:37,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:37,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:31:37,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:37,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:37,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:37,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:37,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:37,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:37,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:37,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:31:37,516 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 02:31:37,516 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.121.115]_1644315882.27 2022-02-08 02:31:37,516 | test | INFO | MainThread | [opd:abort_rebalance:808] Restarting Rebalance after killing at 60 2022-02-08 02:31:37,573 | infra | DEBUG | MainThread | [task:__init__:249] Last known rebalance status_id: bb2502617484b163934e158347397aac 2022-02-08 02:31:37,575 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.121.115]_1644316297.52 2022-02-08 02:31:37,575 | infra | INFO | pool-3-thread-10 | [task:start_task:91] Thread 'Rebalance_task_IN=[]_OUT=[172.23.121.115]_1644316297.52' started 2022-02-08 02:31:37,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L813 2022-02-08 02:31:37,585 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2022-02-08 02:31:37,634 | test | DEBUG | pool-3-thread-10 | [task:start_rebalance:384] Removing node 172.23.121.115:8091 from cluster 2022-02-08 02:31:37,635 | test | DEBUG | pool-3-thread-10 | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.128,ns_1@172.23.121.115,ns_1@172.23.121.132,ns_1@172.23.121.135,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.134,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123,ns_1@172.23.121.131,ns_1@172.23.121.130,ns_1@172.23.121.133', 'ejectedNodes': u'ns_1@172.23.121.115', 'user': 'Administrator'} 2022-02-08 02:31:37,700 | test | DEBUG | pool-3-thread-10 | [rest_client:rebalance:1459] Rebalance operation started 2022-02-08 02:31:37,710 | test | INFO | pool-3-thread-10 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.121.128 | kv | 7.1.0-2255-enterprise | 30.1565690946 | Cluster node | | 172.23.121.115 | kv | 7.1.0-2255-enterprise | 4.1719989762 | --- OUT ---> | | 172.23.121.132 | index, n1ql | 7.1.0-2255-enterprise | 4.67349551857 | Cluster node | | 172.23.121.135 | kv | 7.1.0-2255-enterprise | 91.2026442919 | Cluster node | | 172.23.120.170 | kv | 7.1.0-2255-enterprise | 28.3837416176 | Cluster node | | 172.23.121.129 | kv | 7.1.0-2255-enterprise | 29.2413421372 | Cluster node | | 172.23.121.124 | kv | 7.1.0-2255-enterprise | 31.7748219804 | Cluster node | | 172.23.121.126 | kv | 7.1.0-2255-enterprise | 30.1358695652 | Cluster node | | 172.23.121.134 | index, n1ql | 7.1.0-2255-enterprise | 7.18638902996 | Cluster node | | 172.23.121.116 | kv | 7.1.0-2255-enterprise | 91.8191056911 | Cluster node | | 172.23.121.127 | kv | 7.1.0-2255-enterprise | 31.6150740242 | Cluster node | | 172.23.121.123 | kv | 7.1.0-2255-enterprise | 32.1251174654 | Cluster node | | 172.23.121.131 | index, n1ql | 7.1.0-2255-enterprise | 5.24301569078 | Cluster node | | 172.23.121.130 | index, n1ql | 7.1.0-2255-enterprise | 6.21979140168 | Cluster node | | 172.23.121.133 | index, n1ql | 7.1.0-2255-enterprise | 6.43193085039 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2022-02-08 02:31:37,719 | infra | DEBUG | pool-3-thread-10 | [task:call:316] New rebalance status_id: 8036cae77467e2c6193bcec0a046b223 2022-02-08 02:31:37,730 | test | DEBUG | pool-3-thread-10 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 02:31:37,730 | test | INFO | pool-3-thread-10 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 02:31:37,769 | infra | DEBUG | pool-3-thread-10 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:31:37,773 | infra | DEBUG | pool-3-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:31:37,773 | infra | DEBUG | pool-3-thread-10 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:31:47,793 | test | DEBUG | pool-3-thread-10 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 02:31:47,793 | test | INFO | pool-3-thread-10 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 02:31:47,825 | infra | DEBUG | pool-3-thread-10 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:31:47,829 | infra | DEBUG | pool-3-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:31:47,831 | infra | DEBUG | pool-3-thread-10 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 3603 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:31:57,848 | test | DEBUG | pool-3-thread-10 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 02:31:57,848 | test | INFO | pool-3-thread-10 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 02:31:57,878 | infra | DEBUG | pool-3-thread-10 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:31:57,881 | infra | DEBUG | pool-3-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:31:57,881 | infra | DEBUG | pool-3-thread-10 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:32:07,897 | test | DEBUG | pool-3-thread-10 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.78 % 2022-02-08 02:32:07,898 | test | INFO | pool-3-thread-10 | [task:check:414] Rebalance - status: running, progress: 17.78 2022-02-08 02:32:07,926 | infra | DEBUG | pool-3-thread-10 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:32:07,928 | infra | DEBUG | pool-3-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:32:07,928 | infra | DEBUG | pool-3-thread-10 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:32:17,950 | test | DEBUG | pool-3-thread-10 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.67 % 2022-02-08 02:32:17,950 | test | INFO | pool-3-thread-10 | [task:check:414] Rebalance - status: running, progress: 26.67 2022-02-08 02:32:17,983 | infra | DEBUG | pool-3-thread-10 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:32:17,986 | infra | DEBUG | pool-3-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:32:17,986 | infra | DEBUG | pool-3-thread-10 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:32:27,999 | test | DEBUG | pool-3-thread-10 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 83.33 % 2022-02-08 02:32:27,999 | test | INFO | pool-3-thread-10 | [task:check:414] Rebalance - status: running, progress: 83.33 2022-02-08 02:32:28,072 | infra | DEBUG | pool-3-thread-10 | [task:check:441] Wait before next rebalance progress check 2022-02-08 02:32:28,075 | infra | DEBUG | pool-3-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 02:32:28,075 | infra | DEBUG | pool-3-thread-10 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 02:32:38,101 | infra | DEBUG | pool-3-thread-10 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-08 02:32:38,102 | infra | DEBUG | pool-3-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 02:32:43,121 | test | INFO | pool-3-thread-10 | [task:check:414] Rebalance - status: none, progress: 100 2022-02-08 02:32:43,216 | test | INFO | pool-3-thread-10 | [task:check:473] Rebalance completed with progress: 100% in 65.5150001049 sec 2022-02-08 02:32:43,217 | infra | INFO | pool-3-thread-10 | [task:complete_task:106] Thread 'Rebalance_task_IN=[]_OUT=[172.23.121.115]_1644316297.52' completed 5 queries submitted, 0 failed, 3628 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:33:37,615 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-08 02:33:37,615 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 02:33:42,630 | test | INFO | MainThread | [opd:abort_rebalance:814] Rebalance % = 100 2022-02-08 02:33:42,648 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-08 02:33:42,648 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 02:33:47,668 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 02:33:47,668 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 3652 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:33:57,670 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >100% in 15.0389997959 seconds 2022-02-08 02:33:57,706 | test | DEBUG | MainThread | [rest_client:diag_eval:1476] /diag/eval status on 172.23.120.170:8091: 1 content: false command: ns_orchestrator:needs_rebalance() 2022-02-08 02:33:57,706 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.121.115]_1644316297.52 2022-02-08 02:33:57,786 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3970] Stats: {u'opsPerSec': 15700.7, u'diskFetches': 4, u'dataUsed': 64201760652L, u'itemCount': 53006277, u'quotaPercentUsed': 80.73247549107118, u'storageTotals': {u'hdd': {u'usedByData': 111216141472L, u'used': 201098248512L, u'quotaTotal': 1168799956992L, u'total': 1168799956992L, u'free': 828461102661L}, u'ram': {u'usedByData': 62863315016L, u'used': 108780457984L, u'quotaTotal': 78809923584L, u'total': 110647066624L, u'quotaUsed': 77866205184L, u'quotaUsedPerNode': 8651800576L, u'quotaTotalPerNode': 8756658176L}}, u'diskUsed': 111216141472L, u'vbActiveNumNonResident': 13685785, u'memUsed': 62863315016L} 2022-02-08 02:33:58,904 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+------------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 53006277 | 72.52 GiB | 58.55 GiB | 103.58 GiB | 74.1873916795 | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+------------+---------------+ 2022-02-08 02:33:58,905 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+------------------+------------------+-----------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+------------------+------------------+-----------------+---------------+ | 40000000 | 380000000 | 2000000-20000000 | 2000000-20000000 | 1000000-2000000 | 0-0 | +---------------+---------------+------------------+------------------+-----------------+---------------+ 2022-02-08 02:33:59,321 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 74.1873916795 2022-02-08 02:33:59,322 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:33:59,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:00,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:34:00,407 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:00,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:34:00,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:34:00,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:34:00,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:34:00,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:34:00,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:34:00,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:34:00,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 02:34:00,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 02:34:00,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 02:34:00,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 02:34:00,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 02:34:01,290 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:34:01,293 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:34:01,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:01,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:34:01,763 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:01,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:34:01,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:34:01,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:34:01,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:34:01,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:34:01,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:34:01,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:34:01,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 02:34:02,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 02:34:03,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 02:34:03,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 02:34:03,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 02:34:03,517 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:34:03,519 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:34:03,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:03,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:34:03,921 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:03,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:34:03,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:34:03,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:34:03,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:34:03,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:34:04,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:34:04,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:34:04,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 02:34:04,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 02:34:04,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 02:34:04,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 02:34:04,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 02:34:04,852 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:34:04,855 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 02:34:04,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:05,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 02:34:05,233 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:05,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 02:34:05,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 02:34:05,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 02:34:05,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 02:34:05,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 02:34:05,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 02:34:05,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 02:34:05,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 02:34:05,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 02:34:05,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 02:34:05,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 02:34:05,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 02:34:05,453 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 02:34:05,457 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:34:06,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:07,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:34:07,467 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:07,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:34:07,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:34:07,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:34:07,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:34:07,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:34:07,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:34:07,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:34:07,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 02:34:08,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 02:34:08,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 02:34:08,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 02:34:08,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 02:34:08,917 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:34:08,921 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 02:34:09,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:09,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 02:34:09,315 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:09,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 02:34:09,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 02:34:09,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 02:34:09,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 02:34:09,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 02:34:09,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 02:34:09,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 02:34:09,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 02:34:09,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 02:34:09,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 02:34:09,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 02:34:09,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 02:34:09,532 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 02:34:09,533 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:34:09,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:10,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:34:10,119 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:10,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:34:10,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:34:10,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:34:10,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:34:10,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:34:10,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:34:10,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:34:10,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 02:34:10,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 02:34:10,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 02:34:10,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 02:34:10,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 02:34:11,039 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:34:11,042 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:34:11,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:11,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:34:11,586 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:11,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:34:11,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:34:11,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:34:11,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:34:11,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:34:11,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:34:11,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:34:11,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 02:34:11,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 02:34:11,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 02:34:11,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 02:34:12,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 02:34:12,447 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:34:12,448 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 02:34:12,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:12,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 02:34:12,862 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:12,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 02:34:12,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 02:34:12,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 02:34:12,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 02:34:12,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 02:34:12,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 02:34:12,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 02:34:13,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 02:34:13,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 02:34:13,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 02:34:13,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 02:34:13,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 02:34:13,121 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 02:34:13,122 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 02:34:13,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:13,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 02:34:13,506 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:13,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 02:34:13,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 02:34:13,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 02:34:13,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 02:34:13,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 02:34:13,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 02:34:13,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 02:34:13,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 02:34:13,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 02:34:13,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 02:34:13,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 02:34:13,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 02:34:13,733 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 02:34:13,736 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:34:13,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:14,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:34:14,224 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:14,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:34:14,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:34:14,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:34:14,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:34:14,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:34:14,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:34:14,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:34:14,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 02:34:14,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 02:34:14,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 02:34:14,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 02:34:14,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 02:34:15,108 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:34:15,109 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:34:15,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:16,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:34:17,012 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:17,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:34:17,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:34:17,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:34:17,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:34:17,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:34:17,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:34:17,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:34:17,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 02:34:17,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 02:34:17,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 02:34:17,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 02:34:17,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 02:34:18,391 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:34:18,394 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:34:18,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:18,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:34:18,888 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:18,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:34:18,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:34:18,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:34:18,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:34:18,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:34:19,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:34:19,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:34:19,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 02:34:19,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 02:34:19,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 02:34:19,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 02:34:19,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 02:34:19,878 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:34:19,881 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 02:34:19,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:20,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 02:34:20,305 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:20,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 02:34:20,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 02:34:20,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 02:34:20,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 02:34:20,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 02:34:20,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 02:34:20,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 02:34:20,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 02:34:20,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 02:34:20,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 02:34:20,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 02:34:20,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 02:34:20,555 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 02:34:20,555 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for kvstore is 102777MB 2022-02-08 02:34:20,555 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for wal is 4732MB 2022-02-08 02:34:20,555 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:642] Total Disk usage for keyTree is 7976MB 2022-02-08 02:34:20,555 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:643] Total Disk usage for seqTree is 94515MB 2022-02-08 02:34:20,558 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:34:20,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:20,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:34:21,022 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:21,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:34:21,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:34:21,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:34:21,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:34:21,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:34:21,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:34:21,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:34:21,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 02:34:21,450 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.116 - core(s): 8 2022-02-08 02:34:21,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 02:34:22,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 02:34:22,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 02:34:22,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 02:34:23,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 02:34:23,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 02:34:23,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 02:34:23,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 02:34:24,270 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:34:24,273 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:34:24,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:24,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:34:24,697 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:24,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:34:24,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:34:24,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:34:24,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:34:24,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:34:24,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:34:24,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:34:24,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 02:34:24,842 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.123 - core(s): 8 2022-02-08 02:34:24,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 02:34:25,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 02:34:25,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 02:34:25,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 02:34:25,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 02:34:26,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 02:34:26,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 02:34:26,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 02:34:26,733 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:34:26,736 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:34:26,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:27,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:34:27,375 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:27,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:34:27,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:34:27,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:34:27,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:34:27,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:34:27,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:34:27,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:34:27,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 02:34:27,726 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.124 - core(s): 8 2022-02-08 02:34:27,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 02:34:28,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 02:34:28,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 02:34:28,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 02:34:28,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 02:34:29,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 02:34:29,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 02:34:29,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 02:34:29,637 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:34:29,640 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:34:29,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:29,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:34:30,118 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:30,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:34:30,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:34:30,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:34:30,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:34:30,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:34:30,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:34:30,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:34:30,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 02:34:30,367 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.126 - core(s): 8 2022-02-08 02:34:30,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 02:34:31,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 02:34:31,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 02:34:31,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 02:34:31,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 02:34:31,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 02:34:31,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 02:34:32,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 02:34:32,299 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:34:32,302 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:34:32,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:32,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:34:32,970 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:32,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:34:32,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:34:33,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:34:33,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:34:33,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:34:33,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:34:33,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:34:33,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 02:34:33,193 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.127 - core(s): 8 2022-02-08 02:34:33,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 02:34:33,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 02:34:33,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 02:34:34,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 02:34:34,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 02:34:34,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 02:34:35,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 02:34:35,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 02:34:35,592 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:34:35,595 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:34:35,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:35,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:34:36,068 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:36,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:34:36,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:34:36,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:34:36,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:34:36,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:34:36,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:34:36,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:34:36,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 02:34:36,418 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.128 - core(s): 8 2022-02-08 02:34:36,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 02:34:37,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 02:34:37,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 02:34:37,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 02:34:38,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 02:34:38,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 02:34:38,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 02:34:38,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 02:34:39,259 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:34:39,266 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:34:39,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:39,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:34:39,778 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:39,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:34:39,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:34:39,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:34:39,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:34:39,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:34:39,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:34:39,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:34:39,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 02:34:39,963 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.129 - core(s): 8 2022-02-08 02:34:39,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 02:34:40,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 02:34:40,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 02:34:41,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 02:34:41,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 02:34:41,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 02:34:41,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 02:34:41,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 02:34:42,207 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:34:42,210 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:34:42,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:42,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:34:42,957 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:42,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:34:42,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:34:42,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:34:43,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:34:43,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:34:43,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:34:43,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:34:43,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 02:34:43,263 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.135 - core(s): 8 2022-02-08 02:34:43,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 02:34:44,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 02:34:44,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 02:34:44,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 02:34:44,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 02:34:44,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 02:34:45,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 02:34:45,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 02:34:45,549 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:34:45,551 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:34:46,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:34:46,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:34:46,641 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:34:46,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:34:46,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:34:46,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:34:46,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:34:46,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:34:46,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:34:46,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:34:46,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 02:34:46,936 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.120.170 - core(s): 8 2022-02-08 02:34:46,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 02:34:47,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 02:34:47,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 02:34:48,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 02:34:48,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 02:34:48,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 02:34:48,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 02:34:48,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 02:34:49,105 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:34:49,105 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:723] {'172.23.121.116': {'rw_0:magma': {u'NReadBytesGetPerSec': 707689.1875, u'ReplayWALOffset': u'623:8900608', u'NTablesCreated': 9908, u'RecentReadIOAmpSet': 0.020423395559191704, u'ReadIOAmp': 8.234803199768066, u'BlockCacheMisses': 197840, u'NFlushes': 2482, u'RecentWriteAmp': 1.9595625400543213, u'NumSyncFlushes': 123, u'TotalMemUsed': 39424788, u'NBloomFilterHits': 9889241, u'Fragmentation': 0.33392995595932007, u'NumNormalFlushes': 1090, u'RecentBlockCacheHitRatio': 60.23964309692383, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 5004, u'ReadIOAmp': 174.36363636363637, u'MemtablesMemUsed': 8528496, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 12633419, u'CheckpointOverhead': 9310208, u'NItemsGet': 11, u'NWriteIOs': 7058, u'NReadBytes': 280314577, u'LevelStats': [{u'TargetSize': 125829120, u'LogicalSize': 13958645, u'SizeRatio': 0.11093334356943767, u'NWriteBytes': 26017792, u'BloomFilterMemUsed': 0, u'NumTables': 254, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2301952}, {u'TargetSize': 125829120, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 125829120, u'LogicalSize': 94140834, u'SizeRatio': 0.7481641292572021, u'NWriteBytes': 12701696, u'BloomFilterMemUsed': 0, u'NumTables': 60, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 1597440}, {u'TargetSize': 658245473, u'LogicalSize': 461926901, u'SizeRatio': 0.7017547707464447, u'NWriteBytes': 15687680, u'BloomFilterMemUsed': 0, u'NumTables': 217, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 9314304}, {u'TargetSize': 2582908152L, u'LogicalSize': 1316490954, u'SizeRatio': 0.5096932900926475, u'NWriteBytes': 641974272, u'BloomFilterMemUsed': 0, u'NumTables': 535, u'WastedSpace': 25036878, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 1316429824}], u'NItemsGarbageCollected': 706074, u'NInternalFragmentationCompacts': 14, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 9394287, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 171360, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 154875, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 503370, u'CompactWriter': {u'NDataBlockWrites': 64027, u'NWriteBytes': 182198272, u'NIndexBlockWrites': 1094, u'NBytesIncoming': 292647197, u'NSyncs': 0, u'NInserts': 2246157, u'NWriteIOs': 967}, u'NonResidentBloomFilterSize': 0, u'IteratorMemUsed': 247152, u'NWriteBytes': 733529387, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 1918, u'NReadBytes': 260092, u'NDataBlockReads': 11, u'NBytesOutgoing': 0, u'NIndexBlockReads': 20}, u'NBloomFilterMisses': 0, u'NItemsMerged': 302461, u'LogicalDiskSize': 1316490954, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.5932569589205385, u'BufferMemUsed': 1410048, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 854564053, u'NTableFiles': 1616, u'NFileCountCompacts': 94, u'TableObjectMemUsed': 539760, u'TreeSnapshotMemoryUsed': 243120, u'TotalDiskUsage': 1769525248, u'BytesIncoming': 1236444640, u'NItemsDel': 0, u'PendingCheckpointBytes': 21221376, u'NCompacts': 303, u'NItemsPut': 1784203, u'NSyncs': 1396, u'CompactReader': {u'NReadIOs': 105422, u'NReadBytes': 255011503, u'NDataBlockReads': 102033, u'NBytesOutgoing': 0, u'NIndexBlockReads': 3389}, u'ActiveDiskUsage': 1329643520, u'NTablesDeleted': 2687, u'IterateReader': {u'NReadIOs': 47535, u'NReadBytes': 25042982, u'NDataBlockReads': 24826, u'NBytesOutgoing': 0, u'NIndexBlockReads': 22709}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 187212, u'NWriteBytes': 551331115, u'NIndexBlockWrites': 7674, u'NBytesIncoming': 864705264, u'NSyncs': 1396, u'NInserts': 1402436, u'NWriteIOs': 6091}}, u'NReadBytes': 3956813595L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 205.73333740234375, u'NInternalFragmentationCompacts': 14, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 10368902, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 32768, u'NTxns': 128030, u'NWriteBytes': 1825435648, u'NSegments': 117, u'NSyncs': 128029, u'NLoggedBytes': 1825443840, u'NActiveSegments': 5, u'NWrites': 132007, u'PreallocateSupported': True, u'NInserts': 2374498, u'DiskUsed': 86716416}, u'NGets': 137196, u'NReadBytesSet': 69164357, u'NReadBytesCompact': 481880758, u'IndexResidentRatio': 0.2701457738876343, u'CompactQueueSize': 0, u'NSyncsPerSec': 3.9666666984558105, u'BlockCacheHitRatio': 90.11390686035156, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 3449889855L, u'BloomFilterMemUsedRatio': 0.019321462139487267, u'NonResidentBloomFilterSize': 280904, u'NWriteBytes': 2938068557L, u'BytesOutgoing': 9881, u'RecentReadIOAmp': 2.0921883583068848, u'WriteAmp': 2.145049810409546, u'NBlocksCached': 7115, u'LogicalDataSize': 921377919, u'ActiveBloomFilterMemUsed': 1435544, u'NFileCountCompacts': 342, u'NTableFiles': 2943, u'FlushQueueSize': 8, u'TableObjectMemUsed': 858240, u'BlockCacheQuota': 83045001, u'TotalDiskUsage': 1903976448, u'NPurgerRuns': 0, u'ReadIOAmpSet': 0.0, u'NSyncs': 4188, u'NReadIOsPerSec': 472.8666687011719, u'NReadIOsSetPerSec': 20.066667556762695, u'NTablesDeleted': 7257, u'BloomFilterFPR': 0.014829006977379322, u'NBloomFilterHitsPerSec': 6191.033203125, u'WriteCacheMemUsed': 10823504, u'RecentBloomFilterFPR': 0.016046831384301186, u'BytesPerRead': 24641.57421875, u'BytesIncomingPerSec': 999765.3125, u'NReadIOsGetPerSec': 242.8333282470703, u'NumKVStores': 30, u'NGetStats': 1627, u'TotalIndexBlocksSize': 14264052, u'NGetStatsPerSec': 0.8333333134651184, u'NWriteBytesCompact': 464601088, u'NReadIOsSet': 22967, u'ActiveDataSize': 921377919, u'NWriteBytesCompactPerSec': 368230.40625, u'NBloomFilterMissesPerSec': 100.96666717529297, u'NCommitBatches': 116315, u'NGetStatsComputed': 1627, u'NWriteIOs': 147266, u'NReadIOsGet': 1152749, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 20761250, u'LSMTreeObjectMemUsed': 517440, u'NSetsPerSec': 982.5333251953125, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 2.5999999046325684, u'ManagedFileObjectMemUsed': 298752, u'ReadAmpGet': 342144.0625, u'BlockCacheMemUsed': 21376960, u'NWriterCompacts': 0, u'NReadIOs': 1385580, u'WALBufferMemUsed': 81920, u'ReadAmp': 393446.9375, u'keyStats': {u'NItemsTransformed': 6235518, u'NTablesCreated': 2501, u'ReadIOAmp': 0.7237419148504853, u'MemtablesMemUsed': 926320, u'NBloomFilterHits': 9888907, u'TotalIndexBlocksSize': 1599270, u'CheckpointOverhead': 29155328, u'NItemsGet': 1588406, u'NWriteIOs': 4402, u'NReadBytes': 3672587135L, u'LevelStats': [{u'TargetSize': 125829120, u'LogicalSize': 6625553, u'SizeRatio': 0.05265516440073649, u'NWriteBytes': 72175616, u'BloomFilterMemUsed': 81016, u'NumTables': 291, u'WastedSpace': 211, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 6025216}, {u'TargetSize': 125829120, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 125829120, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 0}, {u'TargetSize': 125829120, u'LogicalSize': 4793413, u'SizeRatio': 0.038094623883565264, u'NWriteBytes': 13860864, u'BloomFilterMemUsed': 91136, u'NumTables': 4, u'WastedSpace': 11258, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 4800512}, {u'TargetSize': 125829120, u'LogicalSize': 66162862, u'SizeRatio': 0.5258151849110921, u'NWriteBytes': 265273344, u'BloomFilterMemUsed': 1257728, u'NumTables': 56, u'WastedSpace': 141007, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 66269184}], u'NItemsGarbageCollected': 874898, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 958716, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 65664, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 1225366, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 439425, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 212688, u'CompactWriter': {u'NDataBlockWrites': 96079, u'NWriteBytes': 281370624, u'NIndexBlockWrites': 1366, u'NBytesIncoming': 362145750, u'NSyncs': 0, u'NInserts': 6235518, u'NWriteIOs': 728}, u'NonResidentBloomFilterSize': 278977, u'IteratorMemUsed': 0, u'NWriteBytes': 358737368, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 1149596, u'NReadBytes': 3449169336L, u'NDataBlockReads': 4104165, u'NBytesOutgoing': 0, u'NIndexBlockReads': 5866266}, u'NBloomFilterMisses': 148722, u'NItemsMerged': 0, u'LogicalDiskSize': 66497778, u'TotalBloomFilterMemUsed': 2080832, u'WriteAmp': 5.366492157405213, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1429880, u'LogicalDataSize': 66497778, u'NTableFiles': 684, u'NFileCountCompacts': 122, u'TableObjectMemUsed': 164160, u'TreeSnapshotMemoryUsed': 166336, u'TotalDiskUsage': 129183744, u'BytesIncoming': 66847646, u'NItemsDel': 0, u'PendingCheckpointBytes': 5173248, u'NCompacts': 124, u'NItemsPut': 1340624, u'NSyncs': 1396, u'CompactReader': {u'NReadIOs': 75770, u'NReadBytes': 223417799, u'NDataBlockReads': 104682, u'NBytesOutgoing': 0, u'NIndexBlockReads': 3409}, u'ActiveDiskUsage': 77094912, u'NTablesDeleted': 2313, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.014816447190865492, u'Writer': {u'NDataBlockWrites': 15692, u'NWriteBytes': 77366744, u'NIndexBlockWrites': 2297, u'NBytesIncoming': 54623736, u'NSyncs': 1396, u'NInserts': 928945, u'NWriteIOs': 3674}}, u'NReadBytesCompactPerSec': 385755.90625, u'TableMetaMemUsed': 900556, u'NDeletes': 0, u'BlockCacheHitsPerSec': 311.70001220703125, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 148855, u'LogicalDiskSize': 1383304820, u'NSets': 995888, u'TotalBloomFilterMemUsed': 2089568, u'BufferMemUsed': 1410048, u'RecentReadAmp': 0.0, u'NGetsPerSec': 116.06666564941406, u'NInserts': 344736, u'NCommitBatchesPerSec': 68.4000015258789, u'FragmentationWithOverhead': 0.5160770416259766, u'NCompactsPerSec': 0.5666666626930237, u'WALMemUsed': 147536, u'MemoryQuota': 108147512, u'NReadBytesPerSec': 1120814.125, u'TreeSnapshotMemoryUsed': 534928, u'BytesIncoming': 1369697074, u'NCompacts': 553, u'NumPersistentFlushes': 1269, u'NWriteIOsPerSec': 84.0, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 6097.2646484375, u'ActiveDiskUsage': 1409400832, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 0.8333333134651184, u'NReadBytesSetPerSec': 57510.3671875, u'BlockCacheHits': 1803355, u'NWriteBytesPerSec': 1959102.625, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 2403, u'ReadIOAmp': 0.811965811965812, u'MemtablesMemUsed': 1368688, u'NBloomFilterHits': 334, u'TotalIndexBlocksSize': 31363, u'CheckpointOverhead': 1409024, u'NItemsGet': 1521, u'NWriteIOs': 3799, u'NReadBytes': 3911883, u'LevelStats': [{u'TargetSize': 125829120, u'LogicalSize': 2979887, u'SizeRatio': 0.023682014147440592, u'NWriteBytes': 18415616, u'BloomFilterMemUsed': 4704, u'NumTables': 295, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2416640}, {u'TargetSize': 125829120, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 125829120, u'LogicalSize': 316088, u'SizeRatio': 0.002512041727701823, u'NWriteBytes': 1024000, u'BloomFilterMemUsed': 960, u'NumTables': 30, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 245760}], u'NItemsGarbageCollected': 455984, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 15899, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 61728, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 5339, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 184498, u'CompactWriter': {u'NDataBlockWrites': 126, u'NWriteBytes': 1032192, u'NIndexBlockWrites': 126, u'NBytesIncoming': 376466, u'NSyncs': 0, u'NInserts': 2898, u'NWriteIOs': 126}, u'NonResidentBloomFilterSize': 1927, u'IteratorMemUsed': 0, u'NWriteBytes': 20366154, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 1235, u'NReadBytes': 460427, u'NDataBlockReads': 1043, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1043}, u'NBloomFilterMisses': 133, u'NItemsMerged': 0, u'LogicalDiskSize': 316088, u'TotalBloomFilterMemUsed': 8736, u'WriteAmp': 0.13130436633231157, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 5664, u'LogicalDataSize': 316088, u'NTableFiles': 643, u'NFileCountCompacts': 126, u'TableObjectMemUsed': 154320, u'TreeSnapshotMemoryUsed': 125472, u'TotalDiskUsage': 5267456, u'BytesIncoming': 155106449, u'NItemsDel': 0, u'PendingCheckpointBytes': 360448, u'NCompacts': 126, u'NItemsPut': 461619, u'NSyncs': 1396, u'CompactReader': {u'NReadIOs': 4104, u'NReadBytes': 3451456, u'NDataBlockReads': 2257, u'NBytesOutgoing': 0, u'NIndexBlockReads': 2257}, u'ActiveDiskUsage': 2662400, u'NTablesDeleted': 2257, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.284796573875803, u'Writer': {u'NDataBlockWrites': 2277, u'NWriteBytes': 19333962, u'NIndexBlockWrites': 2277, u'NBytesIncoming': 3486508, u'NSyncs': 1396, u'NInserts': 24300, u'NWriteIOs': 3673}}}}} 2022-02-08 02:34:49,108 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:728] magma stats fragmentation result {'172.23.121.123': [0.3710159659385681, 0.3645918369293213, 0.3539392948150635, 0.31738802790641785, 0.3214837312698364, 0.3146960437297821, 0.3471758961677551, 0.3430598974227905], '172.23.121.124': [0.33209916949272156, 0.33481934666633606, 0.38130712509155273, 0.3393324315547943, 0.35932475328445435, 0.3111364245414734, 0.3290572762489319, 0.3293977677822113], '172.23.121.135': [0.3143472671508789, 0.36335426568984985, 0.29791733622550964, 0.34730011224746704, 0.36615481972694397, 0.3852057456970215, 0.3328436613082886, 0.3698110282421112], '172.23.121.126': [0.3029825985431671, 0.3490598201751709, 0.3174378275871277, 0.35979408025741577, 0.34700676798820496, 0.3268066644668579, 0.35603636503219604, 0.3745384216308594], '172.23.121.116': [0.33392995595932007, 0.36321118474006653, 0.32724636793136597, 0.35607174038887024, 0.3712424039840698, 0.3603338599205017, 0.381320059299469, 0.34118685126304626], '172.23.121.127': [0.31470632553100586, 0.3050844371318817, 0.32237735390663147, 0.36514267325401306, 0.36490923166275024, 0.3868953287601471, 0.36598801612854004, 0.31414100527763367], '172.23.121.128': [0.32676419615745544, 0.32620295882225037, 0.30784693360328674, 0.3029356300830841, 0.3214668333530426, 0.31263458728790283, 0.31321054697036743, 0.33012306690216064], '172.23.121.129': [0.33883586525917053, 0.3275134265422821, 0.34038060903549194, 0.32990825176239014, 0.34315025806427, 0.3282221853733063, 0.3580508232116699, 0.32534897327423096], '172.23.120.170': [0.2996651232242584, 0.2889353930950165, 0.29436782002449036, 0.31957846879959106, 0.28049272298812866, 0.3126707673072815, 0.255812406539917, 0.31550174951553345]} 5 queries submitted, 0 failed, 3671 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:34:49,654 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.116 is 36.0 2022-02-08 02:34:50,078 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.123 is 36.0 2022-02-08 02:34:50,480 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.124 is 36.0 2022-02-08 02:34:50,838 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.126 is 36.0 2022-02-08 02:34:51,227 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.127 is 36.0 2022-02-08 02:34:51,703 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.128 is 36.0 2022-02-08 02:34:52,107 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.129 is 36.0 2022-02-08 02:34:52,494 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.135 is 36.0 2022-02-08 02:34:52,948 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.120.170 is 36.0 2022-02-08 02:34:52,950 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:677] KV stats fragmentation values {'172.23.121.123': 36.0, '172.23.121.124': 36.0, '172.23.121.135': 36.0, '172.23.121.126': 36.0, '172.23.121.116': 36.0, '172.23.121.127': 36.0, '172.23.121.128': 36.0, '172.23.121.129': 36.0, '172.23.120.170': 36.0} ############################################################ # # Step 7: Crash Magma/memc with Loading of docs # ############################################################ ############################################################ # # RAM FootPrint: 4174.34375 # ############################################################ 2022-02-08 02:34:53,223 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 02:34:53,223 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 67 seconds. Reason: Iteration:0 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 3695 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:36:00,229 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 02:36:00,229 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 02:36:05,233 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:36:05,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:05,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:36:05,660 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:05,660 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:36:05,676 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:36:05,693 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:36:05,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:36:05,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:36:05,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:36:05,767 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:36:05,792 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:36:05,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 02:36:05,877 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['98712 memcached\n'], []) 2022-02-08 02:36:05,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 02:36:05,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 02:36:05,953 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['98712 memcached\n'], []) 2022-02-08 02:36:05,953 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:36:05,956 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:36:06,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:06,704 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:36:06,819 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:06,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:36:06,854 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:36:06,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:36:06,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:36:06,987 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:36:07,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:36:07,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:36:07,072 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:36:07,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 02:36:07,180 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['7225 memcached\n'], []) 2022-02-08 02:36:07,181 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 02:36:07,211 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 02:36:07,236 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['7225 memcached\n'], []) 2022-02-08 02:36:07,237 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:36:07,240 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:36:07,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:08,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:36:08,134 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:08,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:36:08,158 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:36:08,191 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:36:08,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:36:08,237 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:36:08,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:36:08,295 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:36:08,315 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:36:08,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 02:36:08,453 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['126939 memcached\n'], []) 2022-02-08 02:36:08,454 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 02:36:08,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 02:36:08,507 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['126939 memcached\n'], []) 2022-02-08 02:36:08,507 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:36:08,509 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:36:09,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:09,739 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:36:09,986 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:09,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:36:10,005 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:36:10,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:36:10,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:36:10,081 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:36:10,099 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:36:10,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:36:10,154 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:36:10,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 02:36:10,237 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['34000 memcached\n'], []) 2022-02-08 02:36:10,239 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 02:36:10,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 02:36:10,311 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['34000 memcached\n'], []) 2022-02-08 02:36:10,311 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:36:10,313 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:36:10,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:11,061 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:36:11,210 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:11,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:36:11,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:36:11,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:36:11,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:36:11,499 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:36:11,522 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:36:11,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:36:11,627 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:36:11,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 02:36:11,825 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['27651 memcached\n'], []) 2022-02-08 02:36:11,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 02:36:11,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 02:36:11,877 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['27651 memcached\n'], []) 2022-02-08 02:36:11,878 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:36:11,880 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:36:12,367 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:12,596 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:36:12,723 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:12,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:36:12,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:36:12,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:36:12,795 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:36:12,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:36:12,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:36:12,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:36:12,931 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:36:12,933 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 02:36:13,082 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['68899 memcached\n'], []) 2022-02-08 02:36:13,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 02:36:13,198 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 02:36:13,221 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['68899 memcached\n'], []) 2022-02-08 02:36:13,223 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:36:13,223 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:36:13,806 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:14,354 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:36:14,645 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:14,647 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:36:14,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:36:14,701 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:36:14,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:36:14,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:36:14,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:36:14,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:36:14,940 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:36:14,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 02:36:15,020 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['62669 memcached\n'], []) 2022-02-08 02:36:15,022 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 02:36:15,094 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 02:36:15,115 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['62669 memcached\n'], []) 2022-02-08 02:36:15,115 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:36:15,118 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:36:15,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:15,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:36:15,499 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:15,505 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:36:15,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:36:15,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:36:15,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:36:15,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:36:15,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:36:15,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:36:15,605 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:36:15,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 02:36:15,676 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['127231 memcached\n'], []) 2022-02-08 02:36:15,676 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 02:36:15,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 02:36:15,723 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['127231 memcached\n'], []) 2022-02-08 02:36:15,723 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:36:15,726 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:36:15,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:16,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:36:16,282 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:16,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:36:16,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:36:16,308 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:36:16,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:36:16,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:36:16,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:36:16,359 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:36:16,417 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:36:16,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:36:16,607 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['78955 memcached\n'], []) 2022-02-08 02:36:16,607 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 02:36:16,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:36:16,733 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['78955 memcached\n'], []) 2022-02-08 02:36:16,733 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:36:16,734 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:36:16,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:17,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:36:17,128 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:17,130 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:36:17,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:36:17,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:36:17,170 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:36:17,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:36:17,201 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:36:17,216 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:36:17,232 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:36:17,247 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 02:36:17,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:36:17,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:18,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:36:18,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:19,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:19,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:36:19,256 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 02:36:19,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:36:19,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:19,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:36:19,326 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 02:36:19,326 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:36:19,329 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:36:19,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:19,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:36:19,730 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:19,732 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:36:19,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:36:19,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:36:19,772 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:36:19,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:36:19,801 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:36:19,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:36:19,826 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:36:19,844 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 02:36:19,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:36:19,885 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:20,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:36:20,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:20,522 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:20,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:36:20,671 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 02:36:20,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:36:20,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:20,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:36:20,713 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 02:36:20,713 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:36:20,714 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:36:20,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:21,017 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:36:21,082 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:21,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:36:21,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:36:21,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:36:21,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:36:21,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:36:21,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:36:21,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:36:21,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:36:21,194 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 02:36:21,219 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:36:21,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:21,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:36:21,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:21,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:22,073 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:36:22,089 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 02:36:22,089 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:36:22,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:22,119 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:36:22,132 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 02:36:22,134 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:36:22,137 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 02:36:22,237 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:22,459 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 02:36:22,528 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:22,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 02:36:22,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 02:36:22,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 02:36:22,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 02:36:22,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 02:36:22,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 02:36:22,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 02:36:22,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:36:22,641 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 02:36:22,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:36:22,683 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:22,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:36:22,709 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:22,721 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:22,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:36:22,746 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 02:36:22,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:36:22,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:22,772 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:36:22,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:22,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:36:22,844 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:22,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:22,905 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:22,933 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:22,966 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:22,994 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:23,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:23,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:23,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:23,099 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:36:23,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:23,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:23,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:23,165 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:23,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:23,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:23,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:23,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:23,246 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 02:36:23,250 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:36:23,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:23,589 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:36:23,653 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:23,653 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:36:23,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:36:23,678 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:36:23,693 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:36:23,709 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:36:23,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:36:23,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:36:23,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:36:23,762 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 02:36:23,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:36:23,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:24,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:36:24,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:24,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:24,920 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:36:24,934 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 02:36:24,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:36:24,948 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:24,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:36:25,000 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 02:36:25,000 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:36:25,003 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 02:36:25,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:25,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 02:36:25,362 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:25,362 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 02:36:25,377 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 02:36:25,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 02:36:25,400 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 02:36:25,415 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 02:36:25,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 02:36:25,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 02:36:25,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:36:25,467 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 02:36:25,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:36:25,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:25,517 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:36:25,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:25,543 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:25,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:36:25,569 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 02:36:25,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:36:25,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:25,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:36:25,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:25,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:36:25,665 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:25,691 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:25,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:25,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:25,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:25,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:25,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:25,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:25,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:25,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:36:25,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:25,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:25,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:25,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:26,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:26,022 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:26,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:26,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:26,073 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 02:36:26,075 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:36:26,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:26,372 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:36:26,427 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:26,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:36:26,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:36:26,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:36:26,467 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:36:26,487 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:36:26,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:36:26,515 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:36:26,528 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:36:26,540 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 02:36:26,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:36:26,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:26,867 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:36:27,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:27,303 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:27,469 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:36:27,496 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 02:36:27,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:36:27,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:27,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:36:27,539 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 02:36:27,539 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:36:27,542 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:36:27,630 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:27,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:36:27,908 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:27,910 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:36:27,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:36:27,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:36:27,951 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:36:27,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:36:27,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:36:27,999 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:36:28,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:36:28,032 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 02:36:28,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:36:28,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:28,318 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:36:28,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:28,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:28,829 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:36:28,851 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 02:36:28,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:36:28,867 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:28,882 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:36:28,898 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 02:36:28,900 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:36:28,901 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 02:36:28,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:29,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 02:36:29,283 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:29,285 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 02:36:29,298 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 02:36:29,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 02:36:29,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 02:36:29,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 02:36:29,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 02:36:29,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 02:36:29,387 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:36:29,401 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 02:36:29,433 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:36:29,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:29,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:36:29,474 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:29,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:29,499 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:36:29,510 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 02:36:29,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:36:29,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:29,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:36:29,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:29,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:36:29,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:29,640 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:29,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:29,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:29,729 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:29,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:29,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:29,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:29,836 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:29,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:36:29,869 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:29,887 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:29,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:29,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:29,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:29,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:29,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:29,984 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:30,000 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 02:36:30,003 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 02:36:30,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:30,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 02:36:30,397 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:30,397 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 02:36:30,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 02:36:30,426 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 02:36:30,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 02:36:30,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 02:36:30,479 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 02:36:30,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 02:36:30,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:36:30,530 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 02:36:30,586 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:36:30,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:30,619 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:36:30,635 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:30,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:30,665 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:36:30,681 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 02:36:30,681 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:36:30,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:30,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:36:30,724 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:30,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:36:30,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:30,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:30,857 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:30,885 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:30,920 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:30,948 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:30,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:31,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:31,045 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:31,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:36:31,079 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:31,096 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:31,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:31,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:31,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:31,170 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:31,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:31,204 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:31,220 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 02:36:31,226 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:36:31,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:31,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:36:31,657 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:31,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:36:31,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:36:31,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:36:31,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:36:31,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:36:31,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:36:31,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:36:31,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:36:31,801 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 02:36:31,826 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:36:31,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:32,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:36:32,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:32,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:32,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:36:32,664 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 02:36:32,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:36:32,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:32,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:36:32,703 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 02:36:32,703 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:36:32,704 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:36:33,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:33,987 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:36:34,427 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:34,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:36:34,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:36:34,551 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:36:34,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:36:34,742 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:36:34,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:36:34,872 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:36:34,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:36:35,082 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 02:36:35,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:36:35,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:36,522 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:36:36,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:37,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:37,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:36:37,190 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 02:36:37,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:36:37,206 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:37,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:36:37,273 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 02:36:37,273 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:36:37,276 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:36:37,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:37,579 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:36:37,651 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:37,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:36:37,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:36:37,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:36:37,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:36:37,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:36:37,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:36:37,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:36:37,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:36:37,788 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 02:36:37,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:36:37,831 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:38,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:36:38,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:38,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:38,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:36:38,832 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 02:36:38,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:36:38,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:38,874 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:36:38,894 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 02:36:38,894 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:36:38,895 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 02:36:38,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:39,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 02:36:39,279 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:39,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 02:36:39,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 02:36:39,311 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 02:36:39,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 02:36:39,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 02:36:39,364 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 02:36:39,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 02:36:39,395 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:36:39,414 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 02:36:39,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:36:39,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:39,474 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:36:39,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:39,505 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:36:39,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:36:39,536 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 02:36:39,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:36:39,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:36:39,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:36:39,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:39,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:36:39,653 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:39,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:39,729 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:39,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:39,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:39,834 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:39,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:39,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:36:39,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:39,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:36:39,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:39,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:40,012 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:40,030 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:40,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:40,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:40,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:40,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:36:40,134 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 02:36:40,135 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:36:40,138 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:36:40,234 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:40,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:36:40,539 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:40,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:36:40,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:36:40,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:36:40,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:36:40,604 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:36:40,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:36:40,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:36:40,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:36:40,957 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:36:40,957 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.821000099182. 2022-02-08 02:36:40,957 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:36:40,959 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:36:41,130 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:36:41,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:36:41,608 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:36:41,609 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:36:41,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:36:41,951 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:36:42,038 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:36:42,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:36:42,349 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:36:42,387 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:36:42,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:36:42,664 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:36:42,665 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:36:44,665 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:36:44,957 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:36:44,957 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:36:46,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:36:47,213 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:36:47,214 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 3718 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:36:49,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:36:49,436 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:36:49,436 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:36:51,436 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:36:51,628 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:36:51,630 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:36:53,637 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:36:53,880 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:36:53,880 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:36:55,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:36:56,071 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:36:56,071 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:36:58,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:36:58,369 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:36:58,371 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:37:00,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:00,578 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:00,578 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:37:02,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:02,878 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:02,878 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:37:04,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:05,388 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:05,388 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:37:07,392 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:07,594 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:07,595 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:37:09,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:09,835 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:09,835 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:37:11,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:12,132 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:12,132 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:37:14,132 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:14,315 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:14,315 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:37:16,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:16,634 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:16,634 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:37:18,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:18,865 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:18,865 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:37:20,865 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:21,042 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:21,042 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:37:23,043 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:23,246 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:23,246 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:37:25,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:25,424 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:37:25,424 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 44.4670000076. 2022-02-08 02:37:25,426 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:37:25,427 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:37:25,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:37:25,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:37:26,102 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:37:26,104 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:37:26,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:37:26,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:37:26,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:37:26,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:37:26,221 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:37:26,244 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:37:26,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:26,653 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:26,653 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 02:37:28,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:28,884 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:28,885 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 02:37:30,887 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:31,072 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:31,073 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 02:37:33,073 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:33,244 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:37:33,244 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.81999993324. 2022-02-08 02:37:33,244 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:37:33,247 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:37:33,364 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:37:33,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:37:33,752 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:37:33,752 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:37:33,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:37:33,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:37:33,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:37:33,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:37:33,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:37:33,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:37:33,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:34,240 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:37:34,240 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.995000123978. 2022-02-08 02:37:34,240 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:37:34,242 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:37:34,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:37:35,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:37:35,132 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:37:35,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:37:35,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:37:35,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:37:35,204 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:37:35,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:37:35,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:37:35,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:37:35,295 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:35,584 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:37:35,585 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.34500002861. 2022-02-08 02:37:35,585 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:37:35,586 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:37:36,395 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:37:36,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:37:36,880 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:37:36,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:37:36,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:37:36,926 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:37:36,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:37:36,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:37:36,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:37:37,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:37:37,048 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:37,388 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:37:37,391 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.80599999428. 2022-02-08 02:37:37,391 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:37:37,394 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:37:38,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:37:38,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:37:38,966 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:37:38,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:37:39,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:37:39,068 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:37:39,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:37:39,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:37:39,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:37:39,201 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:37:39,227 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:39,532 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:37:39,532 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.14199995995. 2022-02-08 02:37:39,532 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:37:39,535 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:37:40,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:37:40,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:37:40,578 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:37:40,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:37:40,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:37:40,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:37:40,622 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:37:40,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:37:40,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:37:40,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:37:40,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:41,010 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:37:41,010 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.47799992561. 2022-02-08 02:37:41,012 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:37:41,013 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:37:41,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:37:41,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:37:42,503 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:37:42,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:37:42,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:37:42,742 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:37:42,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:37:42,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:37:43,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:37:43,183 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:37:43,285 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:43,964 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:43,964 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 02:37:45,966 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:46,296 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:46,296 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 02:37:48,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:48,651 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:48,651 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 5 queries submitted, 0 failed, 3743 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:37:50,653 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:50,823 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:50,823 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 02:37:52,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:53,076 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:53,078 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 02:37:55,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:55,404 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:55,404 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 02:37:57,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:37:57,792 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:37:57,793 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 02:37:59,795 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:38:00,160 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:38:00,160 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 02:38:02,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:38:02,661 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:38:02,661 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 02:38:04,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:38:05,436 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:38:05,436 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 02:38:07,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:38:07,878 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:38:07,878 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 02:38:09,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:38:10,171 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:38:10,171 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 02:38:12,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:38:12,390 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:38:12,390 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 31.378000021. ############################################################ # # RAM FootPrint: 4171.3515625 # ############################################################ 2022-02-08 02:38:12,444 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 02:38:12,444 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 64 seconds. Reason: Iteration:1 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 3763 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:39:16,447 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 02:39:16,448 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 02:39:21,450 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:39:22,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:39:22,729 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:39:22,898 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:39:22,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:39:22,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:39:22,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:39:22,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:39:22,979 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:39:22,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:39:23,028 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:39:23,043 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:39:23,045 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 02:39:23,118 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['104298 memcached\n'], []) 2022-02-08 02:39:23,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 02:39:23,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 02:39:23,194 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['104298 memcached\n'], []) 2022-02-08 02:39:23,194 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:39:23,196 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:39:24,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:39:24,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:39:25,186 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:39:25,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:39:25,255 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:39:25,354 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:39:25,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:39:25,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:39:25,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:39:25,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:39:25,543 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:39:25,545 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 02:39:25,648 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['11574 memcached\n'], []) 2022-02-08 02:39:25,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 02:39:25,691 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 02:39:25,727 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['11574 memcached\n'], []) 2022-02-08 02:39:25,727 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:39:25,730 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:39:26,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:39:27,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:39:27,575 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:39:27,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:39:27,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:39:27,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:39:27,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:39:27,716 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:39:27,749 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:39:27,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:39:27,851 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:39:27,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 02:39:27,930 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['1007 memcached\n'], []) 2022-02-08 02:39:27,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 02:39:28,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 02:39:28,042 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['1007 memcached\n'], []) 2022-02-08 02:39:28,042 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:39:28,043 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:39:29,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:39:29,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:39:29,940 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:39:29,941 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:39:29,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:39:29,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:39:30,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:39:30,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:39:30,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:39:30,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:39:30,210 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:39:30,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 02:39:30,305 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['38364 memcached\n'], []) 2022-02-08 02:39:30,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 02:39:30,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 02:39:30,361 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['38364 memcached\n'], []) 2022-02-08 02:39:30,361 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:39:30,365 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:39:31,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:39:32,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:39:32,372 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:39:32,372 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:39:32,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:39:32,482 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:39:32,538 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:39:32,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:39:32,586 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:39:32,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:39:32,641 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:39:32,642 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 02:39:32,700 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['31998 memcached\n'], []) 2022-02-08 02:39:32,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 02:39:32,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 02:39:32,753 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['31998 memcached\n'], []) 2022-02-08 02:39:32,753 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:39:32,756 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:39:34,420 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:39:34,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:39:35,092 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:39:35,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:39:35,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:39:35,186 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:39:35,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:39:35,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:39:35,232 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:39:35,265 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:39:35,286 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:39:35,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 02:39:35,359 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['73273 memcached\n'], []) 2022-02-08 02:39:35,359 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 02:39:35,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 02:39:35,405 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['73273 memcached\n'], []) 2022-02-08 02:39:35,405 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:39:35,407 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:39:36,329 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:39:37,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:39:37,875 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:39:37,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:39:37,895 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:39:37,915 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:39:37,931 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:39:37,954 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:39:37,969 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:39:38,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:39:38,025 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:39:38,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 02:39:38,094 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['67015 memcached\n'], []) 2022-02-08 02:39:38,094 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 02:39:38,181 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 02:39:38,207 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['67015 memcached\n'], []) 2022-02-08 02:39:38,207 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:39:38,210 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:39:39,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:39:39,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:39:39,880 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:39:39,880 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:39:39,897 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:39:39,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:39:39,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:39:39,964 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:39:39,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:39:40,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:39:40,015 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:39:40,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 02:39:40,056 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['881 memcached\n'], []) 2022-02-08 02:39:40,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 02:39:40,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 02:39:40,105 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['881 memcached\n'], []) 2022-02-08 02:39:40,107 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:39:40,108 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:39:40,806 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:39:41,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:39:41,345 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:39:41,346 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:39:41,408 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:39:41,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:39:41,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:39:41,507 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:39:41,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:39:41,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:39:41,573 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:39:41,573 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:39:41,634 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['80237 memcached\n'], []) 2022-02-08 02:39:41,635 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 02:39:41,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:39:41,684 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['80237 memcached\n'], []) 2022-02-08 02:39:41,684 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:39:41,687 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:39:41,755 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:39:41,987 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:39:42,055 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:39:42,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:39:42,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:39:42,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:39:42,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:39:42,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:39:42,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:39:42,160 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:39:42,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:39:42,194 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 02:39:42,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:39:42,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:39:44,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:39:44,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:39:44,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:39:44,530 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:39:44,546 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 02:39:44,548 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:39:44,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:39:44,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:39:44,642 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 02:39:44,644 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:39:44,645 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:39:44,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:39:44,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:39:45,010 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:39:45,010 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:39:45,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:39:45,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:39:45,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:39:45,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:39:45,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:39:45,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:39:45,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:39:45,122 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 02:39:45,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:39:45,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:39:47,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:39:47,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:39:47,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:39:47,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:39:47,637 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 02:39:47,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:39:47,653 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:39:47,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:39:47,686 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 02:39:47,686 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:39:47,688 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:39:47,775 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:39:47,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:39:48,036 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:39:48,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:39:48,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:39:48,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:39:48,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:39:48,104 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:39:48,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:39:48,132 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:39:48,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:39:48,164 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 02:39:48,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:39:48,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 3786 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:39:50,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:39:50,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:39:51,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:39:51,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:39:51,336 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 02:39:51,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:39:51,354 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:39:51,408 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:39:51,427 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 02:39:51,427 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:39:51,430 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 02:39:51,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:39:51,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 02:39:51,805 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:39:51,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 02:39:51,821 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 02:39:51,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 02:39:51,849 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 02:39:51,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 02:39:51,882 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 02:39:51,895 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 02:39:51,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:39:51,934 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 02:39:51,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:39:51,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:39:51,992 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:39:52,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:39:52,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:39:52,033 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:39:52,048 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 02:39:52,048 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:39:52,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:39:52,079 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:39:52,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:52,130 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:39:52,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:52,209 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:52,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:52,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:52,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:52,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:52,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:52,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:52,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:52,474 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:39:52,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:52,512 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:52,530 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:52,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:52,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:52,584 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:52,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:52,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:52,638 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 02:39:52,641 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:39:52,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:39:52,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:39:53,040 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:39:53,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:39:53,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:39:53,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:39:53,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:39:53,104 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:39:53,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:39:53,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:39:53,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:39:53,226 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 02:39:53,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:39:53,308 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:39:54,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:39:54,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:39:54,627 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:39:54,698 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:39:54,717 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 02:39:54,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:39:54,755 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:39:54,839 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:39:54,924 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 02:39:54,924 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:39:54,926 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 02:39:55,028 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:39:55,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 02:39:55,339 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:39:55,339 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 02:39:55,352 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 02:39:55,367 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 02:39:55,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 02:39:55,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 02:39:55,415 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 02:39:55,428 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 02:39:55,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:39:55,463 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 02:39:55,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:39:55,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:39:55,522 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:39:55,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:39:55,551 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:39:55,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:39:55,579 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 02:39:55,579 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:39:55,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:39:55,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:39:55,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:55,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:39:55,698 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:55,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:55,770 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:55,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:55,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:55,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:55,905 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:55,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:39:55,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:55,992 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:39:56,017 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:56,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:56,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:56,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:56,091 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:56,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:56,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:56,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:39:56,161 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 02:39:56,164 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:39:56,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:39:56,492 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:39:56,562 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:39:56,563 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:39:56,579 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:39:56,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:39:56,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:39:56,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:39:56,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:39:56,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:39:56,683 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:39:56,703 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 02:39:56,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:39:56,749 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:39:59,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:39:59,354 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:39:59,538 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:39:59,711 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:39:59,727 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 02:39:59,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:39:59,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:39:59,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:39:59,793 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 02:39:59,795 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:39:59,796 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:39:59,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:40:00,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:40:00,408 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:40:00,410 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:40:00,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:40:00,459 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:40:00,477 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:40:00,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:40:00,528 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:40:00,548 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:40:00,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:40:00,605 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 02:40:00,630 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:40:00,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:40:03,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:40:03,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:40:03,571 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:40:03,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:40:03,724 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 02:40:03,724 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:40:03,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:40:03,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:40:03,802 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 02:40:03,803 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:40:03,805 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 02:40:03,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:40:04,135 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 02:40:04,207 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:40:04,209 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 02:40:04,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 02:40:04,239 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 02:40:04,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 02:40:04,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 02:40:04,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 02:40:04,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 02:40:04,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:40:04,331 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 02:40:04,359 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:40:04,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:40:04,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:40:04,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:40:04,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:40:04,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:40:04,451 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 02:40:04,451 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:40:04,467 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:40:04,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:40:04,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:04,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:40:04,566 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:04,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:04,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:04,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:04,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:04,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:04,778 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:04,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:04,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:04,859 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:40:04,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:04,897 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:04,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:04,928 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:04,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:04,966 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:04,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:05,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:05,016 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 02:40:05,019 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 02:40:05,099 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:40:05,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 02:40:05,391 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:40:05,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 02:40:05,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 02:40:05,420 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 02:40:05,433 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 02:40:05,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 02:40:05,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 02:40:05,477 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 02:40:05,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:40:05,509 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 02:40:05,535 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:40:05,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:40:05,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:40:05,579 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:40:05,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:40:05,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:40:05,618 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 02:40:05,619 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:40:05,632 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:40:05,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:40:05,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:05,693 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:40:05,732 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:05,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:05,801 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:05,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:05,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:05,913 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:05,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:05,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:06,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:06,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:40:06,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:06,073 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:06,089 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:06,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:06,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:06,142 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:06,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:06,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:06,194 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 02:40:06,197 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:40:06,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:40:06,520 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:40:06,591 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:40:06,591 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:40:06,607 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:40:06,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:40:06,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:40:06,683 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:40:06,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:40:06,714 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:40:06,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:40:06,749 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 02:40:06,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:40:06,788 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:40:08,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:40:09,239 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:40:09,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:40:09,520 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:40:09,536 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 02:40:09,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:40:09,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:40:09,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:40:09,614 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 02:40:09,615 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:40:09,617 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:40:10,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:40:10,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:40:11,059 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:40:11,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:40:11,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:40:11,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:40:11,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:40:11,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:40:11,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:40:11,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:40:11,206 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:40:11,236 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 02:40:11,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:40:11,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:40:12,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:40:12,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:40:12,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:40:13,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:40:13,101 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 02:40:13,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:40:13,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:40:13,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:40:13,168 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 02:40:13,170 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:40:13,171 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:40:13,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:40:13,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:40:13,555 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:40:13,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:40:13,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:40:13,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:40:13,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:40:13,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:40:13,637 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:40:13,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:40:13,665 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:40:13,683 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 02:40:13,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:40:13,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:40:16,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:40:16,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:40:16,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:40:16,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:40:16,831 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 02:40:16,831 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:40:16,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:40:16,887 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:40:16,905 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 02:40:16,907 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:40:16,914 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 02:40:17,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:40:17,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 02:40:17,326 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:40:17,326 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 02:40:17,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 02:40:17,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 02:40:17,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 02:40:17,385 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 02:40:17,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 02:40:17,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 02:40:17,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:40:17,444 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 02:40:17,471 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:40:17,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:40:17,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:40:17,517 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:40:17,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:40:17,545 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:40:17,559 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 02:40:17,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:40:17,573 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:40:17,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:40:17,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:17,632 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:40:17,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:17,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:17,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:17,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:17,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:17,857 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:17,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:17,931 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:40:17,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:17,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:40:18,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:18,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:18,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:18,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:18,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:18,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:18,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:18,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:40:18,147 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 02:40:18,148 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:40:18,150 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:40:18,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:40:18,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:40:18,595 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:40:18,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:40:18,609 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:40:18,622 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:40:18,635 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:40:18,653 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:40:18,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:40:18,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:40:18,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:19,377 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:40:19,378 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.23000001907. 2022-02-08 02:40:19,378 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:40:19,380 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:40:19,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:40:20,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:40:20,540 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:40:20,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:40:20,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:40:20,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:40:20,676 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:40:20,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:40:20,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:40:20,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:40:20,760 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:21,371 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:40:21,371 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:40:23,372 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:23,526 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:40:23,526 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:40:25,528 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:25,690 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:40:25,690 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.31299996376. 2022-02-08 02:40:25,690 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:40:25,693 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:40:26,007 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:40:26,331 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:40:26,421 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:40:26,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:40:26,438 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:40:26,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:40:26,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:40:26,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:40:26,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:40:26,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:40:26,609 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:27,263 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:40:27,263 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.5720000267. 2022-02-08 02:40:27,263 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:40:27,266 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:40:28,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:40:28,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:40:28,782 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:40:28,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:40:28,806 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:40:28,831 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:40:28,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:40:28,903 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:40:28,926 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:40:28,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:40:28,969 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:29,625 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:40:29,625 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 02:40:31,627 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:31,786 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:40:31,788 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.52400016785. 2022-02-08 02:40:31,788 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:40:31,789 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:40:31,905 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:40:32,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:40:32,239 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:40:32,239 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:40:32,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:40:32,285 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:40:32,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:40:32,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:40:32,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:40:32,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:40:32,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:32,984 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:40:32,984 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 02:40:34,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:35,525 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:40:35,525 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 02:40:37,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:37,936 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:40:37,936 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 02:40:39,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:40,391 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:40:40,391 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 02:40:42,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:42,703 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:40:42,703 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 02:40:44,704 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:45,207 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:40:45,207 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 02:40:47,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:47,647 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:40:47,648 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 5 queries submitted, 0 failed, 3810 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:40:49,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:49,973 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:40:49,973 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 02:40:51,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:52,188 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:40:52,188 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 02:40:54,188 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:54,451 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:40:54,453 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 02:40:56,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:57,153 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:40:57,153 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 02:40:59,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:40:59,558 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:40:59,559 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 02:41:01,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:41:01,943 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:41:01,943 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 02:41:03,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:41:04,259 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:41:04,259 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 02:41:06,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:41:06,434 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:41:06,437 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 34.6489999294. 2022-02-08 02:41:06,437 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:41:06,438 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:41:07,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:41:07,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:41:07,611 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:41:07,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:41:07,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:41:07,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:41:07,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:41:07,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:41:07,716 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:41:07,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:41:07,770 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:41:08,451 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:41:08,453 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.01499986649. 2022-02-08 02:41:08,453 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:41:08,454 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:41:09,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:41:09,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:41:09,865 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:41:09,865 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:41:09,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:41:09,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:41:09,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:41:09,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:41:09,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:41:10,005 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:41:10,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:41:10,716 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:41:10,717 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.26300001144. 2022-02-08 02:41:10,717 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:41:10,717 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:41:11,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:41:11,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:41:11,720 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:41:11,721 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:41:11,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:41:11,778 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:41:11,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:41:11,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:41:11,834 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:41:11,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:41:11,892 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:41:12,546 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:41:12,546 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.82999992371. 2022-02-08 02:41:12,546 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:41:12,549 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:41:13,433 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:41:13,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:41:14,101 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:41:14,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:41:14,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:41:14,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:41:14,204 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:41:14,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:41:14,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:41:14,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:41:14,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:41:15,098 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:41:15,098 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.55100011826. ############################################################ # # RAM FootPrint: 4171.0078125 # ############################################################ 2022-02-08 02:41:15,276 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 02:41:15,276 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 102 seconds. Reason: Iteration:2 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 3836 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 3863 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:42:57,282 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 02:42:57,282 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 02:43:02,286 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:43:03,015 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:03,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:43:04,002 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:04,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:43:04,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:43:04,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:43:04,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:43:04,119 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:43:04,178 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:43:04,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:43:04,334 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:43:04,334 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 02:43:04,431 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['104956 memcached\n'], []) 2022-02-08 02:43:04,431 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 02:43:04,492 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 02:43:04,515 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['104956 memcached\n'], []) 2022-02-08 02:43:04,515 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:43:04,516 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:43:05,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:06,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:43:06,542 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:06,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:43:06,561 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:43:06,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:43:06,619 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:43:06,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:43:06,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:43:06,711 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:43:06,732 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:43:06,732 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 02:43:06,894 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['12465 memcached\n'], []) 2022-02-08 02:43:06,895 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 02:43:06,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 02:43:06,993 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['12465 memcached\n'], []) 2022-02-08 02:43:06,994 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:43:06,997 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:43:07,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:08,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:43:09,006 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:09,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:43:09,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:43:09,073 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:43:09,089 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:43:09,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:43:09,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:43:09,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:43:09,217 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:43:09,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 02:43:09,286 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['1821 memcached\n'], []) 2022-02-08 02:43:09,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 02:43:09,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 02:43:09,345 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['1821 memcached\n'], []) 2022-02-08 02:43:09,345 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:43:09,346 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:43:10,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:11,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:43:11,483 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:11,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:43:11,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:43:11,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:43:11,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:43:11,599 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:43:11,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:43:11,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:43:11,684 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:43:11,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 02:43:11,750 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['39012 memcached\n'], []) 2022-02-08 02:43:11,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 02:43:11,793 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 02:43:11,816 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['39012 memcached\n'], []) 2022-02-08 02:43:11,816 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:43:11,819 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:43:12,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:13,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:43:14,424 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:14,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:43:14,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:43:14,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:43:14,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:43:14,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:43:14,528 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:43:14,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:43:14,601 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:43:14,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 02:43:14,681 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['32661 memcached\n'], []) 2022-02-08 02:43:14,683 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 02:43:14,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 02:43:14,737 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['32661 memcached\n'], []) 2022-02-08 02:43:14,737 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:43:14,740 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:43:16,091 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:16,665 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:43:16,895 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:16,897 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:43:16,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:43:16,984 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:43:16,999 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:43:17,017 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:43:17,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:43:17,061 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:43:17,075 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:43:17,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 02:43:17,134 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['73924 memcached\n'], []) 2022-02-08 02:43:17,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 02:43:17,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 02:43:17,217 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['73924 memcached\n'], []) 2022-02-08 02:43:17,217 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:43:17,219 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:43:18,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:19,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:43:19,211 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:19,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:43:19,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:43:19,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:43:19,272 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:43:19,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:43:19,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:43:19,341 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:43:19,358 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:43:19,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 02:43:19,414 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['67678 memcached\n'], []) 2022-02-08 02:43:19,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 02:43:19,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 02:43:19,467 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['67678 memcached\n'], []) 2022-02-08 02:43:19,467 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:43:19,469 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:43:20,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:20,971 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:43:21,104 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:21,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:43:21,122 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:43:21,145 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:43:21,181 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:43:21,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:43:21,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:43:21,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:43:21,266 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:43:21,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 02:43:21,312 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['1559 memcached\n'], []) 2022-02-08 02:43:21,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 02:43:21,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 02:43:21,365 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['1559 memcached\n'], []) 2022-02-08 02:43:21,365 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:43:21,368 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:43:22,367 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:23,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:43:23,226 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:23,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:43:23,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:43:23,280 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:43:23,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:43:23,331 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:43:23,349 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:43:23,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:43:23,404 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:43:23,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:43:23,482 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['81080 memcached\n'], []) 2022-02-08 02:43:23,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 02:43:23,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:43:23,552 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['81080 memcached\n'], []) 2022-02-08 02:43:23,553 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:43:23,555 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:43:23,640 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:23,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:43:23,944 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:23,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:43:23,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:43:23,977 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:43:23,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:43:24,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:43:24,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:43:24,043 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:43:24,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:43:24,073 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 02:43:24,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:43:24,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:26,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:43:26,244 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:26,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:26,515 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:43:26,530 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 02:43:26,530 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:43:26,543 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:26,591 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:43:26,608 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 02:43:26,608 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:43:26,611 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:43:26,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:26,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:43:27,003 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:27,010 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:43:27,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:43:27,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:43:27,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:43:27,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:43:27,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:43:27,099 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:43:27,112 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:43:27,128 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 02:43:27,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:43:27,168 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:29,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:43:30,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:30,278 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:30,423 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:43:30,437 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 02:43:30,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:43:30,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:30,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:43:30,513 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 02:43:30,515 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:43:30,516 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:43:30,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:30,801 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:43:30,864 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:30,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:43:30,882 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:43:30,895 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:43:30,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:43:30,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:43:30,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:43:30,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:43:30,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:43:30,990 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 02:43:31,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:43:31,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:33,619 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:43:33,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:34,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:34,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:43:34,213 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 02:43:34,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:43:34,227 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:34,288 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:43:34,301 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 02:43:34,302 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:43:34,303 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 02:43:34,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:34,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 02:43:34,677 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:34,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 02:43:34,693 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 02:43:34,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 02:43:34,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 02:43:34,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 02:43:34,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 02:43:34,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 02:43:34,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:43:34,796 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 02:43:34,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:43:34,864 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:34,880 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:43:34,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:34,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:34,918 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:43:34,931 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 02:43:34,931 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:43:34,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:34,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:43:34,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:35,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:43:35,043 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:35,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:35,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:35,145 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:35,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:35,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:35,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:35,295 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:35,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:35,344 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:43:35,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:35,377 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:35,395 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:35,413 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:35,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:35,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:35,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:35,482 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:35,497 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 02:43:35,500 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:43:35,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:35,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:43:35,907 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:35,908 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:43:35,926 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:43:35,941 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:43:35,954 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:43:35,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:43:35,997 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:43:36,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:43:36,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:43:36,068 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 02:43:36,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:43:36,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:37,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:43:37,236 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:37,318 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:37,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:43:37,414 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 02:43:37,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:43:37,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:37,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:43:37,515 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 02:43:37,515 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:43:37,516 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 02:43:37,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:37,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 02:43:37,901 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:37,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 02:43:37,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 02:43:37,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 02:43:37,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 02:43:37,977 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 02:43:37,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 02:43:38,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 02:43:38,017 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:43:38,032 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 02:43:38,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:43:38,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:38,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:43:38,099 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:38,112 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:38,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:43:38,142 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 02:43:38,142 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:43:38,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:38,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:43:38,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:38,219 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:43:38,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:38,295 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:38,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:38,367 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:38,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:38,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:38,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:38,512 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:38,545 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:38,563 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:43:38,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:38,599 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:38,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:38,635 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:38,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:38,673 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:38,691 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:38,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:38,727 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 02:43:38,730 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:43:38,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:39,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:43:39,125 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:39,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:43:39,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:43:39,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:43:39,176 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:43:39,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:43:39,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:43:39,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:43:39,234 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:43:39,250 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 02:43:39,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:43:39,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:41,517 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:43:41,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:42,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:42,186 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:43:42,207 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 02:43:42,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:43:42,221 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:42,255 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:43:42,269 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 02:43:42,269 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:43:42,270 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:43:42,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:42,607 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:43:42,677 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:42,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:43:42,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:43:42,709 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:43:42,724 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:43:42,755 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:43:42,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:43:42,795 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:43:42,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:43:42,858 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 02:43:42,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:43:42,905 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:45,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:43:45,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:45,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:46,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:43:46,019 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 02:43:46,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:43:46,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:46,086 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:43:46,108 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 02:43:46,108 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:43:46,109 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 02:43:46,201 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:46,461 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 02:43:46,535 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:46,535 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 02:43:46,551 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 02:43:46,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 02:43:46,581 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 02:43:46,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 02:43:46,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 02:43:46,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 02:43:46,645 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:43:46,663 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 02:43:46,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:43:46,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:46,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:43:46,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:46,760 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:46,778 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:43:46,792 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 02:43:46,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:43:46,808 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:46,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:43:46,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:46,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:43:46,918 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:46,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:47,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:47,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:47,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:47,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:47,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:47,201 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:47,236 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:47,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:43:47,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:47,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:47,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:47,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:47,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:47,369 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:47,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:47,405 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:47,423 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 02:43:47,424 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 02:43:47,502 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:47,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 02:43:47,818 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:47,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 02:43:47,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 02:43:47,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 02:43:47,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 02:43:47,880 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 02:43:47,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 02:43:47,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 02:43:47,928 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:43:47,944 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 02:43:47,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:43:47,984 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:47,999 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:43:48,015 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:48,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:48,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:43:48,056 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 02:43:48,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:43:48,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:48,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:43:48,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:48,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:43:48,176 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:48,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:48,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:48,285 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:48,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:48,359 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:48,397 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:48,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:43:48,469 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:48,489 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:43:48,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:48,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:48,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:48,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:48,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:48,599 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:48,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:48,640 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:43:48,655 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 02:43:48,658 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:43:48,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 3887 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:43:49,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:43:49,717 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:49,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:43:49,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:43:49,857 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:43:49,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:43:50,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:43:50,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:43:50,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:43:50,278 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:43:50,315 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 02:43:50,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:43:50,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:52,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:43:53,045 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:53,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:53,413 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:43:53,430 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 02:43:53,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:43:53,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:53,492 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:43:53,522 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 02:43:53,523 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:43:53,525 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:43:54,232 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:54,576 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:43:54,740 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:54,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:43:54,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:43:54,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:43:54,831 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:43:54,849 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:43:54,867 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:43:54,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:43:54,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:43:54,930 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 02:43:54,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:43:54,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:56,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:43:56,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:56,716 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:43:56,826 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:43:56,842 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 02:43:56,844 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:43:56,859 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:56,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:43:56,911 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 02:43:56,911 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:43:56,915 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:43:57,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:43:57,242 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:43:57,309 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:43:57,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:43:57,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:43:57,339 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:43:57,354 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:43:57,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:43:57,400 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:43:57,420 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:43:57,436 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:43:57,453 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 02:43:57,482 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:43:57,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:43:59,874 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:44:00,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:44:00,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:44:00,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:44:00,566 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 02:44:00,566 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:44:00,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:44:00,624 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:44:00,641 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 02:44:00,641 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:44:00,644 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 02:44:00,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:44:00,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 02:44:01,028 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:44:01,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 02:44:01,043 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 02:44:01,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 02:44:01,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 02:44:01,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 02:44:01,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 02:44:01,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 02:44:01,140 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:44:01,157 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 02:44:01,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:44:01,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:44:01,211 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:44:01,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:44:01,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:44:01,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:44:01,269 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 02:44:01,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:44:01,288 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:44:01,306 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:44:01,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1 2022-02-08 02:44:01,359 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1 | grep -v 'fatal remote' 2022-02-08 02:44:01,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1 2022-02-08 02:44:01,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1 2022-02-08 02:44:01,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1 2022-02-08 02:44:01,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1 2022-02-08 02:44:01,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1 2022-02-08 02:44:01,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1 2022-02-08 02:44:01,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1 2022-02-08 02:44:01,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1 2022-02-08 02:44:01,721 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:44:01,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:44:01,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:44:01,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:44:01,808 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:44:01,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:44:01,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:44:01,864 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:44:01,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:44:01,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:44:01,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:44:01,931 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:44:01,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:44:01,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:44:01,987 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:44:02,005 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:44:02,022 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:44:02,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:44:02,058 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:44:02,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:44:02,094 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 02:44:02,094 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:44:02,095 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:44:02,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:44:02,408 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:44:02,474 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:44:02,474 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:44:02,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:44:02,505 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:44:02,517 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:44:02,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:44:02,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:44:02,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:44:02,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:03,155 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:44:03,155 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.06199979782. 2022-02-08 02:44:03,155 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:44:03,157 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:44:03,408 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:44:04,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:44:04,165 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:44:04,165 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:44:04,255 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:44:04,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:44:04,352 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:44:04,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:44:04,400 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:44:04,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:44:04,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:05,678 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:05,680 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:44:07,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:07,914 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:07,914 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:44:09,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:10,279 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:10,280 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:44:12,280 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:12,526 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:12,526 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:44:14,528 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:14,986 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:14,986 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:44:16,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:17,150 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:17,151 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:44:19,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:19,322 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:44:19,322 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 16.1660001278. 2022-02-08 02:44:19,322 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:44:19,325 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:44:20,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:44:20,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:44:20,852 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:44:20,854 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:44:20,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:44:20,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:44:20,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:44:20,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:44:21,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:44:21,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:44:21,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:21,667 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:44:21,668 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.34599995613. 2022-02-08 02:44:21,668 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:44:21,671 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:44:22,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:44:23,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:44:23,430 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:44:23,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:44:23,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:44:23,642 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:44:23,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:44:23,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:44:23,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:44:23,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:44:23,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:24,730 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:24,732 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 02:44:26,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:26,890 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:44:26,890 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.22100019455. 2022-02-08 02:44:26,891 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:44:26,891 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:44:27,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:44:28,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:44:28,388 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:44:28,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:44:28,405 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:44:28,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:44:28,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:44:28,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:44:28,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:44:28,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:44:28,548 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:29,158 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:44:29,158 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.26899981499. 2022-02-08 02:44:29,160 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:44:29,161 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:44:29,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:44:30,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:44:30,285 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:44:30,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:44:30,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:44:30,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:44:30,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:44:30,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:44:30,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:44:30,676 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:44:30,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:32,240 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:32,240 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 02:44:34,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:34,517 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:34,519 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 02:44:36,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:37,052 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:37,052 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 02:44:39,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:39,315 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:39,315 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 02:44:41,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:41,660 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:41,660 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 02:44:43,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:44,036 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:44,036 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 02:44:46,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:46,576 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:46,576 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 02:44:48,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:48,881 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:48,881 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 5 queries submitted, 0 failed, 3910 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:44:50,882 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:51,164 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:51,164 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 02:44:53,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:53,631 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:53,632 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 02:44:55,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:55,818 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:44:55,818 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 26.6579999924. 2022-02-08 02:44:55,819 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:44:55,821 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:44:56,826 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:44:57,591 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:44:57,974 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:44:57,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:44:58,005 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:44:58,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:44:58,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:44:58,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:44:58,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:44:58,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:44:58,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:44:59,838 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:44:59,838 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 02:45:01,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:45:02,006 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:45:02,007 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.18899989128. 2022-02-08 02:45:02,007 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:45:02,009 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:45:02,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:45:02,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:45:03,026 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:45:03,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:45:03,040 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:45:03,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:45:03,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:45:03,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:45:03,114 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:45:03,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:45:03,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:45:03,690 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:45:03,691 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.68400001526. 2022-02-08 02:45:03,691 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:45:03,694 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:45:04,617 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:45:05,410 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:45:05,825 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:45:05,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:45:05,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:45:05,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:45:06,058 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:45:06,186 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:45:06,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:45:06,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:45:06,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:45:07,819 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:45:07,819 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 02:45:09,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:45:10,000 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:45:10,002 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 02:45:12,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:45:12,632 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:45:12,634 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 8.9430000782. ############################################################ # # RAM FootPrint: 4161.90234375 # ############################################################ 2022-02-08 02:45:12,674 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 02:45:12,674 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 88 seconds. Reason: Iteration:3 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 3936 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:46:40,680 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 02:46:40,680 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 02:46:45,683 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:46:46,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:46:47,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:46:47,601 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:46:47,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:46:47,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:46:47,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:46:47,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:46:47,865 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:46:47,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:46:47,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:46:48,023 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:46:48,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 02:46:48,088 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['105665 memcached\n'], []) 2022-02-08 02:46:48,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 02:46:48,211 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 02:46:48,233 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['105665 memcached\n'], []) 2022-02-08 02:46:48,234 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:46:48,236 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 5 queries submitted, 0 failed, 3963 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:46:49,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:46:49,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:46:49,881 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:46:49,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:46:49,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:46:49,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:46:49,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:46:49,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:46:49,992 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:46:50,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:46:50,088 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:46:50,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 02:46:50,148 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['13196 memcached\n'], []) 2022-02-08 02:46:50,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 02:46:50,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 02:46:50,211 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['13196 memcached\n'], []) 2022-02-08 02:46:50,213 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:46:50,216 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:46:51,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:46:52,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:46:52,283 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:46:52,285 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:46:52,306 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:46:52,326 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:46:52,346 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:46:52,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:46:52,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:46:52,443 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:46:52,486 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:46:52,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 02:46:52,559 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['2633 memcached\n'], []) 2022-02-08 02:46:52,561 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 02:46:52,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 02:46:52,638 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['2633 memcached\n'], []) 2022-02-08 02:46:52,638 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:46:52,641 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:46:53,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:46:54,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:46:54,223 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:46:54,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:46:54,288 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:46:54,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:46:54,349 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:46:54,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:46:54,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:46:54,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:46:54,441 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:46:54,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 02:46:54,535 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['39734 memcached\n'], []) 2022-02-08 02:46:54,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 02:46:54,566 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 02:46:54,594 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['39734 memcached\n'], []) 2022-02-08 02:46:54,595 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:46:54,596 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:46:55,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:46:56,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:46:56,375 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:46:56,377 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:46:56,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:46:56,413 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:46:56,431 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:46:56,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:46:56,469 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:46:56,499 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:46:56,515 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:46:56,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 02:46:56,575 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['33540 memcached\n'], []) 2022-02-08 02:46:56,576 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 02:46:56,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 02:46:56,630 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['33540 memcached\n'], []) 2022-02-08 02:46:56,630 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:46:56,631 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:46:57,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:46:58,354 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:46:58,655 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:46:58,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:46:58,673 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:46:58,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:46:58,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:46:58,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:46:58,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:46:58,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:46:58,825 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:46:58,826 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 02:46:58,884 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['74638 memcached\n'], []) 2022-02-08 02:46:58,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 02:46:58,913 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 02:46:58,934 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['74638 memcached\n'], []) 2022-02-08 02:46:58,934 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:46:58,937 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:47:00,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:00,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:47:01,046 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:01,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:47:01,061 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:47:01,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:47:01,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:47:01,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:47:01,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:47:01,168 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:47:01,201 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:47:01,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 02:47:01,292 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['68385 memcached\n'], []) 2022-02-08 02:47:01,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 02:47:01,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 02:47:01,336 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['68385 memcached\n'], []) 2022-02-08 02:47:01,336 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:47:01,338 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:47:02,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:02,793 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:47:02,977 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:02,979 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:47:03,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:47:03,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:47:03,091 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:47:03,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:47:03,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:47:03,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:47:03,230 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:47:03,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 02:47:03,296 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['2281 memcached\n'], []) 2022-02-08 02:47:03,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 02:47:03,326 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 02:47:03,349 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['2281 memcached\n'], []) 2022-02-08 02:47:03,349 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:47:03,351 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:47:03,724 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:04,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:47:04,247 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:04,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:47:04,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:47:04,290 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:47:04,306 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:47:04,326 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:47:04,349 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:47:04,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:47:04,411 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:47:04,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:47:04,471 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['81751 memcached\n'], []) 2022-02-08 02:47:04,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 02:47:04,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:47:04,523 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['81751 memcached\n'], []) 2022-02-08 02:47:04,523 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:47:04,526 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:47:04,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:04,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:47:04,898 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:04,900 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:47:04,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:47:04,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:47:04,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:47:04,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:47:04,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:47:04,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:47:05,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:47:05,023 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 02:47:05,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:47:05,061 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:06,834 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:47:07,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:07,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:07,334 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:47:07,346 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 02:47:07,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:47:07,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:07,408 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:47:07,423 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 02:47:07,424 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:47:07,426 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:47:07,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:07,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:47:07,779 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:07,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:47:07,793 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:47:07,808 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:47:07,822 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:47:07,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:47:07,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:47:07,867 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:47:07,880 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:47:07,895 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 02:47:07,920 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:47:07,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:10,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:47:10,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:10,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:10,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:47:10,719 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 02:47:10,719 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:47:10,732 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:10,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:47:10,796 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 02:47:10,796 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:47:10,798 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:47:10,869 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:11,076 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:47:11,134 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:11,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:47:11,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:47:11,165 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:47:11,178 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:47:11,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:47:11,209 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:47:11,221 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:47:11,236 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:47:11,250 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 02:47:11,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:47:11,290 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:13,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:47:13,979 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:14,160 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:14,306 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:47:14,321 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 02:47:14,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:47:14,334 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:14,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:47:14,411 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 02:47:14,413 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:47:14,414 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 02:47:14,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:14,714 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 02:47:14,779 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:14,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 02:47:14,795 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 02:47:14,806 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 02:47:14,822 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 02:47:14,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 02:47:14,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 02:47:14,865 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 02:47:14,877 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:47:14,891 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 02:47:14,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:47:14,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:14,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:47:14,966 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:14,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:14,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:47:15,006 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 02:47:15,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:47:15,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:15,033 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:47:15,048 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:15,061 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:47:15,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:15,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:15,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:15,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:15,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:15,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:15,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:15,168 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:15,181 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:15,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:47:15,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:15,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:15,244 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:15,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:15,275 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:15,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:15,303 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:15,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:15,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:15,349 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:47:15,367 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:15,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:15,400 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:15,415 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:15,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:15,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:15,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:15,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:15,513 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 02:47:15,515 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:47:15,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:15,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:47:15,957 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:15,959 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:47:16,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:47:16,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:47:16,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:47:16,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:47:16,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:47:16,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:47:16,374 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:47:16,430 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 02:47:16,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:47:16,543 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:17,609 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:47:17,903 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:18,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:18,397 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:47:18,549 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 02:47:18,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:47:18,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:18,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:47:18,986 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 02:47:18,986 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:47:18,989 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 02:47:19,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:19,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 02:47:19,364 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:19,364 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 02:47:19,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 02:47:19,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 02:47:19,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 02:47:19,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 02:47:19,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 02:47:19,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 02:47:19,471 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:47:19,487 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 02:47:19,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:47:19,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:19,545 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:47:19,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:19,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:19,589 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:47:19,605 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 02:47:19,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:47:19,622 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:19,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:47:19,655 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:19,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:47:19,686 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:19,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:19,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:19,739 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:19,757 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:19,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:19,788 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:19,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:19,821 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:19,836 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:47:19,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:19,872 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:19,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:19,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:19,920 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:19,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:19,954 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:19,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:19,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:20,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:47:20,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:20,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:20,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:20,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:20,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:20,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:20,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:20,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:20,171 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 02:47:20,174 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:47:20,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:20,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:47:20,536 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:20,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:47:20,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:47:20,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:47:20,579 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:47:20,596 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:47:20,609 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:47:20,624 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:47:20,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:47:20,674 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 02:47:20,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:47:20,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:23,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:47:23,522 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:23,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:23,841 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:47:23,855 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 02:47:23,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:47:23,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:23,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:47:23,914 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 02:47:23,914 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:47:23,917 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:47:23,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:24,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:47:24,286 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:24,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:47:24,303 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:47:24,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:47:24,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:47:24,354 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:47:24,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:47:24,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:47:24,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:47:24,417 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 02:47:24,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:47:24,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:26,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:47:26,933 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:27,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:27,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:47:27,255 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 02:47:27,255 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:47:27,278 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:27,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:47:27,345 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 02:47:27,345 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:47:27,346 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 02:47:27,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:27,653 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 02:47:27,713 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:27,714 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 02:47:27,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 02:47:27,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 02:47:27,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 02:47:27,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 02:47:27,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 02:47:27,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 02:47:27,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:47:27,828 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 02:47:27,864 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:47:27,880 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:27,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:47:27,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:27,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:27,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:47:27,957 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 02:47:27,959 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:47:27,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:27,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:47:28,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:28,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:47:28,045 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:28,063 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:28,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:28,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:28,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:28,135 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:28,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:28,165 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:28,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:28,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:47:28,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:28,229 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:28,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:28,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:28,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:28,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:28,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:28,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:28,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:28,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:47:28,390 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:28,410 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:28,428 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:28,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:28,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:28,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:28,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:28,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:28,539 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 02:47:28,542 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 02:47:28,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:28,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 02:47:28,928 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:28,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 02:47:28,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 02:47:28,954 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 02:47:28,969 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 02:47:28,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 02:47:29,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 02:47:29,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 02:47:29,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:47:29,043 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 02:47:29,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:47:29,086 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:29,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:47:29,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:29,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:29,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:47:29,154 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 02:47:29,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:47:29,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:29,181 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:47:29,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:29,209 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:47:29,224 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:29,242 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:29,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:29,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:29,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:29,306 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:29,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:29,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:29,354 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:29,369 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:47:29,387 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:29,403 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:29,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:29,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:29,448 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:29,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:29,479 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:29,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:29,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:29,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:47:29,543 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:29,561 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:29,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:29,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:29,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:29,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:29,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:29,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:29,681 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 02:47:29,684 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:47:29,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:30,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:47:30,094 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:30,094 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:47:30,112 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:47:30,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:47:30,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:47:30,181 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:47:30,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:47:30,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:47:30,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:47:30,239 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 02:47:30,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:47:30,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:32,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:47:32,897 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:33,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:33,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:47:33,226 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 02:47:33,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:47:33,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:33,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:47:33,296 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 02:47:33,296 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:47:33,299 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:47:34,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:34,454 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:47:34,684 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:34,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:47:34,701 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:47:34,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:47:34,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:47:34,772 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:47:34,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:47:34,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:47:34,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:47:34,854 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 02:47:34,895 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:47:34,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:36,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:47:36,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:36,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:36,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:47:36,668 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 02:47:36,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:47:36,683 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:36,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:47:36,739 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 02:47:36,739 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:47:36,740 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:47:36,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:37,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:47:37,131 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:37,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:47:37,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:47:37,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:47:37,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:47:37,191 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:47:37,204 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:47:37,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:47:37,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:47:37,246 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 02:47:37,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:47:37,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:39,619 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:47:39,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:40,076 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:40,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:47:40,246 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 02:47:40,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:47:40,260 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:40,295 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:47:40,311 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 02:47:40,311 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:47:40,312 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 02:47:40,397 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:40,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 02:47:40,688 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:40,688 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 02:47:40,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 02:47:40,719 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 02:47:40,734 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 02:47:40,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 02:47:40,767 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 02:47:40,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 02:47:40,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:47:40,815 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 02:47:40,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:47:40,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:40,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:47:40,908 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:40,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:47:40,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:47:40,970 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 02:47:40,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:47:40,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:47:41,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:47:41,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:41,035 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:47:41,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:41,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:41,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:41,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:41,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:41,135 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:41,153 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:41,170 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:47:41,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:41,211 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:47:41,234 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:41,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:41,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:41,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:41,298 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:41,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:41,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:41,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:47:41,364 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:41,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:47:41,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:41,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:41,431 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:41,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:41,467 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:41,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:41,505 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:41,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:47:41,539 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 02:47:41,540 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:47:41,542 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:47:41,640 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:41,864 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:47:41,927 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:41,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:47:41,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:47:41,959 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:47:41,979 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:47:41,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:47:42,017 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:47:42,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:47:42,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:47:42,540 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:47:42,540 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.0. 2022-02-08 02:47:42,540 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:47:42,542 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:47:42,683 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:47:42,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:47:43,131 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:47:43,132 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:47:43,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:47:43,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:47:43,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:47:43,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:47:43,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:47:43,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:47:43,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:47:43,970 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:47:43,971 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:47:45,971 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:47:46,160 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:47:46,161 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:47:48,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:47:48,548 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:47:48,548 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 3989 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:47:50,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:47:50,763 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:47:50,763 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:47:52,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:47:53,016 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:47:53,016 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:47:55,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:47:55,433 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:47:55,433 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:47:57,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:47:58,013 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:47:58,013 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:48:00,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:00,278 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:48:00,278 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:48:02,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:02,535 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:48:02,535 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:48:04,535 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:04,786 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:48:04,788 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:48:06,788 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:07,118 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:48:07,119 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:48:09,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:09,325 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:48:09,325 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:48:11,326 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:11,522 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:48:11,522 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:48:13,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:13,711 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:48:13,711 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:48:15,711 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:15,877 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:48:15,877 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 33.3360002041. 2022-02-08 02:48:15,877 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:48:15,878 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:48:16,624 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:48:17,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:48:17,115 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:48:17,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:48:17,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:48:17,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:48:17,227 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:48:17,262 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:48:17,288 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:48:17,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:48:17,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:18,082 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:48:18,082 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 02:48:20,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:20,288 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:48:20,289 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.41199994087. 2022-02-08 02:48:20,289 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:48:20,290 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:48:21,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:48:21,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:48:21,536 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:48:21,538 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:48:21,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:48:21,571 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:48:21,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:48:21,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:48:21,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:48:21,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:48:21,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:22,338 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:48:22,338 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.04999995232. 2022-02-08 02:48:22,338 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:48:22,341 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:48:23,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:48:23,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:48:24,134 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:48:24,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:48:24,288 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:48:24,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:48:24,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:48:24,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:48:24,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:48:24,732 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:48:24,749 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:25,891 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:48:25,891 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 02:48:27,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:28,053 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:48:28,055 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.71499991417. 2022-02-08 02:48:28,055 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:48:28,056 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:48:28,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:48:29,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:48:29,318 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:48:29,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:48:29,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:48:29,374 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:48:29,395 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:48:29,426 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:48:29,454 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:48:29,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:48:29,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:30,144 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:48:30,144 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.08899998665. 2022-02-08 02:48:30,144 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:48:30,145 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:48:31,239 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:48:31,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:48:32,121 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:48:32,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:48:32,257 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:48:32,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:48:32,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:48:32,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:48:32,660 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:48:32,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:48:32,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:34,263 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:48:34,263 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 02:48:36,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:36,601 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:48:36,601 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 02:48:38,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:38,775 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:48:38,776 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 8.63199996948. 2022-02-08 02:48:38,776 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:48:38,778 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:48:39,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:48:39,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:48:39,855 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:48:39,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:48:39,874 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:48:39,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:48:39,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:48:39,928 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:48:39,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:48:39,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:48:40,007 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:40,539 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:48:40,539 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.76399993896. 2022-02-08 02:48:40,540 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:48:40,542 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:48:40,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:48:41,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:48:41,730 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:48:41,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:48:41,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:48:41,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:48:42,135 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:48:42,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:48:42,484 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:48:42,576 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:48:42,676 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:44,174 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:48:44,174 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 02:48:46,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:46,592 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:48:46,592 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 02:48:48,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:48,921 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:48:48,921 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 5 queries submitted, 0 failed, 4012 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:48:50,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:48:51,138 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:48:51,138 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 10.5980000496. ############################################################ # # RAM FootPrint: 4161.8828125 # ############################################################ 2022-02-08 02:48:51,424 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 02:48:51,424 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 82 seconds. Reason: Iteration:4 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 4035 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:50:13,428 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 02:50:13,430 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 02:50:18,431 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:50:19,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:19,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:50:20,075 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:20,076 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:50:20,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:50:20,127 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:50:20,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:50:20,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:50:20,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:50:20,344 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:50:20,417 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:50:20,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 02:50:20,542 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['106337 memcached\n'], []) 2022-02-08 02:50:20,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 02:50:20,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 02:50:20,602 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['106337 memcached\n'], []) 2022-02-08 02:50:20,602 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:50:20,605 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:50:21,469 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:21,903 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:50:22,243 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:22,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:50:22,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:50:22,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:50:22,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:50:22,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:50:22,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:50:22,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:50:22,423 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:50:22,423 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 02:50:22,493 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['13984 memcached\n'], []) 2022-02-08 02:50:22,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 02:50:22,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 02:50:22,562 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['13984 memcached\n'], []) 2022-02-08 02:50:22,562 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:50:22,565 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:50:23,234 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:23,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:50:23,934 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:23,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:50:23,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:50:24,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:50:24,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:50:24,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:50:24,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:50:24,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:50:24,165 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:50:24,165 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 02:50:24,246 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['3436 memcached\n'], []) 2022-02-08 02:50:24,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 02:50:24,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 02:50:24,308 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['3436 memcached\n'], []) 2022-02-08 02:50:24,308 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:50:24,309 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:50:25,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:25,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:50:25,910 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:25,910 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:50:25,933 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:50:25,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:50:25,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:50:25,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:50:26,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:50:26,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:50:26,062 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:50:26,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 02:50:26,135 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['40426 memcached\n'], []) 2022-02-08 02:50:26,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 02:50:26,168 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 02:50:26,193 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['40426 memcached\n'], []) 2022-02-08 02:50:26,193 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:50:26,197 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:50:27,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:27,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:50:28,026 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:28,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:50:28,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:50:28,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:50:28,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:50:28,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:50:28,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:50:28,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:50:28,346 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:50:28,346 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 02:50:28,421 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['34249 memcached\n'], []) 2022-02-08 02:50:28,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 02:50:28,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 02:50:28,470 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['34249 memcached\n'], []) 2022-02-08 02:50:28,471 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:50:28,473 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:50:29,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:29,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:50:29,898 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:29,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:50:29,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:50:29,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:50:29,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:50:29,982 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:50:30,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:50:30,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:50:30,065 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:50:30,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 02:50:30,125 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['75437 memcached\n'], []) 2022-02-08 02:50:30,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 02:50:30,176 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 02:50:30,206 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['75437 memcached\n'], []) 2022-02-08 02:50:30,206 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:50:30,207 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:50:31,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:32,209 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:50:32,381 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:32,382 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:50:32,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:50:32,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:50:32,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:50:32,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:50:32,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:50:32,499 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:50:32,520 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:50:32,520 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 02:50:32,579 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['69108 memcached\n'], []) 2022-02-08 02:50:32,579 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 02:50:32,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 02:50:32,625 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['69108 memcached\n'], []) 2022-02-08 02:50:32,625 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:50:32,628 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:50:33,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:33,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:50:34,045 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:34,045 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:50:34,066 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:50:34,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:50:34,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:50:34,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:50:34,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:50:34,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:50:34,190 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:50:34,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 02:50:34,266 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['2996 memcached\n'], []) 2022-02-08 02:50:34,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 02:50:34,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 02:50:34,318 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['2996 memcached\n'], []) 2022-02-08 02:50:34,318 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:50:34,319 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:50:34,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:35,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:50:35,203 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:35,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:50:35,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:50:35,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:50:35,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:50:35,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:50:35,306 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:50:35,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:50:35,355 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:50:35,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:50:35,430 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['82486 memcached\n'], []) 2022-02-08 02:50:35,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 02:50:35,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:50:35,480 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['82486 memcached\n'], []) 2022-02-08 02:50:35,480 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:50:35,482 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:50:35,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:35,785 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:50:35,851 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:35,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:50:35,867 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:50:35,880 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:50:35,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:50:35,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:50:35,928 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:50:35,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:50:35,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:50:35,973 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 02:50:35,997 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:50:36,012 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:37,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:50:38,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:38,288 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:38,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:50:38,444 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 02:50:38,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:50:38,469 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:38,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:50:38,525 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 02:50:38,526 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:50:38,528 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:50:38,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:38,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:50:38,874 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:38,874 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:50:38,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:50:38,900 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:50:38,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:50:38,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:50:38,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:50:38,954 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:50:38,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:50:38,983 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 02:50:39,010 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:50:39,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:41,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:50:41,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:41,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:42,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:50:42,092 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 02:50:42,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:50:42,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:42,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:50:42,167 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 02:50:42,167 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:50:42,170 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:50:42,255 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:42,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:50:42,515 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:42,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:50:42,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:50:42,540 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:50:42,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:50:42,571 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:50:42,584 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:50:42,596 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:50:42,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:50:42,627 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 02:50:42,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:50:42,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:45,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:50:45,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:45,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:46,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:50:46,086 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 02:50:46,086 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:50:46,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:46,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:50:46,171 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 02:50:46,171 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:50:46,174 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 02:50:46,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:46,489 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 02:50:46,569 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:46,576 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 02:50:46,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 02:50:46,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 02:50:46,622 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 02:50:46,640 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 02:50:46,653 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 02:50:46,665 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 02:50:46,681 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:50:46,697 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 02:50:46,721 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:50:46,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:46,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:50:46,767 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:46,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:46,795 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:50:46,808 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 02:50:46,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:50:46,822 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:46,836 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:50:46,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:46,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:50:46,887 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:46,903 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:46,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:46,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:46,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:46,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:46,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:47,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:47,038 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:47,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:50:47,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:47,091 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:47,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:47,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:47,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:47,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:47,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:47,188 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:47,204 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:47,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:50:47,237 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:47,255 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:47,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:47,288 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:47,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:47,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:47,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:47,357 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:47,374 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 02:50:47,375 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:50:47,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:47,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:50:47,775 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:47,775 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:50:47,790 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:50:47,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:50:47,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:50:47,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:50:47,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:50:47,872 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:50:47,887 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:50:47,901 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 02:50:47,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:50:47,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:48,961 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:50:49,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 4056 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:50:49,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:49,260 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:50:49,275 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 02:50:49,275 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:50:49,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:49,326 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:50:49,344 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 02:50:49,344 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:50:49,345 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 02:50:49,420 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:49,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 02:50:49,711 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:49,711 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 02:50:49,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 02:50:49,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 02:50:49,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 02:50:49,772 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 02:50:49,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 02:50:49,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 02:50:49,834 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:50:49,851 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 02:50:49,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:50:49,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:49,905 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:50:49,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:49,936 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:49,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:50:49,970 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 02:50:49,971 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:50:49,987 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:50,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:50:50,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:50,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:50:50,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:50,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:50,109 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:50,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:50,147 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:50,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:50,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:50,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:50,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:50,229 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:50:50,244 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:50,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:50,278 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:50,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:50,308 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:50,323 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:50,341 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:50,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:50,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:50,392 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:50:50,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:50,428 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:50,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:50,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:50,482 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:50,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:50,517 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:50,535 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:50,552 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 02:50:50,555 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:50:50,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:50,829 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:50:50,888 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:50,890 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:50:50,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:50:50,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:50:50,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:50:50,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:50:50,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:50:50,982 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:50:50,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:50:51,012 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 02:50:51,038 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:50:51,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:53,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:50:53,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:54,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:54,255 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:50:54,269 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 02:50:54,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:50:54,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:54,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:50:54,339 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 02:50:54,341 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:50:54,342 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:50:54,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:54,655 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:50:54,727 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:54,729 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:50:54,744 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:50:54,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:50:54,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:50:54,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:50:54,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:50:54,829 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:50:54,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:50:54,871 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 02:50:54,895 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:50:54,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:57,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:50:57,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:57,614 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:57,752 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:50:57,769 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 02:50:57,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:50:57,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:57,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:50:57,848 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 02:50:57,848 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:50:57,849 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 02:50:57,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:58,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 02:50:58,292 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:58,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 02:50:58,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 02:50:58,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 02:50:58,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 02:50:58,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 02:50:58,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 02:50:58,395 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 02:50:58,410 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:50:58,426 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 02:50:58,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:50:58,467 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:58,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:50:58,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:58,515 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:58,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:50:58,546 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 02:50:58,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:50:58,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:58,584 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:50:58,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:58,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:50:58,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:58,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:58,683 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:58,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:58,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:58,747 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:58,767 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:58,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:58,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:58,831 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:50:58,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:58,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:58,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:58,908 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:58,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:58,946 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:58,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:58,982 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:50:59,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:59,020 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:50:59,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:59,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:59,079 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:59,099 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:59,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:59,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:59,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:59,183 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:50:59,201 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 02:50:59,204 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 02:50:59,295 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:50:59,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 02:50:59,576 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:50:59,576 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 02:50:59,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 02:50:59,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 02:50:59,617 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 02:50:59,635 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 02:50:59,650 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 02:50:59,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 02:50:59,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:50:59,693 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 02:50:59,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:50:59,732 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:59,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:50:59,760 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:59,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:50:59,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:50:59,802 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 02:50:59,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:50:59,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:50:59,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:50:59,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:59,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:50:59,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:59,913 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:59,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:59,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:59,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:50:59,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:51:00,012 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:51:00,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:51:00,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:00,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:51:00,089 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:00,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:00,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:00,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:00,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:00,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:00,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:00,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:00,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:00,236 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:51:00,255 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:00,272 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:00,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:00,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:00,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:00,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:00,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:00,372 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:00,388 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 02:51:00,390 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:51:00,484 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:51:00,716 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:51:00,778 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:51:00,778 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:51:00,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:51:00,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:51:00,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:51:00,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:51:00,849 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:51:00,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:51:00,905 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:51:00,960 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 02:51:00,994 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:51:01,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:51:03,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:51:03,482 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:51:03,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:51:03,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:51:03,801 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 02:51:03,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:51:03,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:51:03,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:51:03,878 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 02:51:03,878 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:51:03,881 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:51:04,607 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:51:05,007 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:51:05,104 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:51:05,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:51:05,142 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:51:05,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:51:05,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:51:05,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:51:05,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:51:05,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:51:05,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:51:05,319 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 02:51:05,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:51:05,364 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:51:06,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:51:07,168 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:51:07,293 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:51:07,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:51:07,428 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 02:51:07,428 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:51:07,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:51:07,474 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:51:07,492 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 02:51:07,493 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:51:07,494 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:51:07,589 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:51:07,834 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:51:07,904 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:51:07,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:51:07,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:51:07,936 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:51:07,951 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:51:07,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:51:07,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:51:08,007 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:51:08,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:51:08,039 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 02:51:08,063 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:51:08,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:51:10,357 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:51:10,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:51:10,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:51:11,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:51:11,042 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 02:51:11,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:51:11,061 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:51:11,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:51:11,124 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 02:51:11,124 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:51:11,125 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 02:51:11,221 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:51:11,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 02:51:11,528 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:51:11,528 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 02:51:11,545 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 02:51:11,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 02:51:11,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 02:51:11,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 02:51:11,602 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 02:51:11,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 02:51:11,630 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:51:11,645 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 02:51:11,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:51:11,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:51:11,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:51:11,714 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:51:11,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:51:11,742 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:51:11,756 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 02:51:11,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:51:11,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:51:11,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:51:11,806 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:51:11,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:51:11,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:51:11,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:51:11,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:51:11,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:51:11,941 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:51:11,964 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:51:11,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:51:12,005 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:51:12,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:12,040 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:51:12,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:12,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:12,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:12,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:12,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:12,140 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:12,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:12,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:51:12,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:12,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:51:12,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:12,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:12,265 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:12,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:12,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:12,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:12,341 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:12,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:51:12,390 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 02:51:12,391 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:51:12,392 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:51:12,466 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:51:12,691 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:51:12,759 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:51:12,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:51:12,775 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:51:12,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:51:12,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:51:12,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:51:12,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:51:12,865 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:51:12,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:13,401 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:51:13,403 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.01200008392. 2022-02-08 02:51:13,403 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:51:13,404 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:51:13,568 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:51:13,826 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:51:13,926 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:51:13,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:51:13,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:51:13,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:51:13,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:51:14,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:51:14,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:51:14,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:51:14,198 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:14,900 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:51:14,900 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:51:16,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:17,125 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:51:17,125 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:51:19,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:19,476 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:51:19,476 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:51:21,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:21,851 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:51:21,851 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:51:23,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:24,095 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:51:24,095 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:51:26,096 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:26,359 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:51:26,361 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:51:28,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:28,598 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:51:28,598 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:51:30,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:30,779 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:51:30,780 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:51:32,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:33,006 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:51:33,006 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:51:35,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:35,187 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:51:35,187 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:51:37,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:37,414 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:51:37,414 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:51:39,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:39,572 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:51:39,572 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:51:41,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:41,776 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:51:41,776 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:51:43,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:43,964 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:51:43,964 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 30.5619997978. 2022-02-08 02:51:43,964 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:51:43,967 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:51:44,405 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:51:44,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:51:45,007 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:51:45,007 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:51:45,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:51:45,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:51:45,081 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:51:45,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:51:45,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:51:45,176 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:51:45,209 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:45,901 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:51:45,901 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 02:51:47,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:48,101 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:51:48,101 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 5 queries submitted, 0 failed, 4080 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:51:50,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:50,328 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:51:50,328 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.36400008202. 2022-02-08 02:51:50,328 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:51:50,331 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:51:50,997 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:51:51,385 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:51:51,539 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:51:51,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:51:51,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:51:51,584 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:51:51,607 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:51:51,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:51:51,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:51:51,698 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:51:51,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:52,453 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:51:52,453 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.12400007248. 2022-02-08 02:51:52,453 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:51:52,456 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:51:53,431 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:51:53,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:51:54,099 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:51:54,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:51:54,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:51:54,158 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:51:54,232 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:51:54,272 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:51:54,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:51:54,354 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:51:54,400 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:55,084 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:51:55,085 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.63100004196. 2022-02-08 02:51:55,085 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:51:55,086 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:51:55,788 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:51:56,262 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:51:56,424 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:51:56,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:51:56,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:51:56,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:51:56,581 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:51:56,635 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:51:56,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:51:56,829 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:51:56,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:51:58,161 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:51:58,161 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 02:52:00,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:52:00,355 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:52:00,355 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.27099990845. 2022-02-08 02:52:00,357 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:52:00,358 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:52:01,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:52:01,602 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:52:02,108 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:52:02,109 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:52:02,224 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:52:02,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:52:02,362 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:52:02,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:52:02,533 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:52:02,701 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:52:02,867 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:52:04,187 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:52:04,187 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 02:52:06,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:52:06,569 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:52:06,569 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 02:52:08,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:52:08,736 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:52:08,736 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 8.3789999485. 2022-02-08 02:52:08,736 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:52:08,737 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:52:09,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:52:09,581 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:52:09,710 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:52:09,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:52:09,729 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:52:09,752 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:52:09,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:52:09,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:52:09,798 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:52:09,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:52:09,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:52:10,388 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:52:10,388 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.65199995041. 2022-02-08 02:52:10,390 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:52:10,391 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:52:11,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:52:12,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:52:12,851 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:52:12,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:52:12,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:52:12,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:52:13,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:52:13,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:52:13,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:52:13,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:52:13,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:52:13,940 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:52:13,940 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.55099987984. ############################################################ # # RAM FootPrint: 4161.62890625 # ############################################################ 2022-02-08 02:52:14,086 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 02:52:14,086 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 62 seconds. Reason: Iteration:5 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 4106 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:53:16,092 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 02:53:16,094 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 02:53:21,096 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:53:21,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:53:22,397 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:53:22,694 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:53:22,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:53:22,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:53:22,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:53:22,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:53:22,844 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:53:22,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:53:22,954 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:53:23,039 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:53:23,040 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 02:53:23,200 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['107037 memcached\n'], []) 2022-02-08 02:53:23,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 02:53:23,237 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 02:53:23,276 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['107037 memcached\n'], []) 2022-02-08 02:53:23,276 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:53:23,282 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:53:24,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:53:24,678 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:53:24,986 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:53:24,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:53:25,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:53:25,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:53:25,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:53:25,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:53:25,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:53:25,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:53:25,198 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:53:25,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 02:53:25,279 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['14809 memcached\n'], []) 2022-02-08 02:53:25,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 02:53:25,311 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 02:53:25,342 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['14809 memcached\n'], []) 2022-02-08 02:53:25,342 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:53:25,345 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:53:26,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:53:26,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:53:26,832 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:53:26,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:53:26,854 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:53:26,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:53:26,897 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:53:26,918 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:53:26,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:53:27,040 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:53:27,059 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:53:27,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 02:53:27,130 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['4206 memcached\n'], []) 2022-02-08 02:53:27,130 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 02:53:27,170 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 02:53:27,203 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['4206 memcached\n'], []) 2022-02-08 02:53:27,203 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:53:27,210 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:53:28,155 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:53:28,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:53:28,816 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:53:28,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:53:28,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:53:28,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:53:28,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:53:28,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:53:28,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:53:28,982 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:53:29,013 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:53:29,015 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 02:53:29,084 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['41126 memcached\n'], []) 2022-02-08 02:53:29,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 02:53:29,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 02:53:29,144 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['41126 memcached\n'], []) 2022-02-08 02:53:29,144 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:53:29,147 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:53:29,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:53:30,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:53:30,628 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:53:30,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:53:30,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:53:30,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:53:30,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:53:30,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:53:30,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:53:30,765 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:53:30,805 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:53:30,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 02:53:30,864 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['34917 memcached\n'], []) 2022-02-08 02:53:30,864 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 02:53:30,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 02:53:30,921 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['34917 memcached\n'], []) 2022-02-08 02:53:30,921 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:53:30,924 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:53:31,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:53:32,369 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:53:32,565 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:53:32,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:53:32,579 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:53:32,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:53:32,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:53:32,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:53:32,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:53:32,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:53:32,713 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:53:32,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 02:53:32,788 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['76139 memcached\n'], []) 2022-02-08 02:53:32,788 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 02:53:32,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 02:53:32,838 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['76139 memcached\n'], []) 2022-02-08 02:53:32,838 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:53:32,841 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:53:33,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:53:34,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:53:34,453 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:53:34,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:53:34,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:53:34,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:53:34,505 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:53:34,530 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:53:34,543 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:53:34,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:53:34,585 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:53:34,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 02:53:34,657 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['69784 memcached\n'], []) 2022-02-08 02:53:34,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 02:53:34,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 02:53:34,720 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['69784 memcached\n'], []) 2022-02-08 02:53:34,720 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:53:34,723 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:53:35,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:53:35,752 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:53:35,910 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:53:35,910 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:53:35,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:53:35,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:53:35,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:53:35,992 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:53:36,010 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:53:36,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:53:36,052 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:53:36,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 02:53:36,112 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['3659 memcached\n'], []) 2022-02-08 02:53:36,112 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 02:53:36,142 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 02:53:36,171 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['3659 memcached\n'], []) 2022-02-08 02:53:36,171 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:53:36,177 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:53:36,931 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:53:37,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:53:37,335 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:53:37,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:53:37,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:53:37,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:53:37,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:53:37,446 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:53:37,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:53:37,484 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:53:37,499 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:53:37,499 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:53:37,563 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['83208 memcached\n'], []) 2022-02-08 02:53:37,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 02:53:37,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:53:37,625 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['83208 memcached\n'], []) 2022-02-08 02:53:37,625 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:53:37,628 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:53:37,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:53:37,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:53:38,013 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:53:38,015 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:53:38,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:53:38,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:53:38,058 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:53:38,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:53:38,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:53:38,102 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:53:38,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:53:38,132 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 02:53:38,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:53:38,183 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:53:40,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:53:40,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:53:40,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:53:40,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:53:40,796 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 02:53:40,798 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:53:40,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:53:40,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:53:40,864 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 02:53:40,865 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:53:40,867 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:53:40,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:53:41,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:53:41,267 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:53:41,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:53:41,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:53:41,295 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:53:41,308 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:53:41,323 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:53:41,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:53:41,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:53:41,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:53:41,378 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 02:53:41,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:53:41,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:53:43,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:53:44,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:53:44,364 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:53:44,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:53:44,523 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 02:53:44,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:53:44,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:53:44,586 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:53:44,601 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 02:53:44,601 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:53:44,604 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:53:44,683 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:53:44,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:53:44,940 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:53:44,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:53:44,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:53:44,964 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:53:44,977 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:53:44,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:53:45,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:53:45,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:53:45,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:53:45,052 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 02:53:45,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:53:45,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:53:47,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:53:47,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:53:48,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:53:48,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:53:48,282 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 02:53:48,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:53:48,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:53:48,354 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:53:48,367 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 02:53:48,367 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:53:48,369 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 02:53:48,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:53:48,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 02:53:48,736 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:53:48,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 02:53:48,752 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 02:53:48,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 02:53:48,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 02:53:48,793 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 02:53:48,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 02:53:48,822 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 02:53:48,836 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:53:48,852 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 02:53:48,877 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:53:48,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:53:48,913 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:53:48,928 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:53:48,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:53:48,956 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:53:48,970 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 02:53:48,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:53:48,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:53:49,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:53:49,015 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:53:49,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:53:49,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:53:49,073 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:53:49,094 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:53:49,112 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:53:49,132 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:53:49,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:53:49,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 5 queries submitted, 0 failed, 4130 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:53:49,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:53:49,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:49,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:53:49,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:49,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:49,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:49,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:49,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:49,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:49,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:49,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:49,367 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:49,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:53:49,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:49,418 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:49,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:49,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:49,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:49,487 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:49,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:49,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:49,539 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 02:53:49,542 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:53:49,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:53:49,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:53:49,957 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:53:49,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:53:49,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:53:50,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:53:50,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:53:50,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:53:50,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:53:50,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:53:50,219 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:53:50,311 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 02:53:50,349 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:53:50,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:53:51,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:53:51,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:53:52,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:53:52,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:53:52,440 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 02:53:52,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:53:52,512 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:53:52,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:53:52,631 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 02:53:52,632 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:53:52,634 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 02:53:52,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:53:52,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 02:53:53,029 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:53:53,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 02:53:53,045 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 02:53:53,061 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 02:53:53,086 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 02:53:53,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 02:53:53,132 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 02:53:53,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 02:53:53,165 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:53:53,181 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 02:53:53,209 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:53:53,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:53:53,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:53:53,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:53:53,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:53:53,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:53:53,299 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 02:53:53,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:53:53,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:53:53,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:53:53,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:53:53,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:53:53,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:53:53,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:53:53,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:53:53,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:53:53,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:53:53,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:53:53,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:53:53,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:53:53,551 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:53,573 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:53:53,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:53,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:53,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:53,640 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:53,655 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:53,673 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:53,688 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:53,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:53:53,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:53,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:53:53,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:53,770 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:53,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:53,803 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:53,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:53,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:53,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:53,877 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:53:53,894 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 02:53:53,895 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:53:53,982 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:53:54,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:53:54,252 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:53:54,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:53:54,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:53:54,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:53:54,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:53:54,306 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:53:54,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:53:54,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:53:54,349 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:53:54,365 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 02:53:54,392 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:53:54,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:53:56,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:53:56,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:53:57,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:53:57,168 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:53:57,183 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 02:53:57,183 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:53:57,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:53:57,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:53:57,246 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 02:53:57,246 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:53:57,249 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:53:57,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:53:57,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:53:57,627 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:53:57,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:53:57,642 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:53:57,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:53:57,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:53:57,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:53:57,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:53:57,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:53:57,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:53:57,753 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 02:53:57,778 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:53:57,793 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:54:00,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:54:00,275 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:54:00,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:54:00,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:54:00,601 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 02:54:00,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:54:00,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:54:00,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:54:00,684 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 02:54:00,686 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:54:00,687 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 02:54:00,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:54:01,015 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 02:54:01,075 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:54:01,076 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 02:54:01,091 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 02:54:01,104 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 02:54:01,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 02:54:01,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 02:54:01,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 02:54:01,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 02:54:01,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:54:01,193 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 02:54:01,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:54:01,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:54:01,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:54:01,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:54:01,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:54:01,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:54:01,315 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 02:54:01,316 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:54:01,329 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:54:01,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:54:01,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:01,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:54:01,395 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:01,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:01,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:01,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:01,471 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:01,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:01,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:01,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:01,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:01,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:54:01,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:01,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:01,624 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:01,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:01,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:01,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:01,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:01,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:01,716 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:01,734 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:54:01,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:01,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:01,790 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:01,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:01,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:01,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:01,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:01,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:01,892 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 02:54:01,894 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 02:54:01,977 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:54:02,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 02:54:02,273 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:54:02,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 02:54:02,288 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 02:54:02,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 02:54:02,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 02:54:02,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 02:54:02,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 02:54:02,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 02:54:02,372 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:54:02,388 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 02:54:02,418 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:54:02,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:54:02,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:54:02,467 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:54:02,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:54:02,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:54:02,509 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 02:54:02,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:54:02,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:54:02,538 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:54:02,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:02,571 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:54:02,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:02,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:02,637 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:02,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:02,678 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:02,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:02,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:02,739 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:02,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:02,772 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:54:02,788 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:02,803 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:02,822 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:02,836 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:02,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:02,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:02,885 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:02,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:02,920 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:02,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:54:02,956 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:02,971 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:02,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:03,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:03,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:03,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:03,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:03,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:03,092 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 02:54:03,094 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:54:03,193 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:54:03,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:54:03,530 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:54:03,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:54:03,548 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:54:03,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:54:03,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:54:03,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:54:03,609 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:54:03,624 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:54:03,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:54:03,660 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 02:54:03,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:54:03,701 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:54:04,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:54:05,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:54:05,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:54:05,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:54:05,545 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 02:54:05,545 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:54:05,571 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:54:05,609 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:54:05,644 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 02:54:05,644 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:54:05,645 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:54:06,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:54:06,640 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:54:06,736 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:54:06,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:54:06,752 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:54:06,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:54:06,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:54:06,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:54:06,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:54:06,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:54:06,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:54:06,881 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 02:54:06,905 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:54:06,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:54:08,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:54:08,642 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:54:08,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:54:08,880 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:54:08,895 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 02:54:08,895 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:54:08,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:54:08,951 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:54:08,967 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 02:54:08,967 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:54:08,970 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:54:09,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:54:09,306 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:54:09,375 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:54:09,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:54:09,390 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:54:09,405 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:54:09,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:54:09,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:54:09,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:54:09,469 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:54:09,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:54:09,499 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 02:54:09,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:54:09,538 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:54:09,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:54:10,237 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:54:10,433 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:54:10,607 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:54:10,624 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 02:54:10,624 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:54:10,640 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:54:10,676 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:54:10,690 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 02:54:10,691 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:54:10,693 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 02:54:10,772 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:54:11,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 02:54:11,065 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:54:11,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 02:54:11,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 02:54:11,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 02:54:11,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 02:54:11,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 02:54:11,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 02:54:11,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 02:54:11,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:54:11,180 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 02:54:11,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:54:11,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:54:11,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:54:11,262 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:54:11,278 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:54:11,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:54:11,309 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 02:54:11,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:54:11,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:54:11,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:54:11,359 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:11,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:54:11,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:11,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:11,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:11,471 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:11,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:11,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:11,533 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:11,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:54:11,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:11,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:54:11,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:11,627 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:11,642 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:11,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:11,673 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:11,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:11,704 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:11,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:54:11,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:11,757 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:54:11,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:11,793 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:11,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:11,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:11,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:11,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:11,890 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:11,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:54:11,923 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 02:54:11,924 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:54:11,926 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:54:12,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:54:12,244 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:54:12,331 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:54:12,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:54:12,346 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:54:12,359 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:54:12,374 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:54:12,390 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:54:12,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:54:12,420 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:54:12,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:13,005 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:54:13,006 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.08100008965. 2022-02-08 02:54:13,006 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:54:13,007 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:54:13,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:54:13,653 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:54:13,892 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:54:13,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:54:13,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:54:13,928 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:54:13,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:54:13,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:54:14,007 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:54:14,061 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:54:14,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:14,278 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:14,279 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:16,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:16,487 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:16,487 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:18,487 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:18,737 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:18,737 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:20,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:20,946 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:20,946 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:22,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:23,190 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:23,190 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:25,191 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:25,415 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:25,415 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:27,415 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:27,673 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:27,673 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:29,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:30,069 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:30,069 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:32,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:32,684 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:32,684 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:34,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:34,884 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:34,885 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:36,885 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:37,151 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:37,153 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:39,153 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:39,334 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:39,335 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:41,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:41,697 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:41,697 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:43,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:43,936 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:43,937 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:45,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:46,184 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:46,184 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:48,186 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:48,365 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:48,365 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 4156 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:54:50,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:50,619 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:50,621 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:52,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:52,838 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:52,838 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:54,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:55,058 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:54:55,058 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:54:57,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:54:57,394 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:54:57,394 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 44.388999939. 2022-02-08 02:54:57,394 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:54:57,398 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:54:58,543 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:54:59,035 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:54:59,223 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:54:59,224 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:54:59,252 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:54:59,288 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:54:59,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:54:59,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:54:59,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:54:59,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:54:59,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:55:00,177 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:55:00,177 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.78200006485. 2022-02-08 02:55:00,177 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:55:00,180 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:55:00,698 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:55:01,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:55:01,157 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:55:01,158 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:55:01,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:55:01,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:55:01,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:55:01,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:55:01,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:55:01,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:55:01,334 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:55:02,098 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:55:02,098 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.92000007629. 2022-02-08 02:55:02,098 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:55:02,099 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:55:03,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:55:03,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:55:03,996 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:55:03,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:55:04,012 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:55:04,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:55:04,058 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:55:04,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:55:04,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:55:04,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:55:04,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:55:05,023 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:55:05,023 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.92599987984. 2022-02-08 02:55:05,023 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:55:05,026 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:55:05,915 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:55:06,219 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:55:06,301 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:55:06,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:55:06,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:55:06,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:55:06,362 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:55:06,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:55:06,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:55:06,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:55:06,448 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:55:07,160 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:55:07,161 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.1360001564. 2022-02-08 02:55:07,161 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:55:07,163 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:55:07,857 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:55:08,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:55:08,296 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:55:08,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:55:08,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:55:08,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:55:08,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:55:08,403 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:55:08,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:55:08,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:55:08,482 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:55:09,086 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:55:09,088 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.92700004578. 2022-02-08 02:55:09,088 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:55:09,089 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:55:09,790 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:55:10,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:55:10,135 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:55:10,135 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:55:10,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:55:10,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:55:10,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:55:10,201 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:55:10,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:55:10,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:55:10,257 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:55:10,769 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:55:10,769 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.6819999218. 2022-02-08 02:55:10,769 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:55:10,773 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:55:10,979 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:55:11,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:55:12,092 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:55:12,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:55:12,242 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:55:12,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:55:12,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:55:12,642 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:55:12,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:55:12,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:55:13,073 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:55:14,061 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:55:14,061 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 02:55:16,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:55:16,239 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:55:16,240 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.47000002861. ############################################################ # # RAM FootPrint: 4161.97265625 # ############################################################ 2022-02-08 02:55:16,273 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 02:55:16,273 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 75 seconds. Reason: Iteration:6 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 4181 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:56:31,279 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 02:56:31,279 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 02:56:36,282 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:56:37,104 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:56:37,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:56:37,959 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:56:37,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:56:38,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:56:38,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:56:38,127 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:56:38,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:56:38,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:56:38,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:56:38,318 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:56:38,318 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 02:56:38,403 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['107683 memcached\n'], []) 2022-02-08 02:56:38,403 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 02:56:38,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 02:56:38,582 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['107683 memcached\n'], []) 2022-02-08 02:56:38,582 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:56:38,585 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:56:39,443 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:56:39,844 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:56:40,038 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:56:40,038 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:56:40,058 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:56:40,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:56:40,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:56:40,119 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:56:40,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:56:40,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:56:40,211 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:56:40,211 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 02:56:40,270 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['15612 memcached\n'], []) 2022-02-08 02:56:40,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 02:56:40,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 02:56:40,335 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['15612 memcached\n'], []) 2022-02-08 02:56:40,335 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:56:40,338 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:56:41,221 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:56:41,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:56:41,786 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:56:41,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:56:41,806 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:56:41,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:56:41,854 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:56:41,877 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:56:41,905 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:56:41,948 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:56:41,970 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:56:41,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 02:56:42,032 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['4978 memcached\n'], []) 2022-02-08 02:56:42,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 02:56:42,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 02:56:42,101 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['4978 memcached\n'], []) 2022-02-08 02:56:42,101 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:56:42,104 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:56:42,954 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:56:43,443 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:56:43,546 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:56:43,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:56:43,563 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:56:43,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:56:43,609 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:56:43,653 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:56:43,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:56:43,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:56:43,723 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:56:43,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 02:56:43,799 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['41761 memcached\n'], []) 2022-02-08 02:56:43,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 02:56:43,826 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 02:56:43,852 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['41761 memcached\n'], []) 2022-02-08 02:56:43,854 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:56:43,855 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:56:44,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:56:45,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:56:45,493 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:56:45,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:56:45,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:56:45,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:56:45,551 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:56:45,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:56:45,591 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:56:45,642 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:56:45,677 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:56:45,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 02:56:45,742 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['35564 memcached\n'], []) 2022-02-08 02:56:45,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 02:56:45,785 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 02:56:45,806 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['35564 memcached\n'], []) 2022-02-08 02:56:45,808 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:56:45,811 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:56:46,683 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:56:47,145 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:56:47,257 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:56:47,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:56:47,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:56:47,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:56:47,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:56:47,339 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:56:47,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:56:47,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:56:47,444 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:56:47,446 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 02:56:47,513 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['76784 memcached\n'], []) 2022-02-08 02:56:47,515 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 02:56:47,540 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 02:56:47,563 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['76784 memcached\n'], []) 2022-02-08 02:56:47,563 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:56:47,565 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:56:48,209 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:56:48,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:56:49,073 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:56:49,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:56:49,094 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:56:49,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:56:49,147 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:56:49,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:56:49,176 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 5 queries submitted, 0 failed, 4201 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:56:49,204 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:56:49,224 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:56:49,224 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 02:56:49,296 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['70455 memcached\n'], []) 2022-02-08 02:56:49,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 02:56:49,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 02:56:49,346 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['70455 memcached\n'], []) 2022-02-08 02:56:49,346 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:56:49,348 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:56:50,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:56:50,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:56:50,796 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:56:50,798 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:56:50,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:56:50,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:56:50,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:56:50,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:56:50,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:56:50,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:56:50,938 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:56:50,938 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 02:56:51,003 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['4307 memcached\n'], []) 2022-02-08 02:56:51,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 02:56:51,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 02:56:51,059 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['4307 memcached\n'], []) 2022-02-08 02:56:51,059 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:56:51,062 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:56:51,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:56:51,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:56:51,523 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:56:51,525 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:56:51,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:56:51,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:56:51,568 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:56:51,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:56:51,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:56:51,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:56:51,625 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 02:56:51,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:56:51,687 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['83840 memcached\n'], []) 2022-02-08 02:56:51,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 02:56:51,732 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 02:56:51,753 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['83840 memcached\n'], []) 2022-02-08 02:56:51,753 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:56:51,756 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:56:51,834 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:56:52,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:56:52,118 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:56:52,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:56:52,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:56:52,147 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:56:52,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:56:52,186 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:56:52,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:56:52,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:56:52,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:56:52,237 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 02:56:52,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:56:52,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:56:52,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:56:52,770 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:56:52,908 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:56:53,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:56:53,046 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 02:56:53,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:56:53,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:56:53,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:56:53,128 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 02:56:53,128 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:56:53,131 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:56:53,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:56:53,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:56:53,489 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:56:53,489 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:56:53,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:56:53,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:56:53,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:56:53,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:56:53,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:56:53,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:56:53,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:56:53,598 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 02:56:53,622 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:56:53,637 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:56:56,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:56:56,329 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:56:56,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:56:56,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:56:56,664 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 02:56:56,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:56:56,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:56:56,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:56:56,740 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 02:56:56,740 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:56:56,742 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:56:56,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:56:57,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:56:57,073 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:56:57,073 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:56:57,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:56:57,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:56:57,112 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:56:57,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:56:57,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:56:57,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:56:57,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:56:57,180 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 02:56:57,204 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:56:57,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:56:59,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:57:00,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:00,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:00,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:57:00,408 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 02:57:00,408 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:57:00,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:00,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:57:00,494 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 02:57:00,496 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:57:00,496 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 02:57:00,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:57:00,801 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 02:57:00,861 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:57:00,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 02:57:00,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 02:57:00,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 02:57:00,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 02:57:00,918 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 02:57:00,931 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 02:57:00,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 02:57:00,959 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:57:00,974 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 02:57:01,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:57:01,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:01,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:57:01,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:01,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:01,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:57:01,089 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 02:57:01,091 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:57:01,104 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:01,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:57:01,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:01,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:57:01,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:01,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:01,219 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:01,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:01,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:01,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:01,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:01,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:01,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:01,364 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:57:01,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:01,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:01,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:01,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:01,446 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:01,461 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:01,477 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:01,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:01,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:01,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:57:01,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:01,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:01,581 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:01,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:01,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:01,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:01,663 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:01,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:01,707 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 02:57:01,711 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:57:01,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:57:02,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:57:02,071 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:57:02,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:57:02,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:57:02,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:57:02,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:57:02,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:57:02,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:57:02,155 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:57:02,170 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:57:02,186 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 02:57:02,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:57:02,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:03,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:57:03,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:03,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:03,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:57:03,653 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 02:57:03,653 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:57:03,704 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:03,752 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:57:03,812 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 02:57:03,812 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:57:03,813 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 02:57:03,892 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:57:04,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 02:57:04,183 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:57:04,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 02:57:04,198 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 02:57:04,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 02:57:04,229 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 02:57:04,247 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 02:57:04,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 02:57:04,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 02:57:04,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:57:04,313 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 02:57:04,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:57:04,362 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:04,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:57:04,395 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:04,413 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:04,428 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:57:04,444 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 02:57:04,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:57:04,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:04,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:57:04,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:04,515 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:57:04,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:04,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:04,591 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:04,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:04,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:04,663 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:04,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:04,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:04,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:04,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:57:04,762 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:04,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:04,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:04,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:04,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:04,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:04,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:04,892 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:04,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:04,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:57:04,941 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:04,959 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:04,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:04,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:05,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:05,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:05,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:05,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:05,076 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 02:57:05,078 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:57:05,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:57:05,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:57:05,431 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:57:05,433 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:57:05,446 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:57:05,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:57:05,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:57:05,489 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:57:05,502 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:57:05,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:57:05,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:57:05,546 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 02:57:05,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:57:05,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:07,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:57:08,178 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:08,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:08,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:57:08,546 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 02:57:08,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:57:08,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:08,604 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:57:08,618 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 02:57:08,618 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:57:08,621 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:57:08,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:57:08,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:57:08,990 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:57:08,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:57:09,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:57:09,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:57:09,035 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:57:09,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:57:09,068 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:57:09,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:57:09,102 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:57:09,118 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 02:57:09,142 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:57:09,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:11,369 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:57:11,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:11,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:11,920 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:57:11,938 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 02:57:11,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:57:11,956 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:12,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:57:12,023 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 02:57:12,023 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:57:12,026 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 02:57:12,109 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:57:12,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 02:57:12,394 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:57:12,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 02:57:12,408 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 02:57:12,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 02:57:12,436 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 02:57:12,451 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 02:57:12,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 02:57:12,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 02:57:12,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:57:12,503 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 02:57:12,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:57:12,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:12,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:57:12,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:12,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:12,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:57:12,607 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 02:57:12,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:57:12,619 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:12,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:57:12,647 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:12,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:57:12,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:12,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:12,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:12,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:12,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:12,801 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:12,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:12,849 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:12,869 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:12,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:57:12,900 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:12,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:12,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:12,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:12,956 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:12,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:12,984 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:13,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:13,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:13,035 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:57:13,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:13,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:13,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:13,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:13,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:13,132 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:13,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:13,170 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:13,186 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 02:57:13,187 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 02:57:13,267 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:57:13,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 02:57:13,565 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:57:13,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 02:57:13,581 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 02:57:13,594 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 02:57:13,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 02:57:13,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 02:57:13,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 02:57:13,663 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 02:57:13,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:57:13,698 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 02:57:13,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:57:13,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:13,757 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:57:13,772 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:13,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:13,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:57:13,815 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 02:57:13,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:57:13,841 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:13,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:57:13,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:13,903 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:57:13,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:13,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:13,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:13,994 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:14,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:14,043 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:14,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:14,086 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:14,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:14,122 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:57:14,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:14,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:14,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:14,183 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:14,198 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:14,216 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:14,232 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:14,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:14,272 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:14,290 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:57:14,308 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:14,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:14,344 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:14,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:14,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:14,400 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:14,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:14,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:14,453 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 02:57:14,456 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:57:14,548 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:57:14,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:57:14,848 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:57:14,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:57:14,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:57:14,874 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:57:14,887 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:57:14,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:57:14,920 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:57:14,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:57:14,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:57:14,967 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 02:57:14,992 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:57:15,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:17,410 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:57:17,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:17,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:17,984 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:57:18,002 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 02:57:18,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:57:18,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:18,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:57:18,072 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 02:57:18,072 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:57:18,075 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:57:18,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:57:19,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:57:19,400 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:57:19,400 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:57:19,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:57:19,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:57:19,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:57:19,482 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:57:19,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:57:19,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:57:19,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:57:19,556 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 02:57:19,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:57:19,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:21,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:57:21,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:21,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:21,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:57:21,526 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 02:57:21,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:57:21,540 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:21,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:57:21,588 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 02:57:21,588 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:57:21,591 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:57:21,688 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:57:21,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:57:22,032 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:57:22,033 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:57:22,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:57:22,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:57:22,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:57:22,109 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:57:22,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:57:22,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:57:22,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:57:22,187 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 02:57:22,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:57:22,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:25,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:57:25,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:25,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:25,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:57:25,726 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 02:57:25,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:57:25,744 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:25,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:57:25,802 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 02:57:25,803 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:57:25,806 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 02:57:25,903 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:57:26,155 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 02:57:26,259 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:57:26,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 02:57:26,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 02:57:26,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 02:57:26,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 02:57:26,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 02:57:26,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 02:57:26,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 02:57:26,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 02:57:26,385 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 02:57:26,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 02:57:26,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:26,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 02:57:26,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:26,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 02:57:26,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 02:57:26,509 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 02:57:26,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 02:57:26,525 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 02:57:26,540 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 02:57:26,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:26,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 02:57:26,604 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:26,624 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:26,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:26,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:26,698 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:26,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:26,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:26,770 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 02:57:26,795 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:26,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 02:57:26,834 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:26,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:26,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:26,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:26,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:26,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:26,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:26,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 02:57:26,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:26,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 02:57:27,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:27,028 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:27,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:27,063 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:27,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:27,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:27,119 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:27,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 02:57:27,154 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 02:57:27,154 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:57:27,157 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 02:57:27,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:57:27,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 02:57:27,578 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:57:27,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 02:57:27,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 02:57:27,635 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 02:57:27,658 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 02:57:27,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 02:57:27,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 02:57:27,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 02:57:27,729 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:57:28,326 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 02:57:28,328 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.17300009727. 2022-02-08 02:57:28,328 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:57:28,331 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 02:57:28,477 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:57:29,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 02:57:29,395 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:57:29,395 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 02:57:29,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 02:57:29,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 02:57:29,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 02:57:29,545 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 02:57:29,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 02:57:29,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 02:57:29,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:57:29,885 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:57:29,887 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:57:31,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:57:32,164 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:57:32,164 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:57:34,165 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:57:34,415 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:57:34,417 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:57:36,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:57:36,641 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:57:36,641 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:57:38,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:57:38,901 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:57:38,901 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:57:40,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:57:41,171 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:57:41,171 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:57:43,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:57:43,401 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:57:43,401 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:57:45,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:57:45,586 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:57:45,586 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:57:47,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:57:48,132 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:57:48,132 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 4226 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:57:50,132 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:57:50,367 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:57:50,367 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:57:52,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:57:52,674 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:57:52,674 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:57:54,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:57:54,973 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:57:54,973 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:57:56,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:57:57,240 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:57:57,240 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:57:59,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:57:59,510 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:57:59,510 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:58:01,512 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:58:01,848 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:58:01,848 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:58:03,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:58:04,407 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:58:04,407 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:58:06,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:58:06,598 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:58:06,598 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:58:08,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:58:08,780 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:58:08,780 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 02:58:10,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:58:10,967 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 02:58:10,967 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 42.6399998665. 2022-02-08 02:58:10,967 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:58:10,970 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 02:58:12,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:58:12,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 02:58:12,823 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:58:12,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 02:58:12,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 02:58:12,910 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 02:58:12,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 02:58:13,012 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 02:58:13,040 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 02:58:13,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 02:58:13,127 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:58:13,766 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 02:58:13,766 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.7990000248. 2022-02-08 02:58:13,766 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:58:13,769 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 02:58:14,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:58:15,275 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 02:58:15,486 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:58:15,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 02:58:15,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 02:58:15,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 02:58:15,553 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 02:58:15,579 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 02:58:15,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 02:58:15,645 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 02:58:15,676 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:58:16,348 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 02:58:16,349 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.58299994469. 2022-02-08 02:58:16,349 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:58:16,351 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 02:58:17,410 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:58:17,864 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 02:58:17,980 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:58:17,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 02:58:18,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 02:58:18,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 02:58:18,048 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 02:58:18,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 02:58:18,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 02:58:18,158 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 02:58:18,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:58:18,773 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 02:58:18,773 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.4240000248. 2022-02-08 02:58:18,773 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:58:18,776 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 02:58:19,012 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:58:19,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 02:58:19,953 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:58:19,954 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 02:58:19,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 02:58:20,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 02:58:20,209 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 02:58:20,341 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 02:58:20,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 02:58:20,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 02:58:20,543 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:58:21,736 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:58:21,737 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 02:58:23,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:58:23,910 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 02:58:23,911 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.13700008392. 2022-02-08 02:58:23,911 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:58:23,914 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 02:58:24,635 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:58:25,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 02:58:25,625 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:58:25,627 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 02:58:25,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 02:58:25,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 02:58:25,951 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 02:58:26,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 02:58:26,130 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 02:58:26,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 02:58:26,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:58:27,581 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:58:27,581 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 02:58:29,581 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:58:30,108 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 02:58:30,108 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 02:58:32,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:58:32,301 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 02:58:32,302 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 8.39199995995. 2022-02-08 02:58:32,302 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:58:32,305 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 02:58:32,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:58:33,255 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 02:58:33,368 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:58:33,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 02:58:33,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 02:58:33,431 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 02:58:33,448 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 02:58:33,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 02:58:33,505 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 02:58:33,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 02:58:33,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:58:34,168 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 02:58:34,168 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.86599993706. 2022-02-08 02:58:34,168 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 02:58:34,171 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 02:58:35,295 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 02:58:35,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 02:58:36,012 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 02:58:36,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 02:58:36,035 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 02:58:36,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 02:58:36,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 02:58:36,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 02:58:36,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 02:58:36,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 02:58:36,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 02:58:37,598 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 02:58:37,598 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.42900013924. ############################################################ # # RAM FootPrint: 4164.90234375 # ############################################################ 2022-02-08 02:58:37,632 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 02:58:37,632 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 79 seconds. Reason: Iteration:7 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 4250 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 4273 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 02:59:56,638 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 02:59:56,638 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 03:00:01,641 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:00:03,232 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:03,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:00:04,312 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:04,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:00:04,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:00:04,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:00:04,533 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:00:04,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:00:04,698 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:00:04,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:00:04,828 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:00:04,829 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:00:04,963 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['108330 memcached\n'], []) 2022-02-08 03:00:04,964 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 03:00:05,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:00:05,029 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['108330 memcached\n'], []) 2022-02-08 03:00:05,029 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:00:05,032 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:00:05,969 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:06,431 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:00:06,625 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:06,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:00:06,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:00:06,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:00:06,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:00:06,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:00:06,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:00:06,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:00:06,858 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:00:06,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:00:06,934 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['16529 memcached\n'], []) 2022-02-08 03:00:06,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 03:00:06,971 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:00:07,013 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['16529 memcached\n'], []) 2022-02-08 03:00:07,013 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:00:07,017 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:00:07,624 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:07,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:00:08,288 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:08,288 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:00:08,339 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:00:08,377 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:00:08,400 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:00:08,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:00:08,469 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:00:08,507 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:00:08,529 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:00:08,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:00:08,591 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['5715 memcached\n'], []) 2022-02-08 03:00:08,591 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 03:00:08,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:00:08,660 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['5715 memcached\n'], []) 2022-02-08 03:00:08,661 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:00:08,664 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:00:09,479 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:09,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:00:09,986 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:09,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:00:10,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:00:10,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:00:10,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:00:10,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:00:10,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:00:10,122 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:00:10,141 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:00:10,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:00:10,203 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['42418 memcached\n'], []) 2022-02-08 03:00:10,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 03:00:10,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:00:10,255 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['42418 memcached\n'], []) 2022-02-08 03:00:10,255 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:00:10,256 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:00:11,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:11,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:00:11,641 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:11,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:00:11,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:00:11,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:00:11,760 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:00:11,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:00:11,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:00:11,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:00:11,842 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:00:11,844 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:00:11,904 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['36210 memcached\n'], []) 2022-02-08 03:00:11,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 03:00:11,941 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:00:11,967 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['36210 memcached\n'], []) 2022-02-08 03:00:11,967 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:00:11,970 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:00:12,632 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:13,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:00:13,275 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:13,275 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:00:13,293 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:00:13,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:00:13,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:00:13,374 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:00:13,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:00:13,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:00:13,444 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:00:13,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:00:13,503 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['77443 memcached\n'], []) 2022-02-08 03:00:13,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 03:00:13,528 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:00:13,552 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['77443 memcached\n'], []) 2022-02-08 03:00:13,552 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:00:13,555 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:00:14,438 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:15,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:00:15,289 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:15,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:00:15,316 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:00:15,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:00:15,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:00:15,382 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:00:15,395 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:00:15,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:00:15,437 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:00:15,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:00:15,515 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['71143 memcached\n'], []) 2022-02-08 03:00:15,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 03:00:15,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:00:15,573 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['71143 memcached\n'], []) 2022-02-08 03:00:15,575 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:00:15,576 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:00:16,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:16,785 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:00:16,950 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:16,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:00:16,964 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:00:16,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:00:17,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:00:17,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:00:17,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:00:17,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:00:17,105 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:00:17,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:00:17,183 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['4955 memcached\n'], []) 2022-02-08 03:00:17,183 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 03:00:17,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:00:17,237 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['4955 memcached\n'], []) 2022-02-08 03:00:17,237 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:00:17,240 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:00:17,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:18,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:00:18,144 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:18,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:00:18,160 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:00:18,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:00:18,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:00:18,219 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:00:18,234 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:00:18,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:00:18,269 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:00:18,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:00:18,338 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['84538 memcached\n'], []) 2022-02-08 03:00:18,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 03:00:18,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:00:18,407 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['84538 memcached\n'], []) 2022-02-08 03:00:18,407 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:00:18,408 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:00:18,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:18,747 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:00:18,809 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:18,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:00:18,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:00:18,836 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:00:18,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:00:18,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:00:18,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:00:18,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:00:18,910 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:00:18,926 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 03:00:18,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:00:18,964 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:20,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:00:21,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:21,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:21,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:00:21,351 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 03:00:21,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:00:21,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:21,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:00:21,426 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 03:00:21,427 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:00:21,428 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:00:21,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:21,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:00:21,793 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:21,795 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:00:21,808 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:00:21,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:00:21,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:00:21,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:00:21,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:00:21,872 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:00:21,885 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:00:21,900 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 03:00:21,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:00:21,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:24,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:00:24,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:24,724 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:24,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:00:24,897 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 03:00:24,897 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:00:24,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:24,956 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:00:24,970 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 03:00:24,970 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:00:24,971 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:00:25,043 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:25,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:00:25,322 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:25,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:00:25,334 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:00:25,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:00:25,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:00:25,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:00:25,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:00:25,395 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:00:25,408 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:00:25,423 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 03:00:25,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:00:25,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:28,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:00:28,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:28,538 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:28,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:00:28,723 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 03:00:28,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:00:28,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:28,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:00:28,812 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 03:00:28,812 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:00:28,813 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 03:00:28,890 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:29,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 03:00:29,180 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:29,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 03:00:29,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 03:00:29,209 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 03:00:29,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 03:00:29,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 03:00:29,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 03:00:29,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 03:00:29,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:00:29,299 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 03:00:29,326 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:00:29,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:29,359 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:00:29,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:29,390 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:29,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:00:29,418 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 03:00:29,418 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:00:29,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:29,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:00:29,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:29,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:00:29,512 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:29,535 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:29,561 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:29,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:29,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:29,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:29,660 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:29,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:29,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:29,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:00:29,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:29,760 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:29,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:29,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:29,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:29,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:29,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:29,859 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:29,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:29,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:00:29,913 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:29,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:29,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:29,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:29,982 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:30,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:30,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:30,038 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:30,055 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 03:00:30,058 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:00:30,219 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:30,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:00:30,581 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:30,581 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:00:30,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:00:30,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:00:30,632 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:00:30,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:00:30,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:00:30,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:00:30,739 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:00:30,773 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 03:00:30,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:00:30,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:32,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:00:32,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:32,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:32,489 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:00:32,516 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 03:00:32,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:00:32,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:32,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:00:32,644 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 03:00:32,645 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:00:32,648 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 03:00:32,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:32,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 03:00:33,016 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:33,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 03:00:33,040 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 03:00:33,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 03:00:33,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 03:00:33,086 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 03:00:33,099 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 03:00:33,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 03:00:33,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:00:33,148 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 03:00:33,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:00:33,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:33,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:00:33,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:33,236 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:33,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:00:33,263 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 03:00:33,265 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:00:33,278 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:33,293 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:00:33,308 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:33,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:00:33,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:33,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:33,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:33,426 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:33,451 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:33,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:33,502 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:33,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:33,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:33,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:00:33,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:33,594 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:33,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:33,624 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:33,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:33,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:33,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:33,686 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:33,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:33,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:00:33,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:33,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:33,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:33,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:33,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:33,826 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:33,844 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:33,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:33,878 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 03:00:33,881 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:00:33,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:34,186 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:00:34,253 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:34,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:00:34,267 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:00:34,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:00:34,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:00:34,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:00:34,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:00:34,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:00:34,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:00:34,368 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 03:00:34,405 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:00:34,420 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:37,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:00:37,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:37,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:37,755 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:00:37,770 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 03:00:37,772 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:00:37,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:37,822 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:00:37,836 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 03:00:37,838 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:00:37,839 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:00:37,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:38,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:00:38,210 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:38,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:00:38,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:00:38,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:00:38,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:00:38,280 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:00:38,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:00:38,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:00:38,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:00:38,354 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 03:00:38,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:00:38,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:40,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:00:40,982 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:41,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:41,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:00:41,308 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 03:00:41,308 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:00:41,323 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:41,372 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:00:41,388 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 03:00:41,388 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:00:41,391 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 03:00:41,482 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:41,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 03:00:41,763 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:41,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 03:00:41,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 03:00:41,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 03:00:41,801 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 03:00:41,816 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 03:00:41,829 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 03:00:41,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 03:00:41,857 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:00:41,872 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 03:00:41,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:00:41,913 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:41,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:00:41,941 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:41,954 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:41,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:00:41,980 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 03:00:41,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:00:41,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:42,007 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:00:42,020 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:42,043 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:00:42,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:42,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:42,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:42,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:42,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:42,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:42,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:42,242 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:42,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:42,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:00:42,295 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:42,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:42,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:42,339 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:42,354 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:42,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:42,382 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:42,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:42,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:42,428 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:00:42,446 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:42,461 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:42,477 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:42,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:42,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:42,528 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:42,545 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:42,563 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:42,579 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 03:00:42,584 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 03:00:42,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:42,910 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 03:00:42,994 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:42,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 03:00:43,010 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 03:00:43,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 03:00:43,038 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 03:00:43,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 03:00:43,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 03:00:43,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 03:00:43,096 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:00:43,111 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 03:00:43,135 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:00:43,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:43,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:00:43,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:43,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:43,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:00:43,220 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 03:00:43,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:00:43,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:43,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:00:43,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:43,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:00:43,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:43,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:43,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:43,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:43,413 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:43,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:43,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:43,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:43,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:43,528 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:00:43,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:43,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:43,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:43,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:43,614 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:43,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:43,645 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:43,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:43,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:43,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:00:43,714 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:43,732 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:43,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:43,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:43,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:43,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:43,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:43,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:43,851 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 03:00:43,852 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:00:43,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:44,206 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:00:44,278 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:44,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:00:44,295 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:00:44,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:00:44,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:00:44,346 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:00:44,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:00:44,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:00:44,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:00:44,440 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 03:00:44,471 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:00:44,489 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:47,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:00:47,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:47,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:47,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:00:47,923 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 03:00:47,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:00:47,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:48,005 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:00:48,032 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 03:00:48,033 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:00:48,036 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:00:48,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 4294 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:00:49,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:00:49,375 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:49,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:00:49,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:00:49,420 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:00:49,479 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:00:49,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:00:49,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:00:49,540 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:00:49,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:00:49,572 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 03:00:49,596 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:00:49,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:51,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:00:51,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:51,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:51,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:00:51,648 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 03:00:51,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:00:51,663 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:51,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:00:51,713 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 03:00:51,714 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:00:51,717 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:00:51,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:52,028 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:00:52,095 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:52,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:00:52,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:00:52,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:00:52,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:00:52,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:00:52,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:00:52,201 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:00:52,219 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:00:52,236 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 03:00:52,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:00:52,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:52,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:00:53,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:53,239 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:53,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:00:53,446 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 03:00:53,446 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:00:53,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:53,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:00:53,516 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 03:00:53,516 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:00:53,519 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 03:00:53,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:53,841 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 03:00:53,908 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:53,908 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 03:00:53,926 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 03:00:53,941 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 03:00:53,956 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 03:00:53,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 03:00:53,987 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 03:00:54,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 03:00:54,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:00:54,042 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 03:00:54,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:00:54,081 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:54,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:00:54,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:54,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:00:54,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:00:54,151 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 03:00:54,153 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:00:54,165 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:00:54,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:00:54,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:54,219 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:00:54,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:54,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:54,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:54,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:54,346 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:54,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:54,397 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:54,423 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:00:54,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:54,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:00:54,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:54,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:54,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:54,528 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:54,545 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:54,561 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:54,576 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:54,594 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:00:54,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:54,627 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:00:54,645 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:54,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:54,681 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:54,698 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:54,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:54,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:54,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:54,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:00:54,786 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 03:00:54,786 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:00:54,789 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:00:54,887 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:55,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:00:55,219 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:55,219 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:00:55,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:00:55,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:00:55,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:00:55,278 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:00:55,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:00:55,308 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:00:55,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:00:55,832 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:00:55,832 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.04500007629. 2022-02-08 03:00:55,832 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:00:55,835 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:00:55,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:00:56,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:00:56,657 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:00:56,658 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:00:56,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:00:56,724 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:00:56,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:00:56,772 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:00:56,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:00:56,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:00:56,834 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:00:57,720 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:00:57,720 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:00:59,721 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:00:59,904 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:00:59,904 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:01:01,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:02,086 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:02,088 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:01:04,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:04,249 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:04,249 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:01:06,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:06,484 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:06,486 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:01:08,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:08,707 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:08,707 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:01:10,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:10,921 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:10,921 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:01:12,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:13,431 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:13,431 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:01:15,433 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:15,782 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:15,783 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:01:17,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:18,180 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:18,181 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:01:20,181 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:20,407 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:20,407 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:01:22,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:22,736 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:22,737 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:01:24,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:24,980 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:24,980 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:01:26,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:27,173 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:27,173 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:01:29,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:29,417 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:29,417 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:01:31,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:31,671 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:31,671 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:01:33,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:33,875 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:33,875 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:01:35,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:36,052 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:01:36,055 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 40.2220001221. 2022-02-08 03:01:36,055 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:01:36,058 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:01:36,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:01:37,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:01:37,378 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:01:37,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:01:37,403 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:01:37,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:01:37,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:01:37,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:01:37,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:01:37,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:01:37,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:38,203 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:01:38,203 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.14800000191. 2022-02-08 03:01:38,203 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:01:38,206 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:01:38,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:01:38,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:01:38,806 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:01:38,806 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:01:38,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:01:38,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:01:38,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:01:38,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:01:38,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:01:38,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:01:38,954 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:39,635 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:01:39,635 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.4319999218. 2022-02-08 03:01:39,635 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:01:39,638 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:01:40,400 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:01:41,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:01:41,286 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:01:41,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:01:41,306 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:01:41,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:01:41,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:01:41,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:01:41,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:01:41,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:01:41,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:42,227 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:01:42,227 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.59200000763. 2022-02-08 03:01:42,227 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:01:42,230 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:01:42,857 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:01:43,188 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:01:43,476 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:01:43,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:01:43,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:01:43,622 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:01:43,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:01:43,778 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:01:43,831 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:01:43,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:01:43,885 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:45,112 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:45,114 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 03:01:47,114 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:47,614 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:47,614 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 5 queries submitted, 0 failed, 4319 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:01:49,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:49,855 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:01:49,857 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.62900018692. 2022-02-08 03:01:49,857 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:01:49,858 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:01:50,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:01:50,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:01:50,779 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:01:50,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:01:50,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:01:50,821 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:01:50,839 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:01:50,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:01:50,882 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:01:50,918 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:01:50,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:51,664 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:01:51,665 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.80899977684. 2022-02-08 03:01:51,665 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:01:51,667 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:01:52,234 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:01:52,479 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:01:52,602 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:01:52,602 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:01:52,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:01:52,637 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:01:52,650 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:01:52,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:01:52,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:01:52,709 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:01:52,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:53,201 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:01:53,203 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.53700017929. 2022-02-08 03:01:53,203 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:01:53,204 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:01:53,436 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:01:54,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:01:54,430 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:01:54,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:01:54,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:01:54,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:01:54,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:01:54,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:01:55,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:01:55,188 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:01:55,224 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:56,091 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:56,092 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:01:58,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:01:58,773 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:01:58,775 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:02:00,775 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:02:01,121 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:02:01,121 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:02:03,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:02:03,693 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:02:03,693 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:02:05,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:02:06,523 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:02:06,523 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:02:08,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:02:08,755 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:02:08,755 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:02:10,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:02:11,328 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:02:11,329 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 18.126999855. ############################################################ # # RAM FootPrint: 4167.0703125 # ############################################################ 2022-02-08 03:02:11,410 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 03:02:11,410 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 113 seconds. Reason: Iteration:8 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 4344 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 4367 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:04:04,414 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 03:04:04,415 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 03:04:09,417 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:04:10,201 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:11,022 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:04:11,369 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:11,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:04:11,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:04:11,489 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:04:11,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:04:11,566 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:04:11,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:04:11,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:04:11,687 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:04:11,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:04:11,775 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['109029 memcached\n'], []) 2022-02-08 03:04:11,775 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 03:04:11,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:04:11,858 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['109029 memcached\n'], []) 2022-02-08 03:04:11,858 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:04:11,861 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:04:12,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:13,551 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:04:13,740 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:13,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:04:13,762 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:04:13,790 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:04:13,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:04:13,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:04:13,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:04:13,936 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:04:13,957 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:04:13,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:04:14,049 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['17401 memcached\n'], []) 2022-02-08 03:04:14,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 03:04:14,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:04:14,137 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['17401 memcached\n'], []) 2022-02-08 03:04:14,138 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:04:14,140 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:04:15,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:16,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:04:16,278 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:16,278 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:04:16,306 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:04:16,334 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:04:16,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:04:16,395 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:04:16,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:04:16,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:04:16,487 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:04:16,489 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:04:16,561 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['6512 memcached\n'], []) 2022-02-08 03:04:16,561 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 03:04:16,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:04:16,627 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['6512 memcached\n'], []) 2022-02-08 03:04:16,627 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:04:16,628 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:04:18,385 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:19,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:04:19,325 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:19,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:04:19,346 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:04:19,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:04:19,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:04:19,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:04:19,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:04:19,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:04:19,536 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:04:19,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:04:19,598 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['43116 memcached\n'], []) 2022-02-08 03:04:19,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 03:04:19,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:04:19,661 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['43116 memcached\n'], []) 2022-02-08 03:04:19,661 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:04:19,663 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:04:20,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:21,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:04:21,558 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:21,558 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:04:21,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:04:21,622 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:04:21,678 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:04:21,698 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:04:21,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:04:21,752 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:04:21,786 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:04:21,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:04:21,861 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['36906 memcached\n'], []) 2022-02-08 03:04:21,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 03:04:21,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:04:21,911 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['36906 memcached\n'], []) 2022-02-08 03:04:21,911 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:04:21,913 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:04:23,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:24,507 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:04:24,798 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:24,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:04:24,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:04:24,854 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:04:24,880 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:04:24,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:04:24,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:04:24,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:04:24,960 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:04:24,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:04:25,068 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['78115 memcached\n'], []) 2022-02-08 03:04:25,068 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 03:04:25,099 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:04:25,121 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['78115 memcached\n'], []) 2022-02-08 03:04:25,121 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:04:25,124 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:04:26,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:27,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:04:27,430 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:27,431 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:04:27,446 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:04:27,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:04:27,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:04:27,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:04:27,548 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:04:27,591 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:04:27,607 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:04:27,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:04:27,719 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['71817 memcached\n'], []) 2022-02-08 03:04:27,719 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 03:04:27,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:04:27,796 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['71817 memcached\n'], []) 2022-02-08 03:04:27,798 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:04:27,799 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:04:28,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:29,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:04:29,381 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:29,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:04:29,397 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:04:29,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:04:29,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:04:29,467 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:04:29,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:04:29,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:04:29,605 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:04:29,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:04:29,653 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['5656 memcached\n'], []) 2022-02-08 03:04:29,653 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 03:04:29,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:04:29,704 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['5656 memcached\n'], []) 2022-02-08 03:04:29,706 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:04:29,709 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:04:30,397 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:30,778 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:04:30,924 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:30,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:04:30,938 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:04:30,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:04:31,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:04:31,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:04:31,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:04:31,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:04:31,138 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:04:31,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:04:31,194 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['85210 memcached\n'], []) 2022-02-08 03:04:31,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 03:04:31,227 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:04:31,250 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['85210 memcached\n'], []) 2022-02-08 03:04:31,250 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:04:31,253 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:04:31,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:31,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:04:31,615 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:31,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:04:31,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:04:31,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:04:31,663 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:04:31,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:04:31,696 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:04:31,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:04:31,724 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:04:31,740 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 03:04:31,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:04:31,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:33,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:04:33,836 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:33,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:34,127 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:04:34,141 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 03:04:34,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:04:34,155 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:34,227 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:04:34,243 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 03:04:34,243 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:04:34,244 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:04:34,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:34,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:04:34,608 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:34,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:04:34,622 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:04:34,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:04:34,645 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:04:34,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:04:34,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:04:34,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:04:34,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:04:34,721 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 03:04:34,747 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:04:34,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:37,186 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:04:37,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:37,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:37,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:04:37,826 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 03:04:37,826 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:04:37,839 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:37,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:04:37,898 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 03:04:37,898 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:04:37,901 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:04:38,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:38,224 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:04:38,283 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:38,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:04:38,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:04:38,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:04:38,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:04:38,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:04:38,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:04:38,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:04:38,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:04:38,394 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 03:04:38,418 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:04:38,431 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:41,306 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:04:41,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:41,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:41,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:04:41,953 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 03:04:41,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:04:41,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:42,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:04:42,042 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 03:04:42,043 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:04:42,045 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 03:04:42,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:42,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 03:04:42,434 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:42,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 03:04:42,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 03:04:42,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 03:04:42,477 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 03:04:42,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 03:04:42,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 03:04:42,525 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 03:04:42,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:04:42,555 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 03:04:42,584 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:04:42,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:42,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:04:42,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:42,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:42,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:04:42,668 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 03:04:42,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:04:42,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:42,701 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:04:42,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:42,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:04:42,775 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:42,801 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:42,829 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:42,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:42,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:42,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:42,951 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:42,977 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:43,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:43,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:04:43,033 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:43,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:43,063 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:43,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:43,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:43,114 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:43,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:43,142 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:43,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:43,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:04:43,191 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:43,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:43,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:43,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:43,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:43,272 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:43,288 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:43,303 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:43,319 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 03:04:43,322 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:04:43,566 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:43,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:04:44,252 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:44,252 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:04:44,323 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:04:44,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:04:44,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:04:44,591 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:04:44,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:04:44,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:04:44,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:04:45,048 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 03:04:45,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:04:45,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:46,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:04:47,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:47,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:47,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:04:47,573 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 03:04:47,573 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:04:47,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:47,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:04:48,020 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 03:04:48,020 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:04:48,023 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 03:04:48,102 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:48,341 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 03:04:48,408 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:48,408 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 03:04:48,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 03:04:48,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 03:04:48,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 03:04:48,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 03:04:48,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 03:04:48,497 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 03:04:48,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:04:48,529 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 03:04:48,553 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:04:48,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:48,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:04:48,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:48,627 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:48,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:04:48,655 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 03:04:48,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:04:48,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:48,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:04:48,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:48,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:04:48,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:48,785 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:48,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:48,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:48,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:48,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:48,926 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:48,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:48,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:48,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:04:49,012 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:49,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:49,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:49,063 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:49,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:49,094 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:49,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:49,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:49,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:49,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:04:49,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 5 queries submitted, 0 failed, 4390 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:04:49,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:49,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:49,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:49,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:49,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:49,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:49,318 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:49,334 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 03:04:49,335 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:04:49,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:49,655 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:04:49,716 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:49,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:04:49,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:04:49,742 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:04:49,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:04:49,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:04:49,798 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:04:49,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:04:49,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:04:49,851 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 03:04:49,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:04:49,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:52,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:04:52,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:52,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:53,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:04:53,121 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 03:04:53,122 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:04:53,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:53,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:04:53,209 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 03:04:53,210 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:04:53,211 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:04:53,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:53,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:04:53,638 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:53,640 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:04:53,655 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:04:53,673 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:04:53,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:04:53,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:04:53,729 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:04:53,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:04:53,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:04:53,776 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 03:04:53,801 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:04:53,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:56,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:04:56,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:56,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:56,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:04:56,654 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 03:04:56,655 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:04:56,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:56,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:04:56,744 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 03:04:56,746 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:04:56,747 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 03:04:56,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:57,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 03:04:57,151 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:57,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 03:04:57,168 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 03:04:57,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 03:04:57,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 03:04:57,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 03:04:57,236 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 03:04:57,252 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 03:04:57,267 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:04:57,285 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 03:04:57,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:04:57,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:57,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:04:57,362 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:57,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:57,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:04:57,411 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 03:04:57,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:04:57,428 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:57,446 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:04:57,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:57,484 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:04:57,522 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:57,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:57,586 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:57,614 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:57,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:57,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:57,704 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:57,734 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:57,762 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:57,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:04:57,803 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:57,821 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:57,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:57,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:57,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:57,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:57,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:57,941 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:57,961 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:57,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:04:58,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:58,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:58,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:58,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:58,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:58,114 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:58,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:58,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:58,174 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 03:04:58,177 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 03:04:58,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:58,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 03:04:58,555 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:58,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 03:04:58,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 03:04:58,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 03:04:58,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 03:04:58,622 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 03:04:58,635 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 03:04:58,650 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 03:04:58,663 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:04:58,677 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 03:04:58,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:04:58,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:58,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:04:58,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:58,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:04:58,778 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:04:58,789 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 03:04:58,790 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:04:58,803 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:04:58,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:04:58,834 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:58,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:04:58,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:58,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:58,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:58,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:59,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:59,035 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:59,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:59,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:04:59,119 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:59,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:04:59,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:59,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:59,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:59,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:59,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:59,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:59,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:59,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:04:59,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:59,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:04:59,316 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:59,334 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:59,349 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:59,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:59,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:59,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:59,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:59,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:04:59,450 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 03:04:59,453 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:04:59,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:04:59,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:04:59,871 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:04:59,872 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:04:59,887 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:04:59,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:04:59,915 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:04:59,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:04:59,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:04:59,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:04:59,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:05:00,000 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 03:05:00,028 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:05:00,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:05:02,530 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:05:02,822 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:05:03,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:05:03,165 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:05:03,188 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 03:05:03,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:05:03,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:05:03,239 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:05:03,253 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 03:05:03,253 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:05:03,256 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:05:03,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:05:04,339 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:05:04,505 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:05:04,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:05:04,522 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:05:04,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:05:04,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:05:04,627 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:05:04,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:05:04,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:05:04,678 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:05:04,694 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 03:05:04,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:05:04,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:05:06,568 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:05:06,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:05:06,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:05:07,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:05:07,066 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 03:05:07,066 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:05:07,081 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:05:07,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:05:07,130 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 03:05:07,131 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:05:07,132 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:05:07,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:05:07,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:05:07,525 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:05:07,525 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:05:07,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:05:07,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:05:07,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:05:07,584 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:05:07,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:05:07,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:05:07,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:05:07,642 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 03:05:07,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:05:07,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:05:10,224 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:05:10,561 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:05:10,760 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:05:10,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:05:10,977 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 03:05:10,977 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:05:10,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:05:11,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:05:11,058 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 03:05:11,058 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:05:11,059 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 03:05:11,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:05:11,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 03:05:11,440 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:05:11,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 03:05:11,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 03:05:11,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 03:05:11,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 03:05:11,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 03:05:11,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 03:05:11,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 03:05:11,551 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:05:11,565 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 03:05:11,589 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:05:11,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:05:11,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:05:11,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:05:11,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:05:11,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:05:11,687 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 03:05:11,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:05:11,701 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:05:11,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:05:11,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:05:11,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:05:11,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:05:11,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:05:11,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:05:11,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:05:11,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:05:11,928 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:05:11,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:05:12,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:05:12,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:05:12,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:05:12,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:05:12,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:05:12,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:05:12,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:05:12,160 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:05:12,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:05:12,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:05:12,209 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:05:12,224 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:05:12,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:05:12,260 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:05:12,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:05:12,298 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:05:12,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:05:12,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:05:12,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:05:12,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:05:12,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:05:12,411 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 03:05:12,411 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:05:12,414 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:05:12,512 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:05:12,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:05:12,803 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:05:12,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:05:12,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:05:12,834 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:05:12,849 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:05:12,869 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:05:12,895 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:05:12,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:05:12,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:13,480 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:05:13,480 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.06900000572. 2022-02-08 03:05:13,480 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:05:13,483 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:05:13,619 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:05:13,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:05:14,072 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:05:14,073 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:05:14,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:05:14,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:05:14,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:05:14,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:05:14,178 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:05:14,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:05:14,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:14,911 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:05:14,911 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:05:16,913 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:17,069 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:05:17,069 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:05:19,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:19,250 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:05:19,250 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:05:21,252 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:21,413 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:05:21,414 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:05:23,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:23,667 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:05:23,667 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:05:25,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:25,987 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:05:25,987 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:05:27,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:28,201 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:05:28,201 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:05:30,201 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:30,631 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:05:30,631 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:05:32,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:32,900 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:05:32,901 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:05:34,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:35,184 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:05:35,184 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:05:37,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:37,378 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:05:37,378 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:05:39,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:39,769 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:05:39,769 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:05:41,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:41,983 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:05:41,983 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:05:43,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:44,223 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:05:44,223 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:05:46,224 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:46,404 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:05:46,405 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:05:48,405 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:48,608 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:05:48,608 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 4415 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:05:50,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:50,816 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:05:50,818 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 37.3370001316. 2022-02-08 03:05:50,818 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:05:50,819 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:05:51,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:05:51,826 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:05:51,984 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:05:51,984 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:05:52,007 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:05:52,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:05:52,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:05:52,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:05:52,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:05:52,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:05:52,224 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:52,898 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:05:52,900 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 03:05:54,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:55,089 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:05:55,091 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.27300000191. 2022-02-08 03:05:55,091 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:05:55,092 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:05:55,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:05:56,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:05:56,292 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:05:56,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:05:56,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:05:56,346 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:05:56,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:05:56,403 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:05:56,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:05:56,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:05:56,489 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:57,226 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:05:57,226 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.13599991798. 2022-02-08 03:05:57,226 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:05:57,229 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:05:58,030 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:05:58,339 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:05:58,492 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:05:58,492 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:05:58,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:05:58,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:05:58,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:05:58,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:05:58,599 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:05:58,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:05:58,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:05:59,503 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:05:59,503 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.27600002289. 2022-02-08 03:05:59,503 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:05:59,505 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:06:00,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:06:00,405 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:06:00,526 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:06:00,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:06:00,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:06:00,571 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:06:00,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:06:00,617 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:06:00,635 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:06:00,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:06:00,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:06:01,397 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:06:01,397 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.89400005341. 2022-02-08 03:06:01,397 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:06:01,400 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:06:02,403 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:06:02,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:06:03,171 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:06:03,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:06:03,239 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:06:03,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:06:03,329 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:06:03,410 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:06:03,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:06:03,596 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:06:03,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:06:04,819 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:06:04,819 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 03:06:06,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:06:06,980 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:06:06,980 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.58299994469. 2022-02-08 03:06:06,980 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:06:06,982 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:06:07,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:06:07,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:06:07,957 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:06:07,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:06:07,971 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:06:07,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:06:08,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:06:08,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:06:08,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:06:08,066 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:06:08,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:06:08,579 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:06:08,581 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.59899997711. 2022-02-08 03:06:08,581 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:06:08,582 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:06:09,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:06:10,298 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:06:10,506 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:06:10,507 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:06:10,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:06:10,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:06:10,617 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:06:10,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:06:10,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:06:10,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:06:10,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:06:11,507 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:06:11,509 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.9279999733. ############################################################ # # RAM FootPrint: 4167.08203125 # ############################################################ 2022-02-08 03:06:11,546 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 03:06:11,548 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 93 seconds. Reason: Iteration:9 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 4441 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:07:44,553 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 03:07:44,553 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 5 queries submitted, 0 failed, 4465 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:07:49,556 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:07:50,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:07:51,204 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:07:51,532 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:07:51,533 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:07:51,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:07:51,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:07:51,650 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:07:51,709 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:07:51,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:07:51,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:07:51,944 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:07:51,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:07:52,016 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['109776 memcached\n'], []) 2022-02-08 03:07:52,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 03:07:52,076 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:07:52,099 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['109776 memcached\n'], []) 2022-02-08 03:07:52,101 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:07:52,102 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:07:52,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:07:53,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:07:53,651 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:07:53,653 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:07:53,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:07:53,709 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:07:53,757 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:07:53,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:07:53,801 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:07:53,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:07:53,855 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:07:53,857 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:07:53,926 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['18349 memcached\n'], []) 2022-02-08 03:07:53,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 03:07:53,971 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:07:54,000 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['18349 memcached\n'], []) 2022-02-08 03:07:54,002 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:07:54,003 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:07:54,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:07:55,303 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:07:55,532 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:07:55,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:07:55,571 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:07:55,596 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:07:55,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:07:55,642 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:07:55,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:07:55,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:07:55,729 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:07:55,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:07:55,811 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['7351 memcached\n'], []) 2022-02-08 03:07:55,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 03:07:55,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:07:55,891 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['7351 memcached\n'], []) 2022-02-08 03:07:55,891 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:07:55,894 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:07:56,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:07:57,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:07:57,693 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:07:57,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:07:57,734 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:07:57,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:07:57,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:07:57,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:07:57,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:07:57,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:07:57,874 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:07:57,874 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:07:57,961 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['43864 memcached\n'], []) 2022-02-08 03:07:57,961 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 03:07:57,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:07:58,016 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['43864 memcached\n'], []) 2022-02-08 03:07:58,017 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:07:58,019 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:07:58,910 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:07:59,403 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:07:59,598 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:07:59,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:07:59,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:07:59,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:07:59,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:07:59,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:07:59,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:07:59,752 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:07:59,790 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:07:59,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:07:59,861 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['37656 memcached\n'], []) 2022-02-08 03:07:59,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 03:07:59,892 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:07:59,959 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['37656 memcached\n'], []) 2022-02-08 03:07:59,960 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:07:59,961 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:08:00,951 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:01,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:08:02,046 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:02,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:08:02,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:08:02,076 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:08:02,089 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:08:02,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:08:02,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:08:02,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:08:02,164 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:08:02,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:08:02,227 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['78930 memcached\n'], []) 2022-02-08 03:08:02,227 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 03:08:02,267 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:08:02,289 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['78930 memcached\n'], []) 2022-02-08 03:08:02,290 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:08:02,292 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:08:03,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:03,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:08:04,188 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:04,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:08:04,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:08:04,232 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:08:04,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:08:04,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:08:04,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:08:04,352 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:08:04,368 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:08:04,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:08:04,437 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['72565 memcached\n'], []) 2022-02-08 03:08:04,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 03:08:04,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:08:04,500 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['72565 memcached\n'], []) 2022-02-08 03:08:04,500 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:08:04,505 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:08:05,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:06,114 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:08:06,311 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:06,311 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:08:06,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:08:06,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:08:06,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:08:06,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:08:06,405 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:08:06,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:08:06,450 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:08:06,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:08:06,510 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['6409 memcached\n'], []) 2022-02-08 03:08:06,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 03:08:06,553 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:08:06,578 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['6409 memcached\n'], []) 2022-02-08 03:08:06,578 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:08:06,582 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:08:07,650 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:07,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:08:08,107 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:08,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:08:08,122 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:08:08,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:08:08,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:08:08,168 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:08:08,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:08:08,206 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:08:08,223 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:08:08,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:08:08,289 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['86044 memcached\n'], []) 2022-02-08 03:08:08,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 03:08:08,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:08:08,334 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['86044 memcached\n'], []) 2022-02-08 03:08:08,335 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:08:08,336 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:08:08,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:08,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:08:08,688 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:08,688 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:08:08,704 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:08:08,719 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:08:08,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:08:08,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:08:08,765 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:08:08,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:08:08,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:08:08,806 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 03:08:08,834 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:08:08,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:10,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:08:11,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:11,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:11,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:08:11,328 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 03:08:11,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:08:11,341 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:11,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:08:11,394 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 03:08:11,394 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:08:11,395 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:08:11,467 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:11,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:08:11,740 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:11,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:08:11,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:08:11,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:08:11,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:08:11,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:08:11,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:08:11,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:08:11,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:08:11,855 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 03:08:11,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:08:11,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:14,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:08:14,571 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:14,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:14,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:08:14,937 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 03:08:14,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:08:14,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:15,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:08:15,020 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 03:08:15,022 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:08:15,023 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:08:15,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:15,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:08:15,361 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:15,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:08:15,374 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:08:15,387 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:08:15,400 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:08:15,415 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:08:15,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:08:15,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:08:15,459 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:08:15,476 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 03:08:15,505 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:08:15,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:18,428 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:08:18,729 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:18,928 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:19,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:08:19,114 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 03:08:19,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:08:19,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:19,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:08:19,190 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 03:08:19,190 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:08:19,193 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 03:08:19,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:19,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 03:08:19,599 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:19,599 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 03:08:19,614 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 03:08:19,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 03:08:19,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 03:08:19,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 03:08:19,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 03:08:19,678 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 03:08:19,693 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:08:19,707 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 03:08:19,734 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:08:19,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:19,765 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:08:19,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:19,795 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:19,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:08:19,828 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 03:08:19,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:08:19,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:19,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:08:19,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:19,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:08:19,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:19,964 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:19,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:20,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:20,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:20,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:20,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:20,140 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:20,165 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:20,183 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:08:20,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:20,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:20,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:20,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:20,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:20,278 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:20,293 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:20,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:20,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:20,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:08:20,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:20,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:20,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:20,413 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:20,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:20,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:20,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:20,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:20,507 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 03:08:20,509 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:08:20,732 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:21,341 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:08:21,786 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:21,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:08:21,897 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:08:22,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:08:22,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:08:22,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:08:22,298 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:08:22,459 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:08:22,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:08:22,683 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 03:08:22,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:08:22,841 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:24,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:08:24,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:24,959 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:25,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:08:25,382 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 03:08:25,382 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:08:25,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:25,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:08:25,799 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 03:08:25,799 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:08:25,801 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 03:08:25,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:26,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 03:08:26,186 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:26,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 03:08:26,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 03:08:26,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 03:08:26,229 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 03:08:26,255 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 03:08:26,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 03:08:26,308 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 03:08:26,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:08:26,339 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 03:08:26,364 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:08:26,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:26,395 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:08:26,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:26,426 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:26,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:08:26,457 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 03:08:26,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:08:26,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:26,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:08:26,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:26,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:08:26,563 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:26,591 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:26,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:26,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:26,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:26,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:26,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:26,767 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:26,793 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:26,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:08:26,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:26,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:26,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:26,874 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:26,890 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:26,905 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:26,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:26,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:26,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:26,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:08:26,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:27,007 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:27,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:27,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:27,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:27,073 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:27,091 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:27,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:27,124 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 03:08:27,127 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:08:27,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:27,403 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:08:27,459 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:27,459 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:08:27,471 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:08:27,484 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:08:27,497 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:08:27,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:08:27,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:08:27,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:08:27,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:08:27,568 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 03:08:27,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:08:27,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:30,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:08:30,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:30,507 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:30,686 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:08:30,710 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 03:08:30,711 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:08:30,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:30,785 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:08:30,801 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 03:08:30,801 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:08:30,802 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:08:30,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:31,122 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:08:31,197 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:31,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:08:31,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:08:31,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:08:31,247 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:08:31,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:08:31,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:08:31,306 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:08:31,323 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:08:31,342 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 03:08:31,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:08:31,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:33,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:08:33,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:34,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:34,285 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:08:34,303 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 03:08:34,303 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:08:34,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:34,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:08:34,384 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 03:08:34,385 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:08:34,388 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 03:08:34,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:34,724 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 03:08:34,798 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:34,798 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 03:08:34,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 03:08:34,844 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 03:08:34,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 03:08:34,892 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 03:08:34,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 03:08:34,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 03:08:34,954 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:08:34,973 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 03:08:34,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:08:35,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:35,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:08:35,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:35,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:35,081 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:08:35,095 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 03:08:35,096 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:08:35,114 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:35,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:08:35,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:35,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:08:35,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:35,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:35,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:35,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:35,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:35,374 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:35,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:35,433 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:35,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:35,477 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:08:35,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:35,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:35,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:35,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:35,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:35,596 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:35,614 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:35,632 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:35,650 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:35,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:08:35,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:35,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:35,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:35,739 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:35,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:35,775 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:35,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:35,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:35,825 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 03:08:35,828 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 03:08:35,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:36,155 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 03:08:36,223 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:36,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 03:08:36,239 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 03:08:36,252 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 03:08:36,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 03:08:36,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 03:08:36,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 03:08:36,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 03:08:36,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:08:36,345 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 03:08:36,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:08:36,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:36,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:08:36,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:36,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:36,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:08:36,456 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 03:08:36,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:08:36,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:36,484 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:08:36,497 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:36,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:08:36,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:36,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:36,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:36,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:36,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:36,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:36,734 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:36,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:36,790 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:36,808 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:08:36,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:36,841 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:36,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:36,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:36,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:36,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:36,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:36,938 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:36,954 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:36,971 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:08:36,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:37,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:37,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:37,043 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:37,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:37,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:37,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:37,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:37,127 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 03:08:37,128 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:08:37,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:37,487 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:08:37,552 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:37,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:08:37,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:08:37,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:08:37,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:08:37,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:08:37,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:08:37,653 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:08:37,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:08:37,684 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 03:08:37,714 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:08:37,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:40,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:08:40,448 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:40,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:40,778 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:08:40,795 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 03:08:40,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:08:40,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:40,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:08:40,868 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 03:08:40,869 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:08:40,871 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:08:41,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:42,022 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:08:42,237 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:42,239 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:08:42,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:08:42,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:08:42,306 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:08:42,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:08:42,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:08:42,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:08:42,382 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:08:42,398 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 03:08:42,423 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:08:42,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:44,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:08:44,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:44,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:44,571 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:08:44,586 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 03:08:44,586 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:08:44,604 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:44,640 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:08:44,657 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 03:08:44,657 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:08:44,660 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:08:44,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:44,966 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:08:45,036 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:45,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:08:45,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:08:45,068 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:08:45,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:08:45,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:08:45,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:08:45,132 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:08:45,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:08:45,167 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 03:08:45,191 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:08:45,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:48,191 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:08:48,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:48,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:48,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:08:48,924 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 03:08:48,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:08:48,941 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:48,992 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:08:49,009 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 03:08:49,009 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:08:49,012 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 03:08:49,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 4485 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:08:49,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 03:08:49,404 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:49,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 03:08:49,418 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 03:08:49,431 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 03:08:49,446 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 03:08:49,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 03:08:49,477 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 03:08:49,492 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 03:08:49,507 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:08:49,523 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 03:08:49,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:08:49,561 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:49,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:08:49,589 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:49,604 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:08:49,617 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:08:49,631 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 03:08:49,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:08:49,645 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:08:49,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:08:49,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:49,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:08:49,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:49,760 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:49,795 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:49,822 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:49,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:49,882 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:49,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:49,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:08:49,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:49,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:08:50,010 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:50,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:50,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:50,066 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:50,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:50,102 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:50,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:50,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:08:50,160 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:50,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:08:50,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:50,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:50,234 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:50,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:50,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:50,290 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:50,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:50,329 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:08:50,348 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 03:08:50,351 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:08:50,351 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:08:50,474 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:50,822 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:08:50,937 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:50,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:08:50,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:08:51,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:08:51,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:08:51,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:08:51,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:08:51,114 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:08:51,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:08:51,644 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:08:51,644 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.29500007629. 2022-02-08 03:08:51,644 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:08:51,647 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:08:51,762 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:08:52,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:08:52,085 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:08:52,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:08:52,104 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:08:52,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:08:52,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:08:52,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:08:52,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:08:52,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:08:52,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:08:52,815 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:08:52,815 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:08:54,816 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:08:55,242 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:08:55,242 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:08:57,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:08:57,559 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:08:57,559 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:08:59,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:08:59,775 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:08:59,775 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:09:01,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:02,160 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:02,160 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:09:04,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:04,355 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:04,357 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:09:06,357 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:06,532 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:06,533 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:09:08,533 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:08,723 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:08,723 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:09:10,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:11,061 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:11,061 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:09:13,061 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:13,265 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:13,265 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:09:15,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:15,500 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:15,500 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:09:17,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:17,686 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:17,686 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:09:19,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:19,867 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:19,868 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:09:21,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:22,076 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:22,076 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:09:24,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:24,246 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:09:24,247 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 32.6029999256. 2022-02-08 03:09:24,247 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:09:24,250 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:09:25,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:09:25,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:09:25,946 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:09:25,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:09:26,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:09:26,227 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:09:26,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:09:26,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:09:26,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:09:26,433 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:09:26,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:27,292 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:27,292 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 03:09:29,298 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:29,553 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:29,553 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 03:09:31,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:31,733 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:09:31,733 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.48500013351. 2022-02-08 03:09:31,733 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:09:31,736 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:09:32,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:09:32,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:09:32,565 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:09:32,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:09:32,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:09:32,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:09:32,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:09:32,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:09:32,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:09:32,744 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:09:32,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:33,561 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:09:33,562 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.82700014114. 2022-02-08 03:09:33,562 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:09:33,563 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:09:34,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:09:34,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:09:34,990 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:09:34,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:09:35,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:09:35,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:09:35,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:09:35,094 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:09:35,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:09:35,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:09:35,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:35,891 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:35,891 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 03:09:37,892 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:38,072 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:09:38,072 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.51099991798. 2022-02-08 03:09:38,072 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:09:38,075 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:09:38,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:09:39,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:09:39,733 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:09:39,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:09:39,770 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:09:39,844 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:09:39,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:09:39,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:09:40,081 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:09:40,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:09:40,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:41,499 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:41,499 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 03:09:43,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:43,719 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:09:43,720 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.64700007439. 2022-02-08 03:09:43,720 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:09:43,721 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:09:44,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:09:45,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:09:45,565 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:09:45,566 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:09:45,622 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:09:45,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:09:45,732 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:09:45,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:09:45,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:09:45,887 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:09:46,040 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:46,663 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:09:46,663 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.9430000782. 2022-02-08 03:09:46,663 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:09:46,664 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:09:47,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:09:47,561 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:09:47,661 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:09:47,663 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:09:47,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:09:47,696 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:09:47,709 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:09:47,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:09:47,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:09:47,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:09:47,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:48,331 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:09:48,331 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.66799998283. 2022-02-08 03:09:48,331 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:09:48,332 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:09:48,602 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:09:49,135 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 5 queries submitted, 0 failed, 4503 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:09:49,664 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:09:49,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:09:49,701 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:09:49,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:09:49,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:09:50,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:09:50,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:09:50,374 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:09:50,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:51,637 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:51,637 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:09:53,637 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:54,240 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:54,240 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:09:56,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:56,569 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:56,569 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:09:58,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:09:59,170 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:09:59,170 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:10:01,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:10:01,469 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:10:01,470 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 13.138999939. ############################################################ # # RAM FootPrint: 4167.9140625 # ############################################################ 2022-02-08 03:10:01,503 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 03:10:01,503 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 68 seconds. Reason: Iteration:10 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 4529 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:11:09,509 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 03:11:09,509 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 03:11:14,512 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:11:15,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:15,979 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:11:16,270 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:16,272 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:11:16,298 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:11:16,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:11:16,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:11:16,469 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:11:16,528 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:11:16,579 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:11:16,605 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:11:16,607 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:11:16,678 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['110481 memcached\n'], []) 2022-02-08 03:11:16,678 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 03:11:16,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:11:16,809 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['110481 memcached\n'], []) 2022-02-08 03:11:16,809 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:11:16,812 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:11:17,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:18,119 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:11:18,325 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:18,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:11:18,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:11:18,410 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:11:18,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:11:18,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:11:18,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:11:18,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:11:18,619 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:11:18,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:11:18,687 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['19251 memcached\n'], []) 2022-02-08 03:11:18,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 03:11:18,742 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:11:18,769 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['19251 memcached\n'], []) 2022-02-08 03:11:18,769 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:11:18,772 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:11:19,619 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:20,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:11:20,471 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:20,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:11:20,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:11:20,522 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:11:20,545 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:11:20,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:11:20,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:11:20,650 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:11:20,677 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:11:20,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:11:20,746 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['8169 memcached\n'], []) 2022-02-08 03:11:20,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 03:11:20,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:11:20,812 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['8169 memcached\n'], []) 2022-02-08 03:11:20,812 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:11:20,815 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:11:21,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:22,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:11:22,316 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:22,316 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:11:22,349 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:11:22,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:11:22,390 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:11:22,415 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:11:22,438 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:11:22,466 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:11:22,497 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:11:22,499 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:11:22,562 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['44528 memcached\n'], []) 2022-02-08 03:11:22,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 03:11:22,609 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:11:22,638 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['44528 memcached\n'], []) 2022-02-08 03:11:22,638 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:11:22,641 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:11:23,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:24,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:11:24,500 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:24,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:11:24,543 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:11:24,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:11:24,663 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:11:24,683 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:11:24,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:11:24,793 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:11:24,809 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:11:24,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:11:24,868 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['38361 memcached\n'], []) 2022-02-08 03:11:24,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 03:11:24,903 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:11:24,931 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['38361 memcached\n'], []) 2022-02-08 03:11:24,931 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:11:24,934 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:11:25,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:26,219 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:11:26,401 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:26,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:11:26,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:11:26,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:11:26,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:11:26,467 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:11:26,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:11:26,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:11:26,528 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:11:26,528 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:11:26,592 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['79591 memcached\n'], []) 2022-02-08 03:11:26,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 03:11:26,619 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:11:26,641 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['79591 memcached\n'], []) 2022-02-08 03:11:26,641 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:11:26,644 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:11:27,977 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:28,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:11:28,782 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:28,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:11:28,808 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:11:28,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:11:28,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:11:28,865 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:11:28,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:11:28,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:11:28,923 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:11:28,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:11:28,980 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['73283 memcached\n'], []) 2022-02-08 03:11:28,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 03:11:29,012 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:11:29,039 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['73283 memcached\n'], []) 2022-02-08 03:11:29,039 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:11:29,042 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:11:30,040 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:30,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:11:30,836 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:30,836 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:11:30,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:11:30,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:11:30,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:11:30,910 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:11:30,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:11:30,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:11:30,969 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:11:30,969 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:11:31,017 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['7116 memcached\n'], []) 2022-02-08 03:11:31,017 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 03:11:31,048 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:11:31,073 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['7116 memcached\n'], []) 2022-02-08 03:11:31,073 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:11:31,076 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:11:31,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:32,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:11:32,194 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:32,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:11:32,209 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:11:32,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:11:32,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:11:32,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:11:32,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:11:32,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:11:32,316 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:11:32,316 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:11:32,380 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['86740 memcached\n'], []) 2022-02-08 03:11:32,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 03:11:32,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:11:32,430 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['86740 memcached\n'], []) 2022-02-08 03:11:32,430 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:11:32,431 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:11:32,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:32,734 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:11:32,799 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:32,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:11:32,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:11:32,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:11:32,841 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:11:32,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:11:32,874 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:11:32,892 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:11:32,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:11:32,923 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 03:11:32,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:11:32,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:35,186 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:11:35,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:35,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:35,724 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:11:35,783 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 03:11:35,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:11:35,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:35,841 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:11:35,854 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 03:11:35,855 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:11:35,857 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:11:35,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:36,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:11:36,200 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:36,201 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:11:36,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:11:36,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:11:36,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:11:36,257 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:11:36,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:11:36,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:11:36,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:11:36,313 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 03:11:36,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:11:36,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:38,869 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:11:39,170 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:39,346 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:39,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:11:39,525 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 03:11:39,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:11:39,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:39,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:11:39,602 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 03:11:39,602 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:11:39,605 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:11:39,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:39,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:11:39,940 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:39,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:11:39,951 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:11:39,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:11:39,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:11:39,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:11:40,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:11:40,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:11:40,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:11:40,040 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 03:11:40,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:11:40,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:42,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:11:43,211 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:43,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:43,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:11:43,566 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 03:11:43,568 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:11:43,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:43,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:11:43,651 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 03:11:43,651 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:11:43,654 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 03:11:43,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:43,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 03:11:44,002 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:44,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 03:11:44,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 03:11:44,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 03:11:44,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 03:11:44,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 03:11:44,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 03:11:44,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 03:11:44,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:11:44,117 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 03:11:44,142 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:11:44,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:44,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:11:44,186 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:44,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:44,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:11:44,223 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 03:11:44,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:11:44,236 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:44,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:11:44,267 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:44,293 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:11:44,326 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:44,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:44,385 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:44,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:44,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:44,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:44,499 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:44,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:44,553 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:44,568 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:11:44,584 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:44,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:44,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:44,627 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:44,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:44,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:44,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:44,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:44,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:44,716 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:11:44,732 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:44,747 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:44,765 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:44,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:44,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:44,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:44,829 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:44,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:44,861 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 03:11:44,864 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:11:44,938 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:45,165 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:11:45,229 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:45,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:11:45,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:11:45,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:11:45,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:11:45,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:11:45,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:11:45,316 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:11:45,331 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:11:45,346 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 03:11:45,377 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:11:45,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:46,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:11:46,948 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:47,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:47,153 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:11:47,232 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 03:11:47,232 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:11:47,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:47,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:11:47,398 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 03:11:47,398 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:11:47,401 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 03:11:47,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:47,742 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 03:11:47,812 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:47,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 03:11:47,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 03:11:47,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 03:11:47,857 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 03:11:47,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 03:11:47,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 03:11:47,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 03:11:47,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:11:47,943 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 03:11:47,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:11:47,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:48,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:11:48,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:48,038 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:48,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:11:48,066 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 03:11:48,066 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:11:48,081 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:48,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:11:48,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:48,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:11:48,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:48,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:48,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:48,257 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:48,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:48,341 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:48,374 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:48,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:48,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:48,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:11:48,479 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:48,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:48,512 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:48,528 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:48,548 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:48,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:48,579 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:48,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:48,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:48,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:11:48,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:48,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:48,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:48,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:48,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:48,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:48,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:48,767 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:48,785 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 03:11:48,786 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:11:48,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:49,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:11:49,118 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:49,119 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:11:49,132 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:11:49,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:11:49,155 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:11:49,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:11:49,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:11:49,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 5 queries submitted, 0 failed, 4553 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:11:49,209 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:11:49,223 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 03:11:49,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:11:49,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:51,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:11:52,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:52,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:52,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:11:52,539 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 03:11:52,540 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:11:52,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:52,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:11:52,617 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 03:11:52,618 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:11:52,618 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:11:52,704 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:52,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:11:53,151 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:53,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:11:53,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:11:53,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:11:53,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:11:53,252 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:11:53,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:11:53,329 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:11:53,369 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:11:53,388 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 03:11:53,426 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:11:53,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:55,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:11:56,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:56,382 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:56,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:11:56,601 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 03:11:56,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:11:56,642 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:56,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:11:56,743 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 03:11:56,744 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:11:56,746 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 03:11:56,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:57,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 03:11:57,150 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:57,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 03:11:57,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 03:11:57,183 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 03:11:57,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 03:11:57,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 03:11:57,237 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 03:11:57,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 03:11:57,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:11:57,289 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 03:11:57,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:11:57,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:57,349 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:11:57,367 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:57,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:57,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:11:57,420 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 03:11:57,420 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:11:57,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:57,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:11:57,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:57,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:11:57,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:57,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:57,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:57,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:57,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:57,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:57,734 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:57,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:57,795 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:57,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:11:57,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:57,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:57,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:57,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:57,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:57,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:57,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:57,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:57,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:57,984 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:11:58,005 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:58,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:58,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:58,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:58,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:58,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:58,147 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:58,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:58,183 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 03:11:58,184 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 03:11:58,267 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:58,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 03:11:58,588 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:11:58,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 03:11:58,604 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 03:11:58,617 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 03:11:58,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 03:11:58,647 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 03:11:58,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 03:11:58,676 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 03:11:58,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:11:58,704 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 03:11:58,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:11:58,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:58,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:11:58,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:58,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:11:58,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:11:58,819 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 03:11:58,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:11:58,834 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:11:58,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:11:58,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:58,890 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:11:58,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:58,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:58,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:59,022 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:59,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:59,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:59,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:59,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:11:59,181 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:59,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:11:59,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:59,229 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:59,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:59,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:59,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:59,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:59,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:59,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:11:59,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:59,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:11:59,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:59,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:59,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:59,446 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:59,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:59,484 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:59,502 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:59,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:11:59,538 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 03:11:59,539 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:11:59,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:11:59,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:12:00,046 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:12:00,048 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:12:00,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:12:00,102 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:12:00,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:12:00,170 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:12:00,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:12:00,198 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:12:00,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:12:00,243 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 03:12:00,267 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:12:00,285 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:12:02,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:12:03,278 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:12:03,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:12:03,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:12:03,894 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 03:12:03,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:12:04,005 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:12:04,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:12:04,230 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 03:12:04,230 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:12:04,233 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:12:04,821 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:12:05,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:12:05,326 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:12:05,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:12:05,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:12:05,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:12:05,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:12:05,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:12:05,428 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:12:05,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:12:05,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:12:05,492 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 03:12:05,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:12:05,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:12:07,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:12:07,561 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:12:07,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:12:07,821 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:12:07,838 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 03:12:07,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:12:07,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:12:07,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:12:07,910 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 03:12:07,911 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:12:07,913 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:12:07,994 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:12:08,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:12:08,332 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:12:08,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:12:08,349 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:12:08,364 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:12:08,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:12:08,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:12:08,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:12:08,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:12:08,459 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:12:08,480 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 03:12:08,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:12:08,520 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:12:11,112 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:12:11,443 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:12:11,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:12:11,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:12:11,878 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 03:12:11,880 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:12:11,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:12:11,936 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:12:11,953 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 03:12:11,954 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:12:11,957 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 03:12:12,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:12:12,278 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 03:12:12,348 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:12:12,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 03:12:12,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 03:12:12,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 03:12:12,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 03:12:12,410 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 03:12:12,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 03:12:12,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 03:12:12,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:12:12,474 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 03:12:12,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:12:12,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:12:12,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:12:12,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:12:12,581 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:12:12,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:12:12,609 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 03:12:12,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:12:12,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:12:12,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:12:12,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:12:12,693 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:12:12,732 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:12:12,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:12:12,798 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:12:12,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:12:12,867 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:12:12,900 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:12:12,931 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:12:12,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:12:12,999 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:12:13,015 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:12:13,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:12:13,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:12:13,068 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:12:13,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:12:13,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:12:13,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:12:13,132 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:12:13,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:12:13,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:12:13,181 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:12:13,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:12:13,221 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:12:13,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:12:13,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:12:13,275 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:12:13,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:12:13,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:12:13,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:12:13,355 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 03:12:13,355 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:12:13,358 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:12:13,448 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:12:13,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:12:13,749 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:12:13,749 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:12:13,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:12:13,778 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:12:13,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:12:13,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:12:13,826 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:12:13,841 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:12:13,857 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:14,417 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:12:14,417 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.06200003624. 2022-02-08 03:12:14,417 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:12:14,420 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:12:14,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:12:14,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:12:15,075 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:12:15,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:12:15,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:12:15,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:12:15,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:12:15,334 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:12:15,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:12:15,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:12:15,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:16,247 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:12:16,249 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:12:18,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:18,611 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:12:18,611 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:12:20,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:20,785 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:12:20,785 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:12:22,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:23,043 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:12:23,043 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:12:25,045 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:25,335 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:12:25,335 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:12:27,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:27,598 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:12:27,598 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:12:29,599 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:29,785 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:12:29,786 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:12:31,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:31,963 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:12:31,963 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:12:33,964 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:34,165 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:12:34,167 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:12:36,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:36,391 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:12:36,391 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:12:38,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:38,608 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:12:38,609 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:12:40,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:40,848 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:12:40,849 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:12:42,849 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:43,062 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:12:43,063 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 28.6459999084. 2022-02-08 03:12:43,063 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:12:43,065 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:12:43,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:12:43,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:12:43,684 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:12:43,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:12:43,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:12:43,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:12:43,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:12:43,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:12:43,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:12:43,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:12:43,872 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:44,516 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:12:44,517 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.45300006866. 2022-02-08 03:12:44,517 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:12:44,519 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:12:45,479 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:12:45,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:12:45,944 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:12:45,946 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:12:45,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:12:45,997 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:12:46,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:12:46,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:12:46,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:12:46,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:12:46,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:46,854 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:12:46,855 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 03:12:48,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:49,029 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:12:49,029 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.5110001564. 2022-02-08 03:12:49,029 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:12:49,030 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 5 queries submitted, 0 failed, 4576 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:12:49,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:12:50,293 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:12:50,509 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:12:50,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:12:50,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:12:50,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:12:50,739 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:12:50,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:12:50,897 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:12:50,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:12:50,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:51,996 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:12:51,996 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 03:12:53,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:54,196 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:12:54,196 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 03:12:56,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:56,404 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:12:56,404 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.375. 2022-02-08 03:12:56,404 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:12:56,407 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:12:57,252 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:12:57,604 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:12:57,727 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:12:57,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:12:57,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:12:57,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:12:57,793 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:12:57,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:12:57,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:12:57,885 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:12:57,908 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:12:58,614 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:12:58,614 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.20900011063. 2022-02-08 03:12:58,615 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:12:58,615 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:12:59,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:13:00,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:13:00,648 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:13:00,650 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:13:00,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:13:00,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:13:01,038 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:13:01,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:13:01,267 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:13:01,372 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:13:01,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:13:02,325 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:13:02,325 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.71099996567. 2022-02-08 03:13:02,325 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:13:02,328 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:13:03,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:13:03,339 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:13:03,437 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:13:03,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:13:03,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:13:03,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:13:03,502 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:13:03,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:13:03,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:13:03,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:13:03,573 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:13:04,157 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:13:04,157 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.83200001717. 2022-02-08 03:13:04,157 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:13:04,160 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:13:04,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:13:04,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:13:05,220 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:13:05,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:13:05,329 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:13:05,466 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:13:05,584 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:13:05,729 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:13:05,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:13:06,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:13:06,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:13:06,691 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:13:06,693 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:13:08,693 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:13:08,996 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:13:08,996 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:13:10,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:13:11,436 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:13:11,437 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:13:13,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:13:13,762 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:13:13,763 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:13:15,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:13:16,609 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:13:16,609 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:13:18,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:13:19,046 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:13:19,046 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:13:21,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:13:21,427 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:13:21,428 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:13:23,428 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:13:23,700 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:13:23,701 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:13:25,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:13:25,890 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:13:25,891 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 21.7330000401. ############################################################ # # RAM FootPrint: 4168.81640625 # ############################################################ 2022-02-08 03:13:25,928 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 03:13:25,928 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 92 seconds. Reason: Iteration:11 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 4600 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 4625 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:14:57,934 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 03:14:57,934 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 03:15:02,937 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:15:03,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:04,522 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:15:04,809 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:04,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:15:04,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:15:04,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:15:04,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:15:05,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:15:05,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:15:05,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:15:05,273 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:15:05,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:15:05,338 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['111177 memcached\n'], []) 2022-02-08 03:15:05,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 03:15:05,403 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:15:05,426 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['111177 memcached\n'], []) 2022-02-08 03:15:05,427 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:15:05,430 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:15:06,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:06,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:15:06,990 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:06,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:15:07,012 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:15:07,033 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:15:07,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:15:07,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:15:07,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:15:07,142 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:15:07,168 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:15:07,168 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:15:07,220 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['20083 memcached\n'], []) 2022-02-08 03:15:07,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 03:15:07,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:15:07,282 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['20083 memcached\n'], []) 2022-02-08 03:15:07,282 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:15:07,285 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:15:08,094 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:08,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:15:08,957 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:08,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:15:08,992 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:15:09,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:15:09,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:15:09,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:15:09,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:15:09,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:15:09,219 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:15:09,219 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:15:09,285 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['8960 memcached\n'], []) 2022-02-08 03:15:09,285 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 03:15:09,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:15:09,351 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['8960 memcached\n'], []) 2022-02-08 03:15:09,351 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:15:09,355 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:15:10,201 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:11,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:15:11,196 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:11,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:15:11,211 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:15:11,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:15:11,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:15:11,267 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:15:11,285 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:15:11,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:15:11,346 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:15:11,346 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:15:11,421 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['45219 memcached\n'], []) 2022-02-08 03:15:11,423 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 03:15:11,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:15:11,482 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['45219 memcached\n'], []) 2022-02-08 03:15:11,482 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:15:11,483 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:15:12,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:13,155 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:15:13,361 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:13,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:15:13,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:15:13,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:15:13,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:15:13,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:15:13,512 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:15:13,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:15:13,568 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:15:13,568 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:15:13,630 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['39036 memcached\n'], []) 2022-02-08 03:15:13,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 03:15:13,655 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:15:13,677 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['39036 memcached\n'], []) 2022-02-08 03:15:13,678 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:15:13,680 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:15:14,446 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:15,058 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:15:15,141 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:15,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:15:15,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:15:15,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:15:15,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:15:15,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:15:15,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:15:15,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:15:15,299 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:15:15,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:15:15,375 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['80307 memcached\n'], []) 2022-02-08 03:15:15,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 03:15:15,403 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:15:15,427 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['80307 memcached\n'], []) 2022-02-08 03:15:15,427 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:15:15,430 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:15:16,920 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:17,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:15:17,921 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:17,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:15:17,936 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:15:17,964 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:15:17,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:15:17,994 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:15:18,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:15:18,038 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:15:18,055 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:15:18,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:15:18,124 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['73945 memcached\n'], []) 2022-02-08 03:15:18,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 03:15:18,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:15:18,168 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['73945 memcached\n'], []) 2022-02-08 03:15:18,170 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:15:18,171 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:15:19,028 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:19,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:15:19,796 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:19,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:15:19,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:15:19,865 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:15:19,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:15:19,897 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:15:19,908 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:15:19,931 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:15:19,944 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:15:19,946 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:15:19,997 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['7787 memcached\n'], []) 2022-02-08 03:15:19,997 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 03:15:20,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:15:20,056 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['7787 memcached\n'], []) 2022-02-08 03:15:20,056 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:15:20,059 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:15:20,836 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:21,178 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:15:21,346 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:21,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:15:21,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:15:21,385 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:15:21,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:15:21,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:15:21,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:15:21,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:15:21,484 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:15:21,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:15:21,539 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['87446 memcached\n'], []) 2022-02-08 03:15:21,540 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 03:15:21,566 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:15:21,589 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['87446 memcached\n'], []) 2022-02-08 03:15:21,591 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:15:21,592 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:15:21,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:21,905 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:15:21,976 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:21,977 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:15:21,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:15:22,007 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:15:22,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:15:22,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:15:22,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:15:22,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:15:22,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:15:22,104 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 03:15:22,130 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:15:22,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:24,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:15:24,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:24,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:24,563 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:15:24,576 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 03:15:24,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:15:24,589 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:24,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:15:24,653 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 03:15:24,654 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:15:24,655 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:15:24,755 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:24,964 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:15:25,026 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:25,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:15:25,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:15:25,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:15:25,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:15:25,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:15:25,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:15:25,114 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:15:25,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:15:25,142 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 03:15:25,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:15:25,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:27,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:15:28,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:28,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:28,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:15:28,615 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 03:15:28,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:15:28,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:28,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:15:28,687 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 03:15:28,687 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:15:28,690 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:15:28,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:28,977 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:15:29,036 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:29,038 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:15:29,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:15:29,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:15:29,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:15:29,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:15:29,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:15:29,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:15:29,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:15:29,148 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 03:15:29,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:15:29,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:31,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:15:32,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:32,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:32,581 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:15:32,595 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 03:15:32,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:15:32,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:32,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:15:32,678 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 03:15:32,678 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:15:32,684 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 03:15:32,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:32,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 03:15:33,068 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:33,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 03:15:33,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 03:15:33,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 03:15:33,112 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 03:15:33,130 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 03:15:33,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 03:15:33,158 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 03:15:33,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:15:33,188 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 03:15:33,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:15:33,229 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:33,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:15:33,257 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:33,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:33,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:15:33,301 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 03:15:33,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:15:33,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:33,329 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:15:33,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:33,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:15:33,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:33,443 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:33,482 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:33,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:33,543 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:33,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:33,607 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:33,640 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:33,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:33,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:15:33,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:33,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:33,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:33,747 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:33,765 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:33,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:33,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:33,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:33,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:33,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:15:33,869 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:33,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:33,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:33,931 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:33,948 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:33,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:33,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:34,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:34,020 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 03:15:34,023 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:15:34,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:34,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:15:35,122 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:35,122 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:15:35,239 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:15:35,339 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:15:35,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:15:35,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:15:35,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:15:35,770 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:15:35,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:15:35,989 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 03:15:36,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:15:36,096 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:37,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:15:37,897 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:38,040 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:38,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:15:38,342 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 03:15:38,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:15:38,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:38,624 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:15:38,709 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 03:15:38,710 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:15:38,711 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 03:15:38,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:39,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 03:15:39,092 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:39,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 03:15:39,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 03:15:39,119 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 03:15:39,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 03:15:39,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 03:15:39,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 03:15:39,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 03:15:39,188 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:15:39,204 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 03:15:39,239 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:15:39,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:39,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:15:39,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:39,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:39,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:15:39,328 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 03:15:39,329 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:15:39,346 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:39,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:15:39,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:39,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:15:39,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:39,471 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:39,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:39,540 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:39,581 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:39,617 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:39,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:39,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:39,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:39,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:15:39,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:39,772 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:39,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:39,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:39,822 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:39,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:39,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:39,874 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:39,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:39,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:15:39,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:39,941 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:39,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:39,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:39,994 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:40,012 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:40,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:40,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:40,062 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 03:15:40,065 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:15:40,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:40,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:15:40,404 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:40,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:15:40,418 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:15:40,431 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:15:40,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:15:40,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:15:40,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:15:40,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:15:40,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:15:40,522 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 03:15:40,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:15:40,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:43,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:15:43,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:43,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:43,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:15:43,762 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 03:15:43,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:15:43,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:43,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:15:43,832 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 03:15:43,832 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:15:43,835 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:15:43,908 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:44,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:15:44,197 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:44,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:15:44,216 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:15:44,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:15:44,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:15:44,272 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:15:44,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:15:44,303 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:15:44,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:15:44,339 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 03:15:44,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:15:44,385 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:46,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:15:47,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:47,252 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:47,438 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:15:47,469 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 03:15:47,469 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:15:47,487 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:47,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:15:47,628 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 03:15:47,628 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:15:47,631 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 03:15:47,721 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:47,961 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 03:15:48,022 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:48,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 03:15:48,038 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 03:15:48,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 03:15:48,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 03:15:48,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 03:15:48,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 03:15:48,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 03:15:48,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:15:48,148 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 03:15:48,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:15:48,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:48,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:15:48,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:48,232 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:48,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:15:48,262 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 03:15:48,262 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:15:48,278 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:48,295 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:15:48,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:48,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:15:48,387 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:48,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:48,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:48,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:48,533 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:48,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:48,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:48,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:48,688 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:48,709 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:15:48,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:48,747 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:48,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:48,788 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:48,808 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:48,826 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:48,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:48,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:48,887 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:48,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:15:48,933 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:48,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:48,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:48,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:49,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:49,038 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:49,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:49,076 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:49,095 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 03:15:49,098 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 03:15:49,178 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 4647 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:15:49,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 03:15:49,483 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:49,484 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 03:15:49,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 03:15:49,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 03:15:49,530 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 03:15:49,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 03:15:49,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 03:15:49,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 03:15:49,594 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:15:49,611 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 03:15:49,640 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:15:49,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:49,673 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:15:49,688 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:49,701 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:49,714 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:15:49,727 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 03:15:49,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:15:49,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:49,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:15:49,775 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:49,803 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:15:49,841 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:49,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:49,910 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:49,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:49,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:50,017 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:50,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:50,089 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:15:50,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:50,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:15:50,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:50,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:50,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:50,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:50,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:50,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:50,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:50,265 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:15:50,280 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:50,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:15:50,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:50,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:50,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:50,377 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:50,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:50,413 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:50,433 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:50,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:15:50,467 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 03:15:50,470 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:15:50,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:51,135 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:15:51,223 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:51,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:15:51,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:15:51,257 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:15:51,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:15:51,334 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:15:51,467 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:15:51,538 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:15:51,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:15:51,591 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 03:15:51,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:15:51,655 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:54,109 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:15:54,522 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:54,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:54,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:15:55,015 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 03:15:55,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:15:55,033 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:55,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:15:55,088 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 03:15:55,088 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:15:55,089 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:15:55,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:56,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:15:56,301 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:56,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:15:56,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:15:56,341 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:15:56,372 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:15:56,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:15:56,405 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:15:56,436 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:15:56,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:15:56,473 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 03:15:56,499 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:15:56,517 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:58,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:15:58,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:58,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:15:58,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:15:58,796 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 03:15:58,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:15:58,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:15:58,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:15:58,862 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 03:15:58,862 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:15:58,865 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:15:58,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:15:59,204 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:15:59,276 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:15:59,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:15:59,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:15:59,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:15:59,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:15:59,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:15:59,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:15:59,400 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:15:59,420 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:15:59,437 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 03:15:59,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:15:59,487 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:16:02,298 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:16:02,624 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:16:02,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:16:03,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:16:03,046 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 03:16:03,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:16:03,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:16:03,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:16:03,119 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 03:16:03,121 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:16:03,122 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 03:16:03,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:16:03,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 03:16:03,529 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:16:03,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 03:16:03,545 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 03:16:03,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 03:16:03,573 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 03:16:03,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 03:16:03,607 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 03:16:03,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 03:16:03,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:16:03,660 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 03:16:03,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:16:03,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:16:03,716 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:16:03,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:16:03,747 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:16:03,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:16:03,778 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 03:16:03,778 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:16:03,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:16:03,806 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:16:03,822 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:16:03,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:16:03,890 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:16:03,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:16:03,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:16:03,992 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:16:04,030 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:16:04,063 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:16:04,099 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:16:04,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:16:04,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:16:04,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:16:04,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:16:04,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:16:04,239 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:16:04,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:16:04,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:16:04,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:16:04,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:16:04,326 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:16:04,352 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:16:04,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:16:04,397 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:16:04,415 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:16:04,433 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:16:04,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:16:04,469 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:16:04,487 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:16:04,505 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:16:04,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:16:04,539 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 03:16:04,539 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:16:04,542 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:16:04,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:16:04,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:16:04,930 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:16:04,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:16:04,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:16:04,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:16:04,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:16:04,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:16:05,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:16:05,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:16:05,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:05,559 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:16:05,561 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.02100014687. 2022-02-08 03:16:05,561 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:16:05,562 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:16:05,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:16:06,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:16:06,200 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:16:06,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:16:06,242 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:16:06,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:16:06,382 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:16:06,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:16:06,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:16:06,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:16:06,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:07,270 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:16:07,270 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:16:09,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:09,493 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:16:09,493 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:16:11,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:11,785 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:16:11,786 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:16:13,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:14,055 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:16:14,056 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:16:16,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:16,213 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:16:16,213 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:16:18,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:18,618 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:16:18,618 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:16:20,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:20,798 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:16:20,799 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:16:22,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:23,101 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:16:23,101 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:16:25,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:25,441 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:16:25,443 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:16:27,443 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:27,717 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:16:27,719 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:16:29,719 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:29,967 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:16:29,967 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:16:31,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:32,388 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:16:32,388 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:16:34,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:34,660 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:16:34,660 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:16:36,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:36,855 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:16:36,855 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:16:38,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:39,026 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:16:39,026 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 33.4649999142. 2022-02-08 03:16:39,026 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:16:39,029 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:16:40,140 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:16:40,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:16:40,706 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:16:40,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:16:40,729 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:16:40,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:16:40,788 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:16:40,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:16:40,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:16:40,892 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:16:40,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:41,536 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:16:41,536 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 03:16:43,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:43,720 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:16:43,720 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 03:16:45,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:45,933 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:16:45,934 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.90799999237. 2022-02-08 03:16:45,934 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:16:45,936 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:16:46,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:16:47,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:16:47,236 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:16:47,237 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:16:47,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:16:47,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:16:47,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:16:47,362 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:16:47,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:16:47,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:16:47,451 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:48,154 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:16:48,154 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.22000002861. 2022-02-08 03:16:48,154 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:16:48,155 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:16:49,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 4673 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:16:49,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:16:49,953 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:16:49,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:16:49,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:16:50,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:16:50,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:16:50,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:16:50,079 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:16:50,127 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:16:50,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:50,940 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:16:50,940 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.78699994087. 2022-02-08 03:16:50,941 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:16:50,943 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:16:52,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:16:52,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:16:52,737 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:16:52,739 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:16:52,765 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:16:52,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:16:52,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:16:52,882 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:16:52,979 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:16:53,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:16:53,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:54,345 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:16:54,345 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 03:16:56,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:56,611 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:16:56,611 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.67000007629. 2022-02-08 03:16:56,611 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:16:56,614 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:16:57,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:16:57,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:16:57,907 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:16:57,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:16:57,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:16:57,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:16:57,999 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:16:58,022 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:16:58,040 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:16:58,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:16:58,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:16:58,632 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:16:58,632 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.02099990845. 2022-02-08 03:16:58,632 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:16:58,634 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:16:59,242 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:16:59,540 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:16:59,631 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:16:59,632 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:16:59,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:16:59,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:16:59,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:16:59,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:16:59,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:16:59,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:16:59,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:17:00,263 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:17:00,265 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.6320002079. 2022-02-08 03:17:00,265 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:17:00,266 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:17:00,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:17:01,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:17:01,645 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:17:01,645 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:17:01,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:17:01,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:17:01,966 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:17:02,094 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:17:02,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:17:02,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:17:02,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:17:02,648 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:17:02,648 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:17:04,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:17:05,207 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:17:05,207 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:17:07,209 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:17:07,976 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:17:07,976 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:17:09,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:17:10,483 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:17:10,483 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:17:12,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:17:12,924 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:17:12,924 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:17:14,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:17:15,132 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:17:15,134 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:17:17,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:17:17,296 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:17:17,296 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 17.0319998264. ############################################################ # # RAM FootPrint: 4168.8984375 # ############################################################ 2022-02-08 03:17:17,332 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 03:17:17,332 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 119 seconds. Reason: Iteration:12 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 4695 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 4717 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:19:16,336 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 03:19:16,336 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 03:19:21,339 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:19:22,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:19:22,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:19:23,078 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:19:23,079 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:19:23,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:19:23,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:19:23,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:19:23,318 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:19:23,369 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:19:23,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:19:23,424 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:19:23,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:19:23,483 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['111897 memcached\n'], []) 2022-02-08 03:19:23,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 03:19:23,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:19:23,592 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['111897 memcached\n'], []) 2022-02-08 03:19:23,594 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:19:23,595 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:19:25,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:19:25,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:19:26,058 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:19:26,058 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:19:26,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:19:26,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:19:26,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:19:26,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:19:26,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:19:26,204 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:19:26,236 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:19:26,236 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:19:26,313 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['20946 memcached\n'], []) 2022-02-08 03:19:26,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 03:19:26,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:19:26,403 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['20946 memcached\n'], []) 2022-02-08 03:19:26,404 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:19:26,405 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:19:27,387 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:19:28,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:19:28,408 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:19:28,408 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:19:28,428 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:19:28,466 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:19:28,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:19:28,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:19:28,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:19:28,586 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:19:28,615 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:19:28,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:19:28,709 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['9771 memcached\n'], []) 2022-02-08 03:19:28,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 03:19:28,747 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:19:28,776 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['9771 memcached\n'], []) 2022-02-08 03:19:28,776 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:19:28,779 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:19:30,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:19:31,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:19:31,908 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:19:31,908 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:19:31,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:19:31,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:19:31,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:19:32,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:19:32,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:19:32,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:19:32,055 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:19:32,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:19:32,125 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['45949 memcached\n'], []) 2022-02-08 03:19:32,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 03:19:32,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:19:32,188 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['45949 memcached\n'], []) 2022-02-08 03:19:32,188 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:19:32,190 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:19:33,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:19:34,604 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:19:34,785 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:19:34,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:19:34,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:19:34,836 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:19:34,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:19:34,869 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:19:34,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:19:34,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:19:34,937 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:19:34,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:19:35,020 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['39776 memcached\n'], []) 2022-02-08 03:19:35,020 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 03:19:35,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:19:35,089 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['39776 memcached\n'], []) 2022-02-08 03:19:35,089 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:19:35,095 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:19:36,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:19:37,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:19:37,332 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:19:37,334 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:19:37,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:19:37,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:19:37,405 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:19:37,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:19:37,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:19:37,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:19:37,519 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:19:37,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:19:37,576 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['80989 memcached\n'], []) 2022-02-08 03:19:37,576 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 03:19:37,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:19:37,638 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['80989 memcached\n'], []) 2022-02-08 03:19:37,638 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:19:37,641 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:19:38,971 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:19:39,609 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:19:39,812 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:19:39,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:19:39,854 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:19:39,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:19:39,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:19:39,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:19:39,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:19:39,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:19:39,990 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:19:39,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:19:40,053 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['74660 memcached\n'], []) 2022-02-08 03:19:40,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 03:19:40,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:19:40,105 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['74660 memcached\n'], []) 2022-02-08 03:19:40,105 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:19:40,108 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:19:41,048 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:19:41,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:19:41,898 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:19:41,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:19:41,913 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:19:41,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:19:41,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:19:41,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:19:41,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:19:42,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:19:42,053 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:19:42,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:19:42,124 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['8503 memcached\n'], []) 2022-02-08 03:19:42,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 03:19:42,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:19:42,184 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['8503 memcached\n'], []) 2022-02-08 03:19:42,184 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:19:42,186 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:19:42,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:19:43,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:19:43,540 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:19:43,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:19:43,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:19:43,573 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:19:43,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:19:43,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:19:43,627 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:19:43,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:19:43,664 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:19:43,665 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:19:43,714 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['88256 memcached\n'], []) 2022-02-08 03:19:43,716 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 03:19:43,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:19:43,759 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['88256 memcached\n'], []) 2022-02-08 03:19:43,759 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:19:43,763 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:19:43,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:19:44,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:19:44,134 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:19:44,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:19:44,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:19:44,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:19:44,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:19:44,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:19:44,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:19:44,224 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:19:44,237 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:19:44,253 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 03:19:44,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:19:44,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:19:46,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:19:47,232 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:19:47,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:19:47,538 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:19:47,552 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 03:19:47,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:19:47,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:19:47,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:19:47,700 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 03:19:47,700 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:19:47,703 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:19:47,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:19:48,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:19:48,071 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:19:48,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:19:48,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:19:48,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:19:48,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:19:48,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:19:48,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:19:48,155 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:19:48,170 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:19:48,184 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 03:19:48,209 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:19:48,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 4739 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:19:51,091 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:19:51,403 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:19:51,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:19:51,749 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:19:51,763 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 03:19:51,765 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:19:51,778 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:19:51,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:19:51,874 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 03:19:51,874 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:19:51,875 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:19:51,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:19:52,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:19:52,256 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:19:52,257 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:19:52,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:19:52,280 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:19:52,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:19:52,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:19:52,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:19:52,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:19:52,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:19:52,368 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 03:19:52,395 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:19:52,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:19:55,237 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:19:55,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:19:55,744 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:19:55,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:19:55,926 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 03:19:55,926 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:19:55,938 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:19:56,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:19:56,016 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 03:19:56,016 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:19:56,019 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 03:19:56,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:19:56,334 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 03:19:56,394 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:19:56,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 03:19:56,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 03:19:56,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 03:19:56,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 03:19:56,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 03:19:56,467 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 03:19:56,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 03:19:56,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:19:56,510 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 03:19:56,535 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:19:56,551 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:19:56,568 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:19:56,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:19:56,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:19:56,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:19:56,630 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 03:19:56,630 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:19:56,642 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:19:56,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:19:56,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:19:56,701 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:19:56,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:19:56,775 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:19:56,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:19:56,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:19:56,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:19:56,920 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:19:56,956 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:19:56,999 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:19:57,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:19:57,045 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:19:57,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:19:57,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:19:57,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:19:57,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:19:57,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:19:57,142 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:19:57,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:19:57,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:19:57,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:19:57,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:19:57,232 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:19:57,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:19:57,267 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:19:57,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:19:57,303 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:19:57,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:19:57,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:19:57,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:19:57,374 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 03:19:57,375 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:19:57,466 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:19:57,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:19:57,983 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:19:57,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:19:58,005 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:19:58,028 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:19:58,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:19:58,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:19:58,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:19:58,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:19:58,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:19:58,265 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 03:19:58,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:19:58,329 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:19:59,729 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:20:00,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:00,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:00,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:20:00,845 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 03:20:00,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:20:00,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:00,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:20:00,915 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 03:20:00,917 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:20:00,918 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 03:20:01,010 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:20:01,242 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 03:20:01,303 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:20:01,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 03:20:01,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 03:20:01,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 03:20:01,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 03:20:01,364 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 03:20:01,377 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 03:20:01,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 03:20:01,405 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:20:01,421 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 03:20:01,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:20:01,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:01,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:20:01,497 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:01,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:01,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:20:01,542 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 03:20:01,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:20:01,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:01,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:20:01,586 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:01,617 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:20:01,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:01,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:01,732 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:01,767 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:01,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:01,849 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:01,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:01,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:01,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:01,969 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:20:01,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:02,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:02,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:02,035 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:02,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:02,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:02,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:02,096 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:02,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:02,127 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:20:02,142 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:02,160 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:02,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:02,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:02,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:02,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:02,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:02,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:02,272 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 03:20:02,273 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:20:02,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:20:02,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:20:02,647 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:20:02,647 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:20:02,663 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:20:02,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:20:02,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:20:02,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:20:02,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:20:02,744 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:20:02,762 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:20:02,779 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 03:20:02,808 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:20:02,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:05,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:20:06,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:06,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:06,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:20:06,430 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 03:20:06,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:20:06,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:06,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:20:06,512 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 03:20:06,512 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:20:06,513 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:20:06,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:20:06,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:20:06,926 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:20:06,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:20:06,956 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:20:06,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:20:07,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:20:07,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:20:07,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:20:07,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:20:07,091 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:20:07,112 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 03:20:07,140 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:20:07,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:09,602 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:20:09,905 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:10,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:10,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:20:10,288 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 03:20:10,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:20:10,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:10,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:20:10,375 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 03:20:10,377 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:20:10,378 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 03:20:10,477 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:20:10,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 03:20:10,809 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:20:10,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 03:20:10,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 03:20:10,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 03:20:10,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 03:20:10,880 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 03:20:10,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 03:20:10,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 03:20:10,931 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:20:10,953 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 03:20:10,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:20:10,999 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:11,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:20:11,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:11,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:11,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:20:11,088 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 03:20:11,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:20:11,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:11,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:20:11,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:11,176 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:20:11,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:11,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:11,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:11,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:11,390 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:11,433 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:11,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:11,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:11,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:11,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:20:11,589 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:11,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:11,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:11,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:11,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:11,683 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:11,701 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:11,721 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:11,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:11,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:20:11,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:11,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:11,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:11,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:11,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:11,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:11,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:11,941 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:11,963 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 03:20:11,966 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 03:20:12,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:20:12,275 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 03:20:12,341 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:20:12,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 03:20:12,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 03:20:12,369 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 03:20:12,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 03:20:12,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 03:20:12,420 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 03:20:12,436 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 03:20:12,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:20:12,464 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 03:20:12,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:20:12,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:12,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:20:12,533 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:12,548 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:12,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:20:12,576 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 03:20:12,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:20:12,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:12,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:20:12,627 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:12,663 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:20:12,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:12,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:12,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:12,822 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:12,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:12,897 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:12,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:12,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:13,005 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:13,022 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:20:13,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:13,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:13,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:13,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:13,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:13,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:13,181 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:13,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:13,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:13,229 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:20:13,247 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:13,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:13,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:13,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:13,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:13,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:13,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:13,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:13,388 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 03:20:13,391 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:20:13,530 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:20:13,806 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:20:13,905 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:20:13,905 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:20:13,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:20:13,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:20:13,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:20:13,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:20:13,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:20:14,010 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:20:14,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:20:14,105 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 03:20:14,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:20:14,147 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:16,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:20:16,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:17,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:17,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:20:17,444 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 03:20:17,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:20:17,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:17,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:20:17,604 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 03:20:17,605 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:20:17,607 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:20:18,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:20:18,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:20:18,819 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:20:18,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:20:18,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:20:18,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:20:18,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:20:18,885 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:20:18,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:20:18,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:20:18,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:20:18,969 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 03:20:18,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:20:19,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:20,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:20:21,122 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:21,257 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:21,392 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:20:21,407 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 03:20:21,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:20:21,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:21,466 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:20:21,492 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 03:20:21,492 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:20:21,493 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:20:21,619 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:20:21,864 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:20:21,933 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:20:21,933 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:20:21,948 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:20:21,964 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:20:21,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:20:22,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:20:22,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:20:22,030 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:20:22,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:20:22,062 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 03:20:22,086 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:20:22,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:25,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:20:25,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:25,734 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:25,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:20:25,959 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 03:20:25,959 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:20:25,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:26,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:20:26,036 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 03:20:26,036 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:20:26,038 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 03:20:26,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:20:26,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 03:20:26,417 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:20:26,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 03:20:26,431 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 03:20:26,446 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 03:20:26,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 03:20:26,482 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 03:20:26,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 03:20:26,507 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 03:20:26,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:20:26,542 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 03:20:26,579 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:20:26,602 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:26,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:20:26,632 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:26,645 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:20:26,660 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:20:26,674 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 03:20:26,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:20:26,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:20:26,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:20:26,719 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:26,755 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:20:26,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:26,834 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:26,877 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:26,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:26,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:26,987 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:27,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:27,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:20:27,091 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:27,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:20:27,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:27,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:27,155 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:27,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:27,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:27,206 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:27,221 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:27,236 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:20:27,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:27,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:20:27,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:27,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:27,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:27,339 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:27,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:27,377 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:27,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:27,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:20:27,430 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 03:20:27,430 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:20:27,431 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:20:27,528 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:20:27,770 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:20:27,838 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:20:27,839 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:20:27,854 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:20:27,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:20:27,890 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:20:27,938 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:20:27,959 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:20:27,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:20:27,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:20:28,506 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:20:28,506 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.07699990273. 2022-02-08 03:20:28,506 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:20:28,509 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:20:28,599 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:20:28,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:20:29,019 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:20:29,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:20:29,035 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:20:29,058 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:20:29,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:20:29,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:20:29,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:20:29,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:20:29,188 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:20:30,072 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:20:30,072 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:20:32,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:20:32,424 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:20:32,424 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:20:34,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:20:34,888 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:20:34,888 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:20:36,890 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:20:37,128 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:20:37,128 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:20:39,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:20:39,306 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:20:39,306 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:20:41,308 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:20:41,625 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:20:41,625 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:20:43,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:20:44,028 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:20:44,029 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:20:46,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:20:46,302 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:20:46,303 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:20:48,303 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:20:48,542 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:20:48,542 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 4760 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:20:50,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:20:50,734 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:20:50,736 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:20:52,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:20:52,910 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:20:52,910 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:20:54,910 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:20:55,085 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:20:55,086 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:20:57,086 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:20:57,267 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:20:57,267 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 28.760999918. 2022-02-08 03:20:57,269 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:20:57,269 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:20:58,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:20:58,512 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:20:58,736 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:20:58,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:20:58,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:20:58,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:20:59,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:20:59,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:20:59,186 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:20:59,272 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:20:59,311 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:21:00,263 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:21:00,263 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 03:21:02,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:21:02,446 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:21:02,447 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 03:21:04,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:21:04,684 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:21:04,684 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 03:21:06,686 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:21:06,930 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:21:06,931 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 9.66299986839. 2022-02-08 03:21:06,931 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:21:06,934 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:21:08,096 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:21:08,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:21:08,586 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:21:08,586 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:21:08,607 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:21:08,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:21:08,647 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:21:08,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:21:08,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:21:08,744 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:21:08,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:21:09,470 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:21:09,470 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.53900003433. 2022-02-08 03:21:09,471 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:21:09,473 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:21:10,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:21:10,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:21:10,568 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:21:10,568 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:21:10,589 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:21:10,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:21:10,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:21:10,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:21:10,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:21:10,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:21:10,788 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:21:11,494 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:21:11,496 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.02400016785. 2022-02-08 03:21:11,496 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:21:11,497 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:21:12,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:21:12,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:21:13,108 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:21:13,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:21:13,130 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:21:13,158 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:21:13,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:21:13,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:21:13,244 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:21:13,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:21:13,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:21:13,927 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:21:13,927 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.43099999428. 2022-02-08 03:21:13,927 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:21:13,928 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:21:14,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:21:15,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:21:15,720 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:21:15,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:21:15,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:21:15,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:21:15,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:21:15,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:21:15,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:21:16,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:21:16,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:21:16,953 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:21:16,953 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.02699995041. 2022-02-08 03:21:16,954 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:21:16,957 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:21:17,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:21:17,839 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:21:17,936 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:21:17,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:21:17,951 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:21:17,969 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:21:17,982 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:21:18,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:21:18,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:21:18,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:21:18,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:21:18,594 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:21:18,595 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.6400001049. 2022-02-08 03:21:18,595 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:21:18,596 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:21:18,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:21:19,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:21:19,720 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:21:19,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:21:19,831 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:21:20,017 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:21:20,086 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:21:20,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:21:20,352 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:21:20,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:21:20,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:21:21,948 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:21:21,950 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:21:23,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:21:24,335 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:21:24,335 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:21:26,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:21:27,198 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:21:27,198 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:21:29,198 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:21:29,542 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:21:29,542 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:21:31,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:21:32,049 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:21:32,049 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:21:34,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:21:34,224 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:21:34,226 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:21:36,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:21:36,418 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:21:36,420 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 17.8250000477. ############################################################ # # RAM FootPrint: 4174.56640625 # ############################################################ 2022-02-08 03:21:36,454 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 03:21:36,454 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 108 seconds. Reason: Iteration:13 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 4786 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 4812 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:23:24,464 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 03:23:24,464 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 03:23:29,467 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:23:30,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:23:31,102 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:23:31,391 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:23:31,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:23:31,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:23:31,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:23:31,545 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:23:31,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:23:31,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:23:31,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:23:31,670 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:23:31,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:23:31,724 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['112640 memcached\n'], []) 2022-02-08 03:23:31,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 03:23:31,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:23:31,802 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['112640 memcached\n'], []) 2022-02-08 03:23:31,802 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:23:31,805 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:23:32,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:23:33,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:23:33,576 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:23:33,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:23:33,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:23:33,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:23:33,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:23:33,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:23:33,808 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:23:33,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:23:33,861 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:23:33,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:23:33,940 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['21856 memcached\n'], []) 2022-02-08 03:23:33,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 03:23:33,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:23:34,000 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['21856 memcached\n'], []) 2022-02-08 03:23:34,000 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:23:34,003 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:23:35,339 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:23:36,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:23:36,617 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:23:36,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:23:36,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:23:36,673 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:23:36,691 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:23:36,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:23:36,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:23:36,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:23:36,838 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:23:36,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:23:36,914 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['10637 memcached\n'], []) 2022-02-08 03:23:36,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 03:23:36,951 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:23:36,980 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['10637 memcached\n'], []) 2022-02-08 03:23:36,980 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:23:36,983 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:23:37,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:23:38,660 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:23:38,849 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:23:38,849 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:23:38,865 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:23:38,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:23:38,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:23:38,936 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:23:38,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:23:38,984 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:23:39,017 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:23:39,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:23:39,078 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['46690 memcached\n'], []) 2022-02-08 03:23:39,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 03:23:39,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:23:39,137 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['46690 memcached\n'], []) 2022-02-08 03:23:39,137 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:23:39,141 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:23:39,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:23:40,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:23:41,250 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:23:41,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:23:41,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:23:41,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:23:41,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:23:41,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:23:41,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:23:41,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:23:41,444 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:23:41,446 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:23:41,509 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['40523 memcached\n'], []) 2022-02-08 03:23:41,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 03:23:41,540 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:23:41,565 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['40523 memcached\n'], []) 2022-02-08 03:23:41,565 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:23:41,568 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:23:42,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:23:42,867 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:23:43,148 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:23:43,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:23:43,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:23:43,387 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:23:43,413 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:23:43,436 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:23:43,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:23:43,515 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:23:43,532 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:23:43,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:23:43,595 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['81797 memcached\n'], []) 2022-02-08 03:23:43,596 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 03:23:43,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:23:43,647 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['81797 memcached\n'], []) 2022-02-08 03:23:43,648 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:23:43,650 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:23:44,676 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:23:45,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:23:46,019 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:23:46,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:23:46,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:23:46,058 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:23:46,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:23:46,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:23:46,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:23:46,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:23:46,168 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:23:46,168 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:23:46,223 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['75402 memcached\n'], []) 2022-02-08 03:23:46,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 03:23:46,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:23:46,269 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['75402 memcached\n'], []) 2022-02-08 03:23:46,269 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:23:46,272 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:23:47,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:23:47,788 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:23:47,974 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:23:47,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:23:47,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:23:48,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:23:48,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:23:48,066 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:23:48,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:23:48,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:23:48,127 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:23:48,127 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:23:48,174 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['9250 memcached\n'], []) 2022-02-08 03:23:48,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 03:23:48,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:23:48,226 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['9250 memcached\n'], []) 2022-02-08 03:23:48,226 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:23:48,229 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:23:48,887 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 4836 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:23:49,318 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:23:49,494 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:23:49,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:23:49,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:23:49,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:23:49,568 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:23:49,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:23:49,614 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:23:49,645 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:23:49,661 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:23:49,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:23:49,739 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['89071 memcached\n'], []) 2022-02-08 03:23:49,739 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 03:23:49,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:23:49,809 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['89071 memcached\n'], []) 2022-02-08 03:23:49,809 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:23:49,812 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:23:49,890 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:23:50,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:23:50,188 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:23:50,188 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:23:50,206 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:23:50,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:23:50,239 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:23:50,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:23:50,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:23:50,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:23:50,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:23:50,311 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 03:23:50,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:23:50,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:23:52,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:23:52,969 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:23:53,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:23:53,285 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:23:53,299 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 03:23:53,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:23:53,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:23:53,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:23:53,377 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 03:23:53,377 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:23:53,378 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:23:53,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:23:53,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:23:53,736 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:23:53,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:23:53,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:23:53,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:23:53,778 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:23:53,793 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:23:53,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:23:53,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:23:53,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:23:53,845 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 03:23:53,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:23:53,892 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:23:56,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:23:57,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:23:57,181 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:23:57,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:23:57,382 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 03:23:57,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:23:57,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:23:57,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:23:57,471 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 03:23:57,471 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:23:57,474 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:23:57,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:23:57,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:23:57,793 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:23:57,793 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:23:57,806 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:23:57,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:23:57,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:23:57,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:23:57,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:23:57,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:23:57,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:23:57,901 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 03:23:57,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:23:57,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:00,693 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:24:01,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:01,198 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:01,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:24:01,391 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 03:24:01,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:24:01,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:01,461 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:24:01,476 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 03:24:01,476 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:24:01,479 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 03:24:01,589 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:24:01,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 03:24:01,936 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:24:01,936 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 03:24:01,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 03:24:01,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 03:24:01,982 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 03:24:01,999 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 03:24:02,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 03:24:02,028 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 03:24:02,045 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:24:02,061 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 03:24:02,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:24:02,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:02,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:24:02,135 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:02,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:02,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:24:02,176 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 03:24:02,176 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:24:02,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:02,204 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:24:02,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:02,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:24:02,293 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:02,329 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:02,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:02,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:02,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:02,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:02,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:02,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:02,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:02,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:24:02,630 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:02,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:02,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:02,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:02,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:02,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:02,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:02,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:02,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:02,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:24:02,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:02,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:02,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:02,849 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:02,867 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:02,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:02,903 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:02,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:02,943 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 03:24:02,947 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:24:03,153 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:24:03,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:24:03,970 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:24:03,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:24:03,992 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:24:04,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:24:04,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:24:04,211 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:24:04,272 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:24:04,403 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:24:04,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:24:04,492 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 03:24:04,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:24:04,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:06,306 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:24:06,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:06,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:06,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:24:06,730 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 03:24:06,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:24:06,778 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:06,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:24:06,864 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 03:24:06,865 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:24:06,867 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 03:24:06,946 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:24:07,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 03:24:07,242 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:24:07,242 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 03:24:07,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 03:24:07,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 03:24:07,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 03:24:07,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 03:24:07,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 03:24:07,334 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 03:24:07,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:24:07,367 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 03:24:07,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:24:07,410 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:07,426 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:24:07,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:07,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:07,469 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:24:07,483 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 03:24:07,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:24:07,499 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:07,515 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:24:07,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:07,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:24:07,604 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:07,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:07,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:07,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:07,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:07,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:07,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:07,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:07,895 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:07,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:24:07,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:07,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:07,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:07,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:07,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:08,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:08,020 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:08,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:08,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:08,066 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:24:08,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:08,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:08,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:08,132 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:08,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:08,165 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:08,183 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:08,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:08,216 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 03:24:08,219 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:24:08,308 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:24:08,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:24:08,565 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:24:08,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:24:08,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:24:08,589 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:24:08,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:24:08,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:24:08,630 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:24:08,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:24:08,655 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:24:08,670 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 03:24:08,696 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:24:08,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:11,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:24:11,548 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:11,739 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:11,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:24:11,923 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 03:24:11,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:24:11,936 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:11,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:24:11,987 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 03:24:11,989 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:24:11,990 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:24:12,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:24:12,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:24:12,436 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:24:12,436 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:24:12,451 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:24:12,466 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:24:12,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:24:12,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:24:12,551 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:24:12,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:24:12,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:24:12,719 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 03:24:12,744 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:24:12,760 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:15,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:24:15,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:15,964 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:16,206 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:24:16,269 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 03:24:16,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:24:16,288 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:16,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:24:16,377 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 03:24:16,377 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:24:16,380 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 03:24:16,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:24:16,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 03:24:16,767 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:24:16,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 03:24:16,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 03:24:16,798 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 03:24:16,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 03:24:16,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 03:24:16,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 03:24:16,865 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 03:24:16,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:24:16,904 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 03:24:16,928 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:24:16,946 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:16,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:24:16,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:16,994 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:17,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:24:17,026 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 03:24:17,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:24:17,043 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:17,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:24:17,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:17,096 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:24:17,114 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:17,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:17,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:17,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:17,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:17,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:17,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:17,234 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:17,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:17,267 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:24:17,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:17,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:17,318 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:17,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:17,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:17,369 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:17,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:17,405 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:17,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:17,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:24:17,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:17,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:17,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:17,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:17,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:17,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:17,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:17,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:17,632 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:17,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:24:17,676 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:17,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:17,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:17,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:17,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:17,775 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:17,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:17,816 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:17,835 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 03:24:17,838 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 03:24:17,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:24:18,155 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 03:24:18,217 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:24:18,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 03:24:18,234 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 03:24:18,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 03:24:18,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 03:24:18,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 03:24:18,295 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 03:24:18,308 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 03:24:18,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:24:18,339 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 03:24:18,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:24:18,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:18,400 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:24:18,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:18,428 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:18,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:24:18,457 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 03:24:18,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:24:18,471 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:18,487 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:24:18,502 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:18,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:24:18,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:18,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:18,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:18,579 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:18,594 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:18,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:18,622 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:18,637 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:18,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:18,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:24:18,681 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:18,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:18,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:18,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:18,742 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:18,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:18,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:18,788 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:18,803 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:18,822 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:24:18,841 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:18,865 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:18,882 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:18,900 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:18,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:18,931 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:18,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:18,966 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:18,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:19,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:24:19,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:19,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:19,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:19,096 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:19,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:19,145 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:19,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:19,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:19,200 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 03:24:19,203 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:24:19,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:24:19,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:24:19,696 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:24:19,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:24:19,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:24:19,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:24:19,762 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:24:19,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:24:19,816 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:24:19,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:24:19,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:24:19,901 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 03:24:19,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:24:19,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:22,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:24:22,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:23,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:23,357 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:24:23,410 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 03:24:23,410 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:24:23,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:23,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:24:23,493 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 03:24:23,493 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:24:23,496 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:24:24,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:24:24,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:24:24,749 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:24:24,749 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:24:24,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:24:24,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:24:24,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:24:24,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:24:24,836 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:24:24,867 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:24:24,903 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:24:24,918 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 03:24:24,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:24:24,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:26,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:24:27,147 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:27,290 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:27,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:24:27,437 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 03:24:27,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:24:27,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:27,499 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:24:27,513 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 03:24:27,513 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:24:27,516 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:24:27,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:24:27,915 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:24:28,013 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:24:28,015 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:24:28,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:24:28,063 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:24:28,079 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:24:28,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:24:28,132 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:24:28,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:24:28,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:24:28,237 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 03:24:28,262 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:24:28,318 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:31,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:24:31,596 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:31,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:32,035 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:24:32,052 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 03:24:32,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:24:32,068 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:32,104 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:24:32,122 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 03:24:32,124 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:24:32,125 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 03:24:32,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:24:32,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 03:24:32,532 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:24:32,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 03:24:32,548 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 03:24:32,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 03:24:32,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 03:24:32,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 03:24:32,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 03:24:32,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 03:24:32,637 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:24:32,653 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 03:24:32,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:24:32,696 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:32,711 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:24:32,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:32,742 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:24:32,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:24:32,769 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 03:24:32,770 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:24:32,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:24:32,801 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:24:32,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:32,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:24:32,844 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:32,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:32,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:32,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:32,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:32,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:32,938 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:32,954 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:24:32,969 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:32,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:24:33,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:33,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:33,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:33,058 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:33,073 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:33,091 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:33,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:33,122 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:24:33,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:33,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:24:33,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:33,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:33,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:33,227 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:33,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:33,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:33,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:33,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:24:33,334 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:33,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:24:33,369 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:33,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:33,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:33,423 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:33,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:33,459 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:33,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:33,497 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:24:33,515 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 03:24:33,516 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:24:33,517 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:24:33,599 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:24:33,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:24:33,913 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:24:33,913 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:24:33,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:24:33,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:24:33,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:24:33,997 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:24:34,012 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:24:34,028 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:24:34,043 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:24:34,555 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:24:34,556 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.04099988937. 2022-02-08 03:24:34,556 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:24:34,559 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:24:34,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:24:34,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:24:35,030 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:24:35,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:24:35,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:24:35,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:24:35,114 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:24:35,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:24:35,176 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:24:35,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:24:35,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:24:35,999 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:24:35,999 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:24:37,999 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:24:38,168 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:24:38,170 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:24:40,170 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:24:40,430 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:24:40,430 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:24:42,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:24:42,701 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:24:42,703 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:24:44,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:24:44,923 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:24:44,923 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:24:46,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:24:47,138 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:24:47,138 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:24:49,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 4863 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:24:49,457 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:24:49,457 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:24:51,459 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:24:51,661 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:24:51,661 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:24:53,663 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:24:53,953 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:24:53,953 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:24:55,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:24:56,177 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:24:56,177 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:24:58,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:24:58,525 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:24:58,525 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:25:00,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:25:00,703 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:25:00,703 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:25:02,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:25:02,993 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:25:02,993 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:25:04,994 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:25:05,207 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:25:05,207 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:25:07,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:25:07,401 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:25:07,401 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:25:09,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:25:09,720 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:25:09,721 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:25:11,721 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:25:11,969 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:25:11,969 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:25:13,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:25:14,239 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:25:14,240 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:25:16,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:25:16,441 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:25:16,443 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 41.8860001564. 2022-02-08 03:25:16,443 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:25:16,444 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:25:17,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:25:17,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:25:17,736 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:25:17,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:25:17,762 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:25:17,790 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:25:17,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:25:17,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:25:17,859 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:25:17,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:25:17,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:25:18,608 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:25:18,608 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.16499996185. 2022-02-08 03:25:18,608 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:25:18,609 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:25:19,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:25:19,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:25:19,809 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:25:19,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:25:19,867 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:25:19,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:25:19,920 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:25:19,956 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:25:19,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:25:20,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:25:20,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:25:20,775 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:25:20,778 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.16900014877. 2022-02-08 03:25:20,778 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:25:20,779 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:25:21,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:25:22,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:25:22,667 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:25:22,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:25:22,729 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:25:22,765 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:25:22,795 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:25:22,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:25:22,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:25:22,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:25:23,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:25:24,214 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:25:24,214 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 03:25:26,216 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:25:26,390 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:25:26,391 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.61299991608. 2022-02-08 03:25:26,391 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:25:26,392 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:25:27,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:25:27,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:25:27,621 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:25:27,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:25:27,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:25:27,673 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:25:27,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:25:27,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:25:27,734 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:25:27,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:25:27,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:25:28,411 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:25:28,411 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.01999998093. 2022-02-08 03:25:28,411 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:25:28,413 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:25:29,224 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:25:29,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:25:29,760 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:25:29,762 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:25:29,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:25:29,808 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:25:29,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:25:29,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:25:29,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:25:29,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:25:29,984 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:25:30,615 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:25:30,615 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.20300006866. 2022-02-08 03:25:30,615 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:25:30,618 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:25:31,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:25:31,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:25:31,638 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:25:31,640 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:25:31,655 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:25:31,681 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:25:31,696 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:25:31,711 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:25:31,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:25:31,757 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:25:31,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:25:32,345 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:25:32,345 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.73099994659. 2022-02-08 03:25:32,345 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:25:32,348 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:25:33,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:25:33,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:25:33,961 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:25:33,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:25:34,005 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:25:34,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:25:34,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:25:34,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:25:34,183 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:25:34,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:25:34,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:25:34,838 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:25:34,839 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.49399995804. ############################################################ # # RAM FootPrint: 4174.65234375 # ############################################################ 2022-02-08 03:25:34,874 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 03:25:34,875 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 99 seconds. Reason: Iteration:14 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 4885 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 4909 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:27:13,881 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 03:27:13,881 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 03:27:18,884 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:27:19,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:27:20,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:27:20,598 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:27:20,599 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:27:20,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:27:20,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:27:20,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:27:20,864 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:27:20,959 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:27:21,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:27:21,114 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:27:21,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:27:21,207 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['113375 memcached\n'], []) 2022-02-08 03:27:21,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 03:27:21,257 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:27:21,279 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['113375 memcached\n'], []) 2022-02-08 03:27:21,279 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:27:21,282 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:27:22,229 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:27:22,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:27:22,901 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:27:22,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:27:22,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:27:22,951 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:27:22,982 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:27:23,038 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:27:23,061 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:27:23,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:27:23,141 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:27:23,142 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:27:23,240 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['22761 memcached\n'], []) 2022-02-08 03:27:23,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 03:27:23,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:27:23,318 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['22761 memcached\n'], []) 2022-02-08 03:27:23,319 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:27:23,321 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:27:24,221 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:27:24,821 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:27:24,999 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:27:25,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:27:25,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:27:25,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:27:25,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:27:25,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:27:25,176 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:27:25,239 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:27:25,269 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:27:25,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:27:25,349 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['11517 memcached\n'], []) 2022-02-08 03:27:25,349 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 03:27:25,385 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:27:25,414 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['11517 memcached\n'], []) 2022-02-08 03:27:25,414 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:27:25,417 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:27:26,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:27:26,821 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:27:27,107 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:27:27,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:27:27,127 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:27:27,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:27:27,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:27:27,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:27:27,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:27:27,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:27:27,298 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:27:27,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:27:27,362 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['47425 memcached\n'], []) 2022-02-08 03:27:27,362 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 03:27:27,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:27:27,418 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['47425 memcached\n'], []) 2022-02-08 03:27:27,420 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:27:27,421 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:27:28,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:27:28,765 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:27:28,937 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:27:28,938 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:27:28,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:27:28,982 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:27:29,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:27:29,035 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:27:29,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:27:29,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:27:29,104 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:27:29,104 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:27:29,236 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['41256 memcached\n'], []) 2022-02-08 03:27:29,237 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 03:27:29,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:27:29,295 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['41256 memcached\n'], []) 2022-02-08 03:27:29,296 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:27:29,298 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:27:30,469 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:27:31,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:27:31,295 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:27:31,295 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:27:31,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:27:31,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:27:31,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:27:31,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:27:31,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:27:31,413 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:27:31,433 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:27:31,433 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:27:31,530 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['82536 memcached\n'], []) 2022-02-08 03:27:31,530 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 03:27:31,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:27:31,615 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['82536 memcached\n'], []) 2022-02-08 03:27:31,615 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:27:31,617 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:27:32,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:27:32,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:27:33,114 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:27:33,114 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:27:33,176 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:27:33,290 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:27:33,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:27:33,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:27:33,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:27:33,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:27:33,492 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:27:33,492 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:27:33,575 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['76133 memcached\n'], []) 2022-02-08 03:27:33,576 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 03:27:33,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:27:33,628 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['76133 memcached\n'], []) 2022-02-08 03:27:33,628 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:27:33,631 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:27:34,428 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:27:35,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:27:35,140 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:27:35,140 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:27:35,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:27:35,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:27:35,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:27:35,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:27:35,221 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:27:35,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:27:35,273 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:27:35,275 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:27:35,335 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['9988 memcached\n'], []) 2022-02-08 03:27:35,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 03:27:35,387 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:27:35,408 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['9988 memcached\n'], []) 2022-02-08 03:27:35,410 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:27:35,411 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:27:36,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:27:36,678 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:27:36,894 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:27:36,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:27:36,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:27:36,979 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:27:37,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:27:37,022 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:27:37,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:27:37,086 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:27:37,105 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:27:37,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:27:37,164 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['89924 memcached\n'], []) 2022-02-08 03:27:37,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 03:27:37,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:27:37,216 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['89924 memcached\n'], []) 2022-02-08 03:27:37,217 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:27:37,217 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:27:37,293 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:27:37,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:27:37,568 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:27:37,568 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:27:37,584 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:27:37,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:27:37,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:27:37,630 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:27:37,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:27:37,658 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:27:37,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:27:37,693 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 03:27:37,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:27:37,744 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:27:39,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:27:40,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:27:40,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:27:40,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:27:40,345 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 03:27:40,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:27:40,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:27:40,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:27:40,421 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 03:27:40,421 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:27:40,426 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:27:40,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:27:40,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:27:40,786 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:27:40,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:27:40,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:27:40,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:27:40,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:27:40,841 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:27:40,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:27:40,865 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:27:40,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:27:40,894 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 03:27:40,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:27:40,936 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:27:43,760 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:27:44,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:27:44,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:27:44,477 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:27:44,496 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 03:27:44,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:27:44,512 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:27:44,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:27:44,586 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 03:27:44,586 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:27:44,588 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:27:44,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:27:44,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:27:44,947 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:27:44,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:27:44,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:27:44,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:27:44,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:27:45,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:27:45,020 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:27:45,033 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:27:45,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:27:45,062 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 03:27:45,089 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:27:45,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:27:47,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:27:48,255 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:27:48,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:27:48,619 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:27:48,634 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 03:27:48,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:27:48,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:27:48,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:27:48,729 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 03:27:48,730 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:27:48,732 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 03:27:48,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:27:49,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 03:27:49,134 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:27:49,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 03:27:49,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 03:27:49,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 03:27:49,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 03:27:49,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 03:27:49,211 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 5 queries submitted, 0 failed, 4935 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:27:49,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 03:27:49,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:27:49,257 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 03:27:49,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:27:49,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:27:49,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:27:49,326 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:27:49,341 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:27:49,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:27:49,371 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 03:27:49,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:27:49,385 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:27:49,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:27:49,415 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:49,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:27:49,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:49,461 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:49,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:49,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:49,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:49,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:49,543 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:49,561 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:49,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:49,594 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:27:49,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:49,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:49,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:49,660 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:49,676 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:49,691 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:49,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:49,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:49,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:49,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:27:49,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:49,785 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:49,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:49,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:49,829 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:49,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:49,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:49,877 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:49,892 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:49,910 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:27:49,928 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:49,948 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:49,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:49,984 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:50,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:50,020 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:50,038 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:50,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:50,073 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 03:27:50,075 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:27:50,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:27:50,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:27:51,278 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:27:51,278 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:27:51,334 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:27:51,428 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:27:51,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:27:51,742 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:27:51,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:27:52,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:27:52,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:27:52,509 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 03:27:52,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:27:52,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:27:54,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:27:54,579 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:27:54,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:27:55,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:27:55,335 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 03:27:55,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:27:55,423 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:27:55,622 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:27:55,753 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 03:27:55,753 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:27:55,755 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 03:27:55,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:27:56,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 03:27:56,134 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:27:56,135 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 03:27:56,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 03:27:56,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 03:27:56,181 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 03:27:56,201 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 03:27:56,216 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 03:27:56,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 03:27:56,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:27:56,263 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 03:27:56,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:27:56,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:27:56,316 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:27:56,331 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:27:56,344 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:27:56,357 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:27:56,371 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 03:27:56,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:27:56,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:27:56,400 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:27:56,413 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:56,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:27:56,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:56,461 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:56,477 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:56,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:56,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:56,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:56,543 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:56,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:27:56,576 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:56,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:27:56,630 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:56,647 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:56,665 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:56,683 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:56,701 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:56,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:56,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:56,752 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:27:56,767 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:56,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:27:56,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:56,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:56,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:56,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:56,865 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:56,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:56,895 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:56,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:27:56,926 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:56,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:27:56,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:56,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:56,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:57,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:57,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:57,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:57,068 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:57,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:27:57,101 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 03:27:57,102 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:27:57,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:27:57,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:27:57,469 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:27:57,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:27:57,484 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:27:57,497 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:27:57,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:27:57,530 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:27:57,545 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:27:57,561 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:27:57,576 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:27:57,594 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 03:27:57,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:27:57,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:28:00,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:28:00,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:28:00,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:28:00,966 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:28:00,982 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 03:28:00,982 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:28:01,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:28:01,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:28:01,061 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 03:28:01,062 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:28:01,062 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:28:01,147 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:28:01,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:28:01,490 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:28:01,492 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:28:01,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:28:01,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:28:01,548 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:28:01,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:28:01,604 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:28:01,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:28:01,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:28:01,660 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 03:28:01,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:28:01,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:28:04,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:28:04,696 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:28:04,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:28:05,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:28:05,088 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 03:28:05,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:28:05,104 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:28:05,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:28:05,170 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 03:28:05,171 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:28:05,173 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 03:28:05,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:28:05,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 03:28:05,569 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:28:05,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 03:28:05,586 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 03:28:05,609 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 03:28:05,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 03:28:05,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 03:28:05,663 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 03:28:05,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 03:28:05,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:28:05,710 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 03:28:05,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:28:05,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:28:05,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:28:05,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:28:05,822 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:28:05,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:28:05,861 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 03:28:05,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:28:05,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:28:05,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:28:05,905 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:05,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:28:05,938 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:05,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:05,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:05,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:06,007 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:06,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:06,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:06,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:06,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:06,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:28:06,109 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:06,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:06,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:06,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:06,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:06,198 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:06,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:06,234 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:06,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:06,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:28:06,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:06,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:06,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:06,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:06,392 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:06,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:06,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:06,454 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:06,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:06,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:28:06,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:06,533 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:06,551 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:06,566 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:06,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:06,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:06,617 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:06,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:06,654 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 03:28:06,655 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 03:28:06,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:28:06,979 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 03:28:07,043 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:28:07,045 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 03:28:07,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 03:28:07,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 03:28:07,086 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 03:28:07,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 03:28:07,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 03:28:07,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 03:28:07,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:28:07,170 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 03:28:07,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:28:07,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:28:07,232 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:28:07,247 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:28:07,262 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:28:07,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:28:07,290 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 03:28:07,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:28:07,306 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:28:07,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:28:07,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:07,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:28:07,372 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:07,390 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:07,410 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:07,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:07,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:07,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:07,482 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:07,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:07,517 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:07,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:28:07,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:07,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:07,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:07,604 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:07,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:07,637 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:07,653 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:07,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:07,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:07,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:28:07,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:07,739 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:07,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:07,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:07,803 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:07,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:07,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:07,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:07,865 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:07,882 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:28:07,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:07,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:07,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:07,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:07,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:07,984 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:08,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:08,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:08,038 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 03:28:08,039 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:28:08,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:28:08,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:28:08,431 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:28:08,431 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:28:08,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:28:08,461 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:28:08,477 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:28:08,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:28:08,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:28:08,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:28:08,561 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:28:08,578 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 03:28:08,614 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:28:08,632 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:28:11,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:28:11,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:28:11,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:28:11,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:28:11,786 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 03:28:11,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:28:11,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:28:11,836 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:28:11,849 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 03:28:11,851 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:28:11,852 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:28:12,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:28:12,933 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:28:13,048 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:28:13,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:28:13,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:28:13,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:28:13,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:28:13,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:28:13,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:28:13,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:28:13,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:28:13,226 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 03:28:13,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:28:13,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:28:15,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:28:15,770 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:28:15,908 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:28:16,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:28:16,061 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 03:28:16,061 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:28:16,079 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:28:16,122 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:28:16,138 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 03:28:16,138 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:28:16,141 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:28:16,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:28:16,474 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:28:16,546 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:28:16,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:28:16,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:28:16,584 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:28:16,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:28:16,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:28:16,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:28:16,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:28:16,676 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:28:16,697 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 03:28:16,729 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:28:16,749 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:28:19,760 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:28:20,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:28:20,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:28:20,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:28:20,559 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 03:28:20,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:28:20,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:28:20,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:28:20,637 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 03:28:20,637 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:28:20,638 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 03:28:20,757 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:28:20,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 03:28:21,059 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:28:21,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 03:28:21,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 03:28:21,089 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 03:28:21,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 03:28:21,122 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 03:28:21,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 03:28:21,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 03:28:21,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:28:21,184 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 03:28:21,219 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:28:21,234 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:28:21,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:28:21,267 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:28:21,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:28:21,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:28:21,311 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 03:28:21,311 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:28:21,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:28:21,341 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:28:21,357 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:21,374 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:28:21,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:21,408 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:21,436 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:21,466 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:21,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:21,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:21,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:21,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:28:21,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:21,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:28:21,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:21,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:21,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:21,632 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:21,650 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:21,665 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:21,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:21,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:28:21,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:21,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:28:21,752 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:21,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:21,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:21,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:21,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:21,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:21,867 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:21,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:28:21,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:21,918 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:28:21,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:21,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:21,982 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:21,999 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:22,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:22,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:22,066 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:22,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:28:22,109 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 03:28:22,111 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:28:22,112 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:28:22,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:28:22,443 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:28:22,529 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:28:22,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:28:22,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:28:22,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:28:22,573 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:28:22,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:28:22,609 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:28:22,622 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:28:22,640 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:28:23,243 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:28:23,243 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.132999897. 2022-02-08 03:28:23,243 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:28:23,246 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:28:23,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:28:23,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:28:23,671 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:28:23,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:28:23,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:28:23,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:28:23,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:28:23,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:28:23,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:28:23,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:28:23,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:28:24,438 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:28:24,438 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:28:26,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:28:26,608 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:28:26,608 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:28:28,609 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:28:28,877 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:28:28,878 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:28:30,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:28:31,220 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:28:31,220 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:28:33,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:28:33,474 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:28:33,474 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:28:35,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:28:35,667 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:28:35,668 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:28:37,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:28:37,848 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:28:37,848 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:28:39,849 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:28:40,042 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:28:40,042 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:28:42,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:28:42,275 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:28:42,275 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:28:44,275 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:28:44,569 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:28:44,569 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:28:46,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:28:46,983 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:28:46,984 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:28:48,984 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:28:49,171 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:28:49,171 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 4959 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:28:51,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:28:51,348 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:28:51,348 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:28:53,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:28:53,546 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:28:53,548 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 30.3040001392. 2022-02-08 03:28:53,548 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:28:53,549 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:28:54,354 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:28:54,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:28:54,990 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:28:54,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:28:55,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:28:55,119 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:28:55,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:28:55,252 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:28:55,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:28:55,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:28:55,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:28:56,095 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:28:56,095 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 03:28:58,096 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:28:58,286 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:28:58,286 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 03:29:00,288 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:29:00,450 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:29:00,450 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 03:29:02,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:29:02,665 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:29:02,665 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 03:29:04,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:29:04,871 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:29:04,872 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 11.3249998093. 2022-02-08 03:29:04,872 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:29:04,875 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:29:05,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:29:06,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:29:06,178 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:29:06,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:29:06,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:29:06,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:29:06,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:29:06,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:29:06,323 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:29:06,369 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:29:06,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:29:07,089 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:29:07,089 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.21700000763. 2022-02-08 03:29:07,091 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:29:07,092 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:29:07,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:29:08,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:29:08,242 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:29:08,242 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:29:08,262 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:29:08,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:29:08,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:29:08,329 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:29:08,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:29:08,390 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:29:08,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:29:09,038 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:29:09,039 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.94799995422. 2022-02-08 03:29:09,039 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:29:09,040 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:29:09,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:29:10,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:29:10,272 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:29:10,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:29:10,290 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:29:10,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:29:10,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:29:10,403 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:29:10,418 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:29:10,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:29:10,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:29:11,099 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:29:11,101 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.06200003624. 2022-02-08 03:29:11,101 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:29:11,102 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:29:12,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:29:12,596 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:29:12,969 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:29:12,969 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:29:13,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:29:13,119 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:29:13,242 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:29:13,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:29:13,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:29:13,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:29:13,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:29:14,904 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:29:14,904 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 03:29:16,905 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:29:17,086 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:29:17,088 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.98599982262. 2022-02-08 03:29:17,088 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:29:17,088 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:29:17,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:29:17,977 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:29:18,092 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:29:18,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:29:18,112 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:29:18,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:29:18,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:29:18,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:29:18,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:29:18,206 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:29:18,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:29:18,690 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:29:18,691 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.60399985313. 2022-02-08 03:29:18,691 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:29:18,694 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:29:18,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:29:19,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:29:19,854 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:29:19,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:29:19,900 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:29:20,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:29:20,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:29:20,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:29:20,267 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:29:20,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:29:20,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:29:21,690 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:29:21,690 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:29:23,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:29:23,951 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:29:23,953 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:29:25,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:29:26,142 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:29:26,142 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.4509999752. ############################################################ # # RAM FootPrint: 4168.0078125 # ############################################################ 2022-02-08 03:29:26,174 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 03:29:26,174 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 94 seconds. Reason: Iteration:15 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 4984 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 5002 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:31:00,178 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 03:31:00,178 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 03:31:05,183 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:31:06,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:06,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:31:07,198 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:07,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:31:07,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:31:07,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:31:07,467 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:31:07,525 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:31:07,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:31:07,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:31:07,680 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:31:07,681 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:31:07,811 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['114090 memcached\n'], []) 2022-02-08 03:31:07,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 03:31:07,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:31:07,881 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['114090 memcached\n'], []) 2022-02-08 03:31:07,881 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:31:07,884 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:31:08,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:09,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:31:09,588 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:09,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:31:09,640 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:31:09,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:31:09,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:31:09,798 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:31:09,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:31:09,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:31:09,868 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:31:09,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:31:09,927 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['23691 memcached\n'], []) 2022-02-08 03:31:09,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 03:31:09,961 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:31:09,992 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['23691 memcached\n'], []) 2022-02-08 03:31:09,992 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:31:09,993 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:31:10,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:11,408 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:31:11,676 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:11,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:31:11,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:31:11,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:31:11,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:31:11,788 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:31:11,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:31:11,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:31:11,894 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:31:11,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:31:11,970 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['12348 memcached\n'], []) 2022-02-08 03:31:11,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 03:31:12,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:31:12,036 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['12348 memcached\n'], []) 2022-02-08 03:31:12,036 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:31:12,038 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:31:12,999 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:13,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:31:13,799 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:13,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:31:13,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:31:13,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:31:13,874 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:31:13,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:31:13,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:31:14,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:31:14,029 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:31:14,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:31:14,095 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['48166 memcached\n'], []) 2022-02-08 03:31:14,096 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 03:31:14,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:31:14,150 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['48166 memcached\n'], []) 2022-02-08 03:31:14,150 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:31:14,151 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:31:15,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:15,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:31:15,790 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:15,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:31:15,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:31:15,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:31:15,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:31:15,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:31:15,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:31:15,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:31:16,017 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:31:16,017 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:31:16,108 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['41983 memcached\n'], []) 2022-02-08 03:31:16,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 03:31:16,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:31:16,161 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['41983 memcached\n'], []) 2022-02-08 03:31:16,161 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:31:16,164 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:31:16,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:17,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:31:17,411 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:17,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:31:17,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:31:17,448 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:31:17,466 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:31:17,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:31:17,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:31:17,528 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:31:17,542 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:31:17,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:31:17,609 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['83251 memcached\n'], []) 2022-02-08 03:31:17,609 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 03:31:17,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:31:17,655 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['83251 memcached\n'], []) 2022-02-08 03:31:17,657 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:31:17,660 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:31:19,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:19,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:31:19,835 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:19,836 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:31:19,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:31:19,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:31:19,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:31:19,908 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:31:19,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:31:19,959 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:31:19,973 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:31:19,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:31:20,033 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['76870 memcached\n'], []) 2022-02-08 03:31:20,033 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 03:31:20,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:31:20,082 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['76870 memcached\n'], []) 2022-02-08 03:31:20,084 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:31:20,085 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:31:20,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:21,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:31:21,608 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:21,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:31:21,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:31:21,650 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:31:21,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:31:21,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:31:21,709 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:31:21,739 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:31:21,753 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:31:21,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:31:21,805 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['10729 memcached\n'], []) 2022-02-08 03:31:21,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 03:31:21,834 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:31:21,855 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['10729 memcached\n'], []) 2022-02-08 03:31:21,857 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:31:21,858 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:31:22,507 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:22,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:31:22,930 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:22,931 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:31:22,948 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:31:22,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:31:22,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:31:23,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:31:23,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:31:23,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:31:23,051 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:31:23,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:31:23,114 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['90619 memcached\n'], []) 2022-02-08 03:31:23,114 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 03:31:23,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:31:23,164 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['90619 memcached\n'], []) 2022-02-08 03:31:23,164 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:31:23,167 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:31:23,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:23,474 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:31:23,536 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:23,538 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:31:23,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:31:23,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:31:23,581 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:31:23,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:31:23,614 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:31:23,627 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:31:23,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:31:23,657 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 03:31:23,686 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:31:23,701 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:26,066 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:31:26,359 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:26,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:26,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:31:26,690 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 03:31:26,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:31:26,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:26,744 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:31:26,759 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 03:31:26,759 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:31:26,760 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:31:26,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:27,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:31:27,115 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:27,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:31:27,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:31:27,145 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:31:27,160 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:31:27,178 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:31:27,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:31:27,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:31:27,216 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:31:27,230 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 03:31:27,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:31:27,272 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:30,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:31:30,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:30,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:30,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:31:30,851 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 03:31:30,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:31:30,867 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:30,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:31:30,934 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 03:31:30,934 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:31:30,937 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:31:31,043 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:31,242 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:31:31,299 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:31,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:31:31,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:31:31,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:31:31,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:31:31,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:31:31,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:31:31,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:31:31,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:31:31,408 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 03:31:31,433 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:31:31,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:34,474 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:31:34,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:35,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:35,181 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:31:35,197 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 03:31:35,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:31:35,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:35,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:31:35,295 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 03:31:35,296 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:31:35,298 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 03:31:35,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:35,607 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 03:31:35,667 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:35,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 03:31:35,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 03:31:35,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 03:31:35,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 03:31:35,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 03:31:35,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 03:31:35,755 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 03:31:35,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:31:35,785 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 03:31:35,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:31:35,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:35,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:31:35,859 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:35,874 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:35,887 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:31:35,900 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 03:31:35,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:31:35,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:35,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:31:35,941 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:35,959 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:31:35,977 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:35,994 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:36,015 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:36,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:36,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:36,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:36,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:36,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:36,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:36,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:31:36,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:36,178 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:36,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:36,209 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:36,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:36,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:36,257 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:36,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:36,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:36,303 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:31:36,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:36,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:36,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:36,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:36,387 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:36,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:36,418 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:36,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:36,451 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:36,469 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:31:36,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:36,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:36,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:36,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:36,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:36,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:36,589 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:36,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:36,621 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 03:31:36,622 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:31:36,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:37,102 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:31:37,224 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:37,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:31:37,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:31:37,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:31:37,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:31:37,285 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:31:37,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:31:37,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:31:37,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:31:37,358 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 03:31:37,385 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:31:37,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:38,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:31:39,206 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:39,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:39,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:31:39,480 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 03:31:39,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:31:39,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:39,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:31:39,578 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 03:31:39,578 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:31:39,581 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 03:31:39,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:39,895 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 03:31:39,963 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:39,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 03:31:39,979 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 03:31:39,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 03:31:40,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 03:31:40,028 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 03:31:40,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 03:31:40,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 03:31:40,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:31:40,088 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 03:31:40,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:31:40,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:40,155 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:31:40,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:40,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:40,201 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:31:40,217 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 03:31:40,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:31:40,232 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:40,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:31:40,260 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:40,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:31:40,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:40,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:40,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:40,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:40,395 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:40,415 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:40,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:40,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:40,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:40,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:31:40,520 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:40,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:40,553 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:40,571 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:40,586 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:40,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:40,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:40,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:40,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:40,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:31:40,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:40,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:40,721 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:40,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:40,752 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:40,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:40,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:40,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:40,816 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:40,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:31:40,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:40,865 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:40,880 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:40,895 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:40,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:40,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:40,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:40,961 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:40,980 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 03:31:40,982 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:31:41,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:41,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:31:41,321 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:41,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:31:41,334 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:31:41,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:31:41,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:31:41,374 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:31:41,387 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:31:41,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:31:41,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:31:41,430 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 03:31:41,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:31:41,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:44,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:31:44,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:44,576 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:44,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:31:44,769 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 03:31:44,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:31:44,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:44,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:31:44,838 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 03:31:44,838 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:31:44,841 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:31:44,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:45,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:31:45,312 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:45,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:31:45,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:31:45,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:31:45,357 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:31:45,387 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:31:45,403 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:31:45,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:31:45,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:31:45,474 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 03:31:45,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:31:45,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:48,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:31:48,443 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:48,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:48,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:31:48,796 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 03:31:48,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:31:48,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:48,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:31:48,877 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 03:31:48,877 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:31:48,880 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 03:31:48,954 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:49,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 5 queries submitted, 0 failed, 5028 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:31:49,276 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:49,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 03:31:49,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 03:31:49,311 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 03:31:49,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 03:31:49,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 03:31:49,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 03:31:49,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 03:31:49,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:31:49,413 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 03:31:49,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:31:49,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:49,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:31:49,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:49,502 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:49,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:31:49,532 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 03:31:49,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:31:49,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:49,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:31:49,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:49,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:31:49,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:49,635 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:49,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:49,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:49,696 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:49,714 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:49,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:49,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:49,775 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:49,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:31:49,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:49,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:49,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:49,864 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:49,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:49,905 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:49,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:49,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:49,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:49,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:31:49,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:50,015 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:50,033 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:50,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:50,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:50,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:50,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:50,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:50,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:50,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:31:50,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:50,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:50,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:50,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:50,255 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:50,275 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:50,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:50,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:50,335 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 03:31:50,336 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 03:31:50,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:50,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 03:31:50,724 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:50,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 03:31:50,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 03:31:50,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 03:31:50,770 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 03:31:50,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 03:31:50,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 03:31:50,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 03:31:50,829 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:31:50,846 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 03:31:50,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:31:50,890 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:50,905 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:31:50,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:50,936 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:50,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:31:50,961 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 03:31:50,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:31:50,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:50,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:31:51,005 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:51,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:31:51,043 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:51,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:51,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:51,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:51,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:51,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:51,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:51,183 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:31:51,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:51,219 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:31:51,239 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:51,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:51,272 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:51,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:51,303 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:51,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:51,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:51,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:31:51,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:51,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:31:51,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:51,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:51,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:51,467 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:51,482 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:51,497 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:51,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:51,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:31:51,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:51,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:31:51,581 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:51,596 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:51,619 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:51,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:51,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:51,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:51,691 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:51,709 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:31:51,726 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 03:31:51,727 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:31:51,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:52,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:31:52,915 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:52,915 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:31:52,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:31:53,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:31:53,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:31:53,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:31:53,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:31:53,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:31:53,568 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:31:53,585 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 03:31:53,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:31:53,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:56,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:31:56,747 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:56,994 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:31:57,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:31:57,295 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 03:31:57,295 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:31:57,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:31:57,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:31:57,385 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 03:31:57,385 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:31:57,388 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:31:58,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:31:58,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:31:58,628 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:31:58,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:31:58,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:31:58,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:31:58,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:31:58,836 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:31:58,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:31:58,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:31:58,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:31:58,924 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 03:31:58,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:31:58,961 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:32:00,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:32:01,147 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:32:01,290 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:32:01,438 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:32:01,454 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 03:32:01,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:32:01,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:32:01,505 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:32:01,522 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 03:32:01,522 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:32:01,523 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:32:01,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:32:01,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:32:01,941 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:32:01,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:32:01,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:32:01,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:32:01,997 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:32:02,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:32:02,038 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:32:02,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:32:02,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:32:02,099 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 03:32:02,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:32:02,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:32:05,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:32:05,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:32:05,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:32:05,971 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:32:05,990 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 03:32:05,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:32:06,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:32:06,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:32:06,073 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 03:32:06,075 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:32:06,076 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 03:32:06,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:32:06,448 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 03:32:06,516 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:32:06,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 03:32:06,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 03:32:06,548 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 03:32:06,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 03:32:06,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 03:32:06,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 03:32:06,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 03:32:06,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:32:06,644 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 03:32:06,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:32:06,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:32:06,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:32:06,719 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:32:06,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:32:06,747 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:32:06,763 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 03:32:06,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:32:06,778 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:32:06,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:32:06,816 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:32:06,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:32:06,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:32:06,880 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:32:06,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:32:06,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:32:06,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:32:06,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:32:06,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:32:07,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:32:07,020 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:32:07,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:32:07,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:32:07,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:32:07,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:32:07,109 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:32:07,127 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:32:07,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:32:07,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:32:07,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:32:07,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:32:07,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:32:07,229 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:32:07,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:32:07,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:32:07,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:32:07,290 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:32:07,306 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:32:07,323 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:32:07,339 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:32:07,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:32:07,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:32:07,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:32:07,420 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:32:07,438 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:32:07,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:32:07,474 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:32:07,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:32:07,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:32:07,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:32:07,543 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 03:32:07,543 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:32:07,546 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:32:07,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:32:07,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:32:07,921 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:32:07,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:32:07,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:32:07,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:32:07,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:32:07,982 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:32:07,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:32:08,017 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:32:08,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:08,559 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:32:08,561 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.01700019836. 2022-02-08 03:32:08,561 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:32:08,562 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:32:08,647 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:32:08,918 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:32:09,063 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:32:09,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:32:09,081 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:32:09,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:32:09,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:32:09,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:32:09,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:32:09,316 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:32:09,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:09,973 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:32:09,973 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:32:11,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:12,194 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:32:12,194 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:32:14,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:14,407 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:32:14,407 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:32:16,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:16,598 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:32:16,598 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:32:18,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:18,760 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:32:18,762 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:32:20,762 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:20,957 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:32:20,957 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:32:22,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:23,141 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:32:23,141 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:32:25,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:25,319 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:32:25,319 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:32:27,326 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:27,592 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:32:27,592 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:32:29,594 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:29,812 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:32:29,812 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:32:31,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:32,095 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:32:32,095 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:32:34,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:34,276 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:32:34,278 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:32:36,278 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:36,502 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:32:36,502 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:32:38,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:38,714 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:32:38,714 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:32:40,716 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:40,900 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:32:40,901 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 32.3399999142. 2022-02-08 03:32:40,901 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:32:40,903 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:32:41,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:32:42,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:32:42,283 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:32:42,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:32:42,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:32:42,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:32:42,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:32:42,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:32:42,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:32:42,451 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:32:42,479 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:43,101 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:32:43,101 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.2009999752. 2022-02-08 03:32:43,101 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:32:43,104 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:32:44,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:32:44,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:32:44,719 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:32:44,719 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:32:44,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:32:44,760 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:32:44,795 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:32:44,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:32:44,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:32:44,900 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:32:44,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:45,726 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:32:45,726 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 03:32:47,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:47,907 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:32:47,907 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.80599999428. 2022-02-08 03:32:47,907 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:32:47,910 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:32:48,660 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:32:48,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:32:49,101 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:32:49,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:32:49,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:32:49,158 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:32:49,178 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:32:49,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:32:49,219 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 5 queries submitted, 0 failed, 5050 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:32:49,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:32:49,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:49,907 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:32:49,907 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.0. 2022-02-08 03:32:49,907 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:32:49,910 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:32:51,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:32:51,454 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:32:51,657 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:32:51,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:32:51,678 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:32:51,709 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:32:51,739 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:32:51,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:32:51,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:32:51,834 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:32:51,867 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:52,661 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:32:52,661 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.7539999485. 2022-02-08 03:32:52,661 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:32:52,664 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:32:53,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:32:54,030 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:32:54,440 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:32:54,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:32:54,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:32:54,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:32:54,826 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:32:54,885 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:32:54,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:32:55,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:32:55,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:56,210 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:32:56,211 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 03:32:58,211 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:32:58,816 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:32:58,816 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 03:33:00,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:33:01,348 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:33:01,349 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 03:33:03,349 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:33:03,640 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:33:03,640 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 10.978000164. 2022-02-08 03:33:03,640 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:33:03,641 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:33:04,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:33:04,413 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:33:04,513 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:33:04,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:33:04,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:33:04,561 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:33:04,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:33:04,594 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:33:04,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:33:04,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:33:04,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:33:05,285 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:33:05,285 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.64499998093. 2022-02-08 03:33:05,285 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:33:05,286 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:33:05,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:33:06,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:33:06,470 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:33:06,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:33:06,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:33:06,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:33:06,760 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:33:06,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:33:06,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:33:07,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:33:07,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:33:07,752 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:33:07,753 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:33:09,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:33:10,144 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:33:10,145 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:33:12,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:33:12,595 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:33:12,595 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:33:14,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:33:15,105 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:33:15,107 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:33:17,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:33:17,434 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:33:17,434 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 12.1489999294. ############################################################ # # RAM FootPrint: 4174.9609375 # ############################################################ 2022-02-08 03:33:17,469 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 03:33:17,470 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 115 seconds. Reason: Iteration:16 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 5067 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 5088 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:35:12,479 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 03:35:12,479 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 03:35:17,482 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:35:18,492 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:35:19,341 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:35:19,585 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:35:19,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:35:19,602 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:35:19,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:35:19,704 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:35:19,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:35:19,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:35:19,793 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:35:19,825 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:35:19,826 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:35:19,944 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['114761 memcached\n'], []) 2022-02-08 03:35:19,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 03:35:19,987 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:35:20,016 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['114761 memcached\n'], []) 2022-02-08 03:35:20,017 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:35:20,019 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:35:21,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:35:21,931 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:35:22,145 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:35:22,145 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:35:22,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:35:22,188 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:35:22,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:35:22,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:35:22,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:35:22,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:35:22,355 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:35:22,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:35:22,420 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['24564 memcached\n'], []) 2022-02-08 03:35:22,420 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 03:35:22,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:35:22,480 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['24564 memcached\n'], []) 2022-02-08 03:35:22,480 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:35:22,483 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:35:23,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:35:24,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:35:24,948 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:35:24,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:35:25,005 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:35:25,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:35:25,094 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:35:25,119 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:35:25,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:35:25,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:35:25,230 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:35:25,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:35:25,316 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['13221 memcached\n'], []) 2022-02-08 03:35:25,318 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 03:35:25,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:35:25,378 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['13221 memcached\n'], []) 2022-02-08 03:35:25,378 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:35:25,381 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:35:26,647 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:35:27,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:35:27,592 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:35:27,594 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:35:27,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:35:27,637 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:35:27,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:35:27,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:35:27,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:35:27,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:35:27,835 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:35:27,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:35:27,901 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['48882 memcached\n'], []) 2022-02-08 03:35:27,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 03:35:27,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:35:27,970 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['48882 memcached\n'], []) 2022-02-08 03:35:27,970 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:35:27,971 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:35:29,262 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:35:29,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:35:30,242 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:35:30,242 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:35:30,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:35:30,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:35:30,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:35:30,377 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:35:30,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:35:30,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:35:30,444 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:35:30,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:35:30,503 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['42702 memcached\n'], []) 2022-02-08 03:35:30,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 03:35:30,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:35:30,556 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['42702 memcached\n'], []) 2022-02-08 03:35:30,556 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:35:30,559 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:35:31,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:35:32,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:35:32,555 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:35:32,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:35:32,568 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:35:32,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:35:32,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:35:32,617 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:35:32,630 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:35:32,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:35:32,720 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:35:32,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:35:32,782 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['83939 memcached\n'], []) 2022-02-08 03:35:32,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 03:35:32,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:35:32,836 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['83939 memcached\n'], []) 2022-02-08 03:35:32,836 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:35:32,838 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:35:34,017 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:35:34,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:35:34,852 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:35:34,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:35:34,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:35:34,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:35:34,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:35:34,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:35:34,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:35:34,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:35:34,990 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:35:34,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:35:35,055 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['77595 memcached\n'], []) 2022-02-08 03:35:35,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 03:35:35,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:35:35,144 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['77595 memcached\n'], []) 2022-02-08 03:35:35,145 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:35:35,148 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:35:36,048 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:35:36,632 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:35:36,805 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:35:36,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:35:36,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:35:36,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:35:36,872 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:35:36,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:35:36,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:35:36,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:35:36,976 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:35:36,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:35:37,052 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['11450 memcached\n'], []) 2022-02-08 03:35:37,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 03:35:37,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:35:37,141 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['11450 memcached\n'], []) 2022-02-08 03:35:37,141 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:35:37,142 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:35:37,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:35:38,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:35:38,375 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:35:38,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:35:38,397 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:35:38,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:35:38,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:35:38,469 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:35:38,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:35:38,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:35:38,536 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:35:38,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:35:38,611 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['91365 memcached\n'], []) 2022-02-08 03:35:38,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 03:35:38,642 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:35:38,670 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['91365 memcached\n'], []) 2022-02-08 03:35:38,670 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:35:38,671 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:35:38,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:35:38,979 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:35:39,049 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:35:39,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:35:39,063 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:35:39,076 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:35:39,089 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:35:39,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:35:39,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:35:39,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:35:39,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:35:39,161 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 03:35:39,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:35:39,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:35:42,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:35:42,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:35:42,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:35:43,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:35:43,095 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 03:35:43,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:35:43,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:35:43,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:35:43,183 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 03:35:43,183 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:35:43,184 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:35:43,275 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:35:43,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:35:43,552 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:35:43,553 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:35:43,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:35:43,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:35:43,591 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:35:43,607 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:35:43,619 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:35:43,632 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:35:43,645 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:35:43,661 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 03:35:43,691 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:35:43,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:35:46,571 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:35:46,900 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:35:47,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:35:47,262 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:35:47,276 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 03:35:47,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:35:47,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:35:47,341 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:35:47,355 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 03:35:47,357 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:35:47,358 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:35:47,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:35:47,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:35:47,716 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:35:47,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:35:47,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:35:47,744 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:35:47,757 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:35:47,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:35:47,788 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:35:47,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:35:47,816 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:35:47,835 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 03:35:47,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:35:47,880 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 5113 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:35:50,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:35:51,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:35:51,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:35:51,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:35:51,559 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 03:35:51,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:35:51,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:35:51,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:35:51,634 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 03:35:51,634 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:35:51,637 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 03:35:51,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:35:51,971 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 03:35:52,038 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:35:52,038 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 03:35:52,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 03:35:52,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 03:35:52,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 03:35:52,102 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 03:35:52,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 03:35:52,130 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 03:35:52,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:35:52,160 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 03:35:52,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:35:52,198 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:35:52,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:35:52,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:35:52,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:35:52,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:35:52,272 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 03:35:52,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:35:52,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:35:52,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:35:52,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:52,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:35:52,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:52,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:52,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:52,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:52,443 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:52,461 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:52,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:52,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:52,522 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:52,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:35:52,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:52,573 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:52,589 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:52,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:52,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:52,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:52,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:52,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:52,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:52,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:35:52,719 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:52,734 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:52,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:52,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:52,780 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:52,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:52,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:52,841 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:52,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:52,874 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:35:52,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:52,910 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:52,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:52,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:52,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:52,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:52,997 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:53,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:53,032 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 03:35:53,035 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:35:53,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:35:53,420 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:35:53,551 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:35:53,551 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:35:53,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:35:53,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:35:53,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:35:53,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:35:53,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:35:53,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:35:53,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:35:53,743 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 03:35:53,767 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:35:53,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:35:55,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:35:55,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:35:56,028 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:35:56,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:35:56,232 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 03:35:56,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:35:56,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:35:56,395 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:35:56,433 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 03:35:56,433 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:35:56,436 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 03:35:56,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:35:56,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 03:35:56,832 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:35:56,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 03:35:56,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 03:35:56,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 03:35:56,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 03:35:56,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 03:35:56,908 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 03:35:56,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 03:35:56,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:35:56,953 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 03:35:56,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:35:57,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:35:57,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:35:57,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:35:57,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:35:57,068 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:35:57,084 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 03:35:57,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:35:57,099 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:35:57,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:35:57,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:57,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:35:57,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:57,191 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:57,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:57,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:57,255 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:57,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:57,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:57,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:35:57,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:57,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:35:57,374 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:57,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:57,408 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:57,426 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:57,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:57,459 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:57,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:57,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:35:57,512 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:57,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:35:57,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:57,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:57,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:57,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:57,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:57,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:57,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:57,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:35:57,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:57,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:35:57,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:57,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:57,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:57,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:57,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:57,790 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:57,806 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:57,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:35:57,839 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 03:35:57,842 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:35:57,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:35:58,147 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:35:58,210 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:35:58,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:35:58,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:35:58,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:35:58,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:35:58,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:35:58,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:35:58,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:35:58,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:35:58,325 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 03:35:58,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:35:58,369 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:36:01,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:36:01,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:36:01,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:36:02,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:36:02,305 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 03:36:02,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:36:02,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:36:02,362 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:36:02,381 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 03:36:02,381 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:36:02,392 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:36:02,471 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:36:02,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:36:02,865 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:36:02,865 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:36:02,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:36:02,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:36:02,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:36:02,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:36:03,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:36:03,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:36:03,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:36:03,194 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 03:36:03,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:36:03,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:36:05,997 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:36:06,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:36:06,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:36:06,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:36:06,868 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 03:36:06,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:36:06,887 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:36:06,948 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:36:06,967 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 03:36:06,967 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:36:06,969 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 03:36:07,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:36:07,311 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 03:36:07,385 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:36:07,385 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 03:36:07,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 03:36:07,418 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 03:36:07,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 03:36:07,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 03:36:07,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 03:36:07,492 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 03:36:07,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:36:07,526 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 03:36:07,553 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:36:07,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:36:07,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:36:07,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:36:07,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:36:07,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:36:07,667 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 03:36:07,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:36:07,686 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:36:07,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:36:07,719 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:07,742 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:36:07,767 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:07,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:07,822 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:07,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:07,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:07,897 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:07,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:07,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:07,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:07,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:36:08,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:08,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:08,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:08,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:08,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:08,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:08,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:08,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:08,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:08,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:36:08,178 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:08,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:08,211 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:08,229 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:08,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:08,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:08,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:08,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:08,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:08,341 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:36:08,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:08,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:08,400 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:08,420 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:08,438 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:08,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:08,477 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:08,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:08,513 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 03:36:08,515 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 03:36:08,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:36:08,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 03:36:08,894 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:36:08,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 03:36:08,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 03:36:08,923 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 03:36:08,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 03:36:08,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 03:36:08,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 03:36:08,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 03:36:09,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:36:09,019 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 03:36:09,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:36:09,066 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:36:09,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:36:09,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:36:09,112 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:36:09,130 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:36:09,161 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 03:36:09,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:36:09,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:36:09,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:36:09,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:09,232 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:36:09,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:09,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:09,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:09,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:09,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:09,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:09,387 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:09,408 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:09,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:09,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:36:09,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:09,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:09,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:09,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:09,520 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:09,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:09,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:09,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:09,584 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:09,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:36:09,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:09,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:09,647 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:09,663 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:09,678 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:09,696 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:09,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:09,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:09,742 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:09,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:36:09,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:09,793 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:09,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:09,829 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:09,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:09,865 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:09,882 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:09,903 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:09,920 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 03:36:09,921 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:36:10,033 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:36:10,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:36:10,358 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:36:10,359 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:36:10,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:36:10,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:36:10,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:36:10,436 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:36:10,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:36:10,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:36:10,479 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:36:10,493 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 03:36:10,520 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:36:10,538 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:36:13,178 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:36:13,489 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:36:13,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:36:13,892 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:36:13,928 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 03:36:13,928 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:36:13,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:36:13,992 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:36:14,006 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 03:36:14,007 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:36:14,009 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:36:14,752 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:36:15,112 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:36:15,253 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:36:15,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:36:15,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:36:15,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:36:15,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:36:15,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:36:15,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:36:15,382 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:36:15,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:36:15,418 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 03:36:15,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:36:15,469 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:36:17,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:36:17,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:36:17,795 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:36:17,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:36:17,957 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 03:36:17,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:36:17,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:36:18,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:36:18,023 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 03:36:18,023 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:36:18,026 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:36:18,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:36:18,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:36:18,678 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:36:18,678 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:36:18,696 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:36:18,709 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:36:18,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:36:18,739 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:36:18,752 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:36:18,765 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:36:18,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:36:18,795 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 03:36:18,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:36:18,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:36:21,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:36:22,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:36:22,505 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:36:22,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:36:22,726 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 03:36:22,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:36:22,744 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:36:22,785 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:36:22,802 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 03:36:22,802 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:36:22,812 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 03:36:22,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:36:23,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 03:36:23,213 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:36:23,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 03:36:23,229 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 03:36:23,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 03:36:23,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 03:36:23,280 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 03:36:23,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 03:36:23,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 03:36:23,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:36:23,342 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 03:36:23,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:36:23,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:36:23,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:36:23,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:36:23,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:36:23,451 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:36:23,464 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 03:36:23,466 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:36:23,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:36:23,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:36:23,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:23,535 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:36:23,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:23,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:23,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:23,630 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:23,653 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:23,678 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:23,701 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:23,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:36:23,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:23,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:36:23,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:23,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:23,821 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:23,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:23,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:23,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:23,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:23,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:36:23,926 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:23,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:36:23,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:23,977 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:23,994 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:24,015 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:24,035 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:24,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:24,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:24,086 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:36:24,104 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:24,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:36:24,142 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:24,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:24,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:24,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:24,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:24,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:24,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:24,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:36:24,290 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 03:36:24,290 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:36:24,292 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:36:24,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:36:24,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:36:24,684 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:36:24,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:36:24,698 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:36:24,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:36:24,726 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:36:24,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:36:24,757 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:36:24,772 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:36:24,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:36:25,407 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:36:25,407 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.11600017548. 2022-02-08 03:36:25,407 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:36:25,410 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:36:25,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:36:25,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:36:25,888 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:36:25,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:36:25,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:36:25,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:36:25,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:36:25,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:36:26,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:36:26,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:36:26,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:36:26,951 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:36:26,953 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:36:28,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:36:29,207 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:36:29,207 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:36:31,209 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:36:31,423 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:36:31,423 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:36:33,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:36:33,599 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:36:33,599 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:36:35,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:36:35,809 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:36:35,811 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:36:37,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:36:38,194 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:36:38,194 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:36:40,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:36:40,453 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:36:40,453 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:36:42,454 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:36:42,716 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:36:42,716 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:36:44,716 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:36:44,888 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:36:44,890 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:36:46,890 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:36:47,266 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:36:47,266 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 5138 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:36:49,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:36:49,621 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:36:49,621 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:36:51,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:36:51,812 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:36:51,813 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:36:53,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:36:54,061 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:36:54,061 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:36:56,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:36:56,259 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:36:56,259 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:36:58,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:36:58,469 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:36:58,469 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:37:00,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:37:00,673 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:37:00,691 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 35.2839999199. 2022-02-08 03:37:00,691 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:37:00,723 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:37:01,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:37:01,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:37:01,818 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:37:01,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:37:01,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:37:01,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:37:01,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:37:01,961 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:37:01,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:37:02,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:37:02,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:37:02,846 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:37:02,846 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 03:37:04,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:37:05,069 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:37:05,069 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.37800002098. 2022-02-08 03:37:05,069 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:37:05,072 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:37:05,982 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:37:06,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:37:06,480 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:37:06,482 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:37:06,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:37:06,535 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:37:06,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:37:06,584 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:37:06,622 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:37:06,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:37:06,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:37:07,591 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:37:07,591 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.52100014687. 2022-02-08 03:37:07,591 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:37:07,592 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:37:08,272 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:37:08,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:37:08,763 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:37:08,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:37:08,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:37:08,834 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:37:08,857 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:37:08,903 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:37:08,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:37:08,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:37:08,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:37:09,634 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:37:09,634 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.04299998283. 2022-02-08 03:37:09,634 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:37:09,635 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:37:10,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:37:10,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:37:11,029 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:37:11,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:37:11,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:37:11,076 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:37:11,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:37:11,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:37:11,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:37:11,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:37:11,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:37:11,957 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:37:11,959 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.32500004768. 2022-02-08 03:37:11,959 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:37:11,960 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:37:12,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:37:12,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:37:12,953 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:37:12,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:37:12,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:37:13,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:37:13,020 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:37:13,066 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:37:13,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:37:13,124 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:37:13,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:37:13,813 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:37:13,813 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.85499978065. 2022-02-08 03:37:13,813 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:37:13,815 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:37:14,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:37:14,714 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:37:14,805 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:37:14,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:37:14,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:37:14,844 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:37:14,857 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:37:14,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:37:14,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:37:14,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:37:14,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:37:15,496 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:37:15,496 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.68300008774. 2022-02-08 03:37:15,497 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:37:15,499 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:37:15,806 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:37:16,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:37:16,769 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:37:16,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:37:16,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:37:17,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:37:17,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:37:17,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:37:17,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:37:17,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:37:17,627 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:37:17,895 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:37:17,897 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:37:19,897 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:37:20,380 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:37:20,381 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:37:22,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:37:22,825 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:37:22,825 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:37:24,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:37:25,187 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:37:25,187 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:37:27,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:37:28,056 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:37:28,056 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:37:30,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:37:30,338 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:37:30,338 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:37:32,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:37:32,663 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:37:32,663 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:37:34,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:37:34,934 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:37:34,934 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:37:36,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:37:37,476 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:37:37,476 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 21.9790000916. ############################################################ # # RAM FootPrint: 4168.26953125 # ############################################################ 2022-02-08 03:37:37,753 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 03:37:37,753 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 72 seconds. Reason: Iteration:17 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 5162 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 5185 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:38:49,759 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 03:38:49,760 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 03:38:54,765 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:38:55,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:38:56,237 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:38:56,617 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:38:56,617 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:38:56,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:38:56,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:38:56,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:38:56,795 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:38:56,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:38:56,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:38:56,914 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:38:56,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:38:56,990 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['115544 memcached\n'], []) 2022-02-08 03:38:56,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 03:38:57,020 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:38:57,043 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['115544 memcached\n'], []) 2022-02-08 03:38:57,043 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:38:57,046 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:38:57,979 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:38:58,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:38:58,578 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:38:58,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:38:58,599 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:38:58,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:38:58,647 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:38:58,678 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:38:58,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:38:58,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:38:58,757 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:38:58,757 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:38:58,818 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['25469 memcached\n'], []) 2022-02-08 03:38:58,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 03:38:58,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:38:58,878 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['25469 memcached\n'], []) 2022-02-08 03:38:58,878 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:38:58,881 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:38:59,808 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:00,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:39:00,627 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:00,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:39:00,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:39:00,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:39:00,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:39:00,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:39:00,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:39:00,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:39:00,829 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:39:00,829 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:39:00,915 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['14051 memcached\n'], []) 2022-02-08 03:39:00,915 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 03:39:00,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:39:00,987 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['14051 memcached\n'], []) 2022-02-08 03:39:00,987 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:39:00,992 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:39:01,816 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:02,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:39:02,388 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:02,390 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:39:02,408 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:39:02,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:39:02,512 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:39:02,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:39:02,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:39:02,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:39:02,611 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:39:02,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:39:02,674 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['49628 memcached\n'], []) 2022-02-08 03:39:02,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 03:39:02,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:39:02,733 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['49628 memcached\n'], []) 2022-02-08 03:39:02,733 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:39:02,736 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:39:03,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:04,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:39:04,664 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:04,665 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:39:04,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:39:04,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:39:04,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:39:04,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:39:04,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:39:04,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:39:04,799 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:39:04,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:39:04,867 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['43440 memcached\n'], []) 2022-02-08 03:39:04,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 03:39:04,908 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:39:04,931 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['43440 memcached\n'], []) 2022-02-08 03:39:04,933 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:39:04,934 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:39:05,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:06,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:39:06,539 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:06,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:39:06,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:39:06,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:39:06,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:39:06,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:39:06,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:39:06,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:39:06,660 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:39:06,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:39:06,736 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['84708 memcached\n'], []) 2022-02-08 03:39:06,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 03:39:06,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:39:06,785 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['84708 memcached\n'], []) 2022-02-08 03:39:06,786 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:39:06,910 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:39:08,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:08,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:39:08,835 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:08,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:39:08,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:39:08,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:39:08,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:39:08,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:39:08,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:39:09,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:39:09,019 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:39:09,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:39:09,078 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['78369 memcached\n'], []) 2022-02-08 03:39:09,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 03:39:09,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:39:09,130 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['78369 memcached\n'], []) 2022-02-08 03:39:09,130 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:39:09,150 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:39:10,186 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:10,829 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:39:11,094 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:11,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:39:11,112 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:39:11,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:39:11,176 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:39:11,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:39:11,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:39:11,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:39:11,266 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:39:11,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:39:11,322 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['12190 memcached\n'], []) 2022-02-08 03:39:11,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 03:39:11,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:39:11,371 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['12190 memcached\n'], []) 2022-02-08 03:39:11,371 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:39:11,423 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:39:12,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:12,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:39:12,924 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:12,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:39:12,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:39:13,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:39:13,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:39:13,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:39:13,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:39:13,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:39:13,102 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:39:13,104 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:39:13,411 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['92151 memcached\n'], []) 2022-02-08 03:39:13,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 03:39:13,446 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:39:13,467 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['92151 memcached\n'], []) 2022-02-08 03:39:13,467 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:39:13,469 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:39:13,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:13,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:39:13,836 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:13,836 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:39:13,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:39:13,867 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:39:13,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:39:13,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:39:13,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:39:13,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:39:13,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:39:13,954 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 03:39:14,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:39:14,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:16,617 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:39:16,910 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:17,076 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:17,232 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:39:17,246 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 03:39:17,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:39:17,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:17,306 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:39:17,321 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 03:39:17,321 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:39:17,323 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:39:17,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:17,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:39:17,704 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:17,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:39:17,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:39:17,732 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:39:17,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:39:17,762 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:39:17,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:39:17,790 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:39:17,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:39:17,821 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 03:39:17,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:39:17,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:20,799 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:39:21,119 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:21,326 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:21,499 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:39:21,515 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 03:39:21,515 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:39:21,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:21,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:39:21,614 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 03:39:21,614 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:39:21,615 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:39:21,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:21,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:39:21,951 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:21,951 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:39:21,966 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:39:21,977 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:39:21,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:39:22,007 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:39:22,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:39:22,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:39:22,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:39:22,063 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 03:39:22,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:39:22,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:25,147 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:39:25,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:25,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:25,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:39:25,877 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 03:39:25,877 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:39:25,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:25,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:39:25,967 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 03:39:25,967 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:39:25,970 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 03:39:26,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:26,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 03:39:26,326 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:26,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 03:39:26,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 03:39:26,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 03:39:26,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 03:39:26,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 03:39:26,403 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 03:39:26,418 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 03:39:26,436 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:39:26,466 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 03:39:26,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:39:26,515 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:26,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:39:26,561 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:26,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:26,604 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:39:26,621 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 03:39:26,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:39:26,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:26,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:39:26,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:26,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:39:26,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:26,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:26,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:26,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:26,816 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:26,839 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:26,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:26,887 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:26,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:26,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:39:26,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:26,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:26,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:26,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:27,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:27,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:27,035 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:27,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:27,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:27,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:39:27,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:27,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:27,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:27,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:27,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:27,181 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:27,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:27,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:27,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:27,247 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:39:27,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:27,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:27,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:27,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:27,339 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:27,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:27,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:27,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:27,415 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 03:39:27,417 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:39:27,538 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:27,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:39:27,880 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:27,880 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:39:27,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:39:27,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:39:27,936 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:39:27,961 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:39:27,992 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:39:28,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:39:28,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:39:28,161 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 03:39:28,186 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:39:28,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:29,836 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:39:30,252 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:30,492 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:30,673 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:39:30,690 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 03:39:30,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:39:30,704 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:30,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:39:30,775 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 03:39:30,775 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:39:30,776 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 03:39:30,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:31,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 03:39:31,144 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:31,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 03:39:31,158 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 03:39:31,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 03:39:31,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 03:39:31,201 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 03:39:31,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 03:39:31,234 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 03:39:31,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:39:31,276 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 03:39:31,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:39:31,316 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:31,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:39:31,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:31,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:31,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:39:31,388 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 03:39:31,390 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:39:31,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:31,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:39:31,431 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:31,466 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:39:31,492 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:31,512 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:31,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:31,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:31,579 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:31,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:31,622 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:31,647 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:31,665 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:31,681 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:39:31,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:31,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:31,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:31,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:31,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:31,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:31,793 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:31,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:31,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:31,844 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:39:31,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:31,880 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:31,897 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:31,913 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:31,928 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:31,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:31,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:31,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:31,992 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:32,010 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:39:32,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:32,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:32,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:32,079 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:32,096 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:32,114 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:32,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:32,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:32,164 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 03:39:32,167 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:39:32,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:32,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:39:32,519 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:32,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:39:32,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:39:32,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:39:32,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:39:32,576 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:39:32,589 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:39:32,602 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:39:32,617 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:39:32,631 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 03:39:32,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:39:32,676 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:35,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:39:35,775 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:35,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:36,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:39:36,203 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 03:39:36,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:39:36,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:36,260 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:39:36,279 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 03:39:36,279 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:39:36,282 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:39:36,377 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:36,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:39:36,736 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:36,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:39:36,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:39:36,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:39:36,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:39:36,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:39:36,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:39:37,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:39:37,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:39:37,408 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 03:39:37,433 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:39:37,482 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:40,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:39:40,734 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:40,920 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:41,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:39:41,121 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 03:39:41,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:39:41,183 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:41,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:39:41,315 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 03:39:41,315 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:39:41,318 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 03:39:41,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:41,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 03:39:41,723 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:41,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 03:39:41,742 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 03:39:41,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 03:39:41,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 03:39:41,793 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 03:39:41,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 03:39:41,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 03:39:41,841 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:39:41,858 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 03:39:41,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:39:41,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:41,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:39:41,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:41,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:41,969 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:39:41,983 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 03:39:41,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:39:42,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:42,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:39:42,033 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:42,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:39:42,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:42,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:42,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:42,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:42,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:42,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:42,236 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:42,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:42,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:42,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:39:42,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:42,344 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:42,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:42,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:42,397 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:42,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:42,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:42,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:42,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:42,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:39:42,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:42,530 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:42,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:42,568 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:42,586 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:42,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:42,630 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:42,650 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:42,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:42,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:39:42,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:42,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:42,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:42,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:42,795 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:42,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:42,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:42,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:42,875 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 03:39:42,877 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 03:39:42,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:43,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 03:39:43,270 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:43,272 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 03:39:43,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 03:39:43,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 03:39:43,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 03:39:43,334 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 03:39:43,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 03:39:43,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 03:39:43,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:39:43,391 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 03:39:43,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:39:43,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:43,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:39:43,471 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:43,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:43,502 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:39:43,516 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 03:39:43,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:39:43,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:43,548 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:39:43,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:43,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:39:43,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:43,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:43,663 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:43,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:43,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:43,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:43,760 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:43,785 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:43,806 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:43,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:39:43,841 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:43,857 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:43,872 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:43,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:43,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:43,920 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:43,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:43,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:43,969 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:43,984 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:39:44,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:44,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:44,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:44,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:44,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:44,079 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:44,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:44,112 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:44,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:44,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:39:44,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:44,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:44,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:44,236 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:44,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:44,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:44,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:44,308 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:44,325 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 03:39:44,326 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:39:44,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:44,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:39:45,318 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:45,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:39:45,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:39:45,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:39:45,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:39:45,423 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:39:45,438 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:39:45,459 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:39:45,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:39:45,601 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 03:39:45,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:39:45,647 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:48,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:39:48,795 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:49,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 5211 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:39:49,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:39:49,295 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 03:39:49,295 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:39:49,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:49,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:39:49,368 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 03:39:49,369 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:39:49,371 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:39:50,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:50,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:39:50,542 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:50,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:39:50,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:39:50,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:39:50,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:39:50,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:39:50,655 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:39:50,686 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:39:50,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:39:50,720 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 03:39:50,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:39:50,767 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:52,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:39:53,193 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:53,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:53,484 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:39:53,499 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 03:39:53,499 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:39:53,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:53,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:39:53,585 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 03:39:53,585 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:39:53,588 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:39:53,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:53,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:39:54,020 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:54,020 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:39:54,040 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:39:54,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:39:54,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:39:54,102 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:39:54,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:39:54,135 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:39:54,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:39:54,173 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 03:39:54,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:39:54,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:57,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:39:57,716 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:57,936 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:58,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:39:58,170 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 03:39:58,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:39:58,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:58,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:39:58,243 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 03:39:58,243 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:39:58,246 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 03:39:58,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:39:58,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 03:39:58,654 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:39:58,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 03:39:58,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 03:39:58,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 03:39:58,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 03:39:58,716 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 03:39:58,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 03:39:58,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 03:39:58,762 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:39:58,776 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 03:39:58,808 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:39:58,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:58,841 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:39:58,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:58,869 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:39:58,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:39:58,898 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 03:39:58,900 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:39:58,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:39:58,933 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:39:58,946 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:58,969 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:39:58,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:59,020 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:59,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:59,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:59,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:59,122 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:59,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:59,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:39:59,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:59,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:39:59,232 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:59,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:59,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:59,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:59,298 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:59,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:59,331 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:59,346 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:39:59,364 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:59,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:39:59,397 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:59,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:59,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:59,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:59,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:59,479 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:59,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:59,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:39:59,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:59,545 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:39:59,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:59,579 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:59,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:59,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:59,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:59,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:59,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:59,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:39:59,704 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 03:39:59,704 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:39:59,707 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:39:59,788 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:40:00,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:40:00,104 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:40:00,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:40:00,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:40:00,132 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:40:00,147 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:40:00,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:40:00,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:40:00,191 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:40:00,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:00,812 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:40:00,813 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.10899996758. 2022-02-08 03:40:00,813 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:40:00,815 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:40:00,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:40:01,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:40:01,391 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:40:01,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:40:01,413 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:40:01,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:40:01,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:40:01,586 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:40:01,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:40:01,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:40:01,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:03,108 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:40:03,108 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:40:05,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:05,318 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:40:05,318 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:40:07,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:07,565 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:40:07,565 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:40:09,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:09,811 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:40:09,811 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:40:11,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:11,986 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:40:11,986 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:40:13,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:14,184 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:40:14,184 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:40:16,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:16,417 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:40:16,417 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:40:18,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:18,632 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:40:18,634 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:40:20,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:20,812 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:40:20,812 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:40:22,812 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:23,003 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:40:23,003 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:40:25,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:25,263 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:40:25,263 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:40:27,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:27,440 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:40:27,440 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 26.6270000935. 2022-02-08 03:40:27,441 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:40:27,444 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:40:27,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:40:27,948 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:40:28,062 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:40:28,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:40:28,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:40:28,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:40:28,160 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:40:28,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:40:28,216 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:40:28,244 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:40:28,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:29,035 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:40:29,036 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.59400010109. 2022-02-08 03:40:29,036 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:40:29,038 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:40:29,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:40:29,895 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:40:30,089 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:40:30,091 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:40:30,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:40:30,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:40:30,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:40:30,188 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:40:30,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:40:30,237 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:40:30,260 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:31,217 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:40:31,217 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 03:40:33,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:33,506 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:40:33,506 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 03:40:35,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:35,779 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:40:35,780 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.74499988556. 2022-02-08 03:40:35,780 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:40:35,782 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:40:36,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:40:36,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:40:36,865 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:40:36,867 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:40:36,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:40:36,913 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:40:36,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:40:36,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:40:36,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:40:37,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:40:37,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:37,720 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:40:37,720 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.94000005722. 2022-02-08 03:40:37,720 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:40:37,723 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:40:38,617 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:40:39,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:40:39,454 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:40:39,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:40:39,479 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:40:39,586 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:40:39,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:40:39,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:40:39,829 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:40:39,885 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:40:39,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:41,105 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:40:41,105 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 03:40:43,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:43,349 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:40:43,351 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 03:40:45,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:45,584 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:40:45,585 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.86400008202. 2022-02-08 03:40:45,585 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:40:45,586 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:40:46,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:40:47,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:40:47,535 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:40:47,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:40:47,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:40:47,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:40:48,030 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:40:48,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:40:48,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:40:48,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:40:48,522 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 5235 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:40:49,798 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:40:49,798 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 03:40:51,803 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:51,973 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:40:51,973 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.38800001144. 2022-02-08 03:40:51,973 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:40:51,974 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:40:52,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:40:52,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:40:53,091 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:40:53,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:40:53,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:40:53,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:40:53,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:40:53,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:40:53,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:40:53,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:40:53,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:53,742 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:40:53,743 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.76900005341. 2022-02-08 03:40:53,743 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:40:53,744 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:40:54,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:40:55,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:40:55,632 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:40:55,632 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:40:55,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:40:55,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:40:55,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:40:56,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:40:56,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:40:56,451 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:40:56,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:40:57,884 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:40:57,884 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:40:59,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:41:00,117 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:41:00,118 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:41:02,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:41:02,431 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:41:02,431 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 8.68899989128. ############################################################ # # RAM FootPrint: 4166.58203125 # ############################################################ 2022-02-08 03:41:02,711 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 03:41:02,711 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 87 seconds. Reason: Iteration:18 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 5260 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:42:29,716 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 03:42:29,717 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 03:42:34,719 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:42:35,466 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:42:36,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:42:36,467 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:42:36,467 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:42:36,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:42:36,525 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:42:36,543 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:42:36,617 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:42:36,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:42:36,772 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:42:36,836 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:42:36,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:42:36,970 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['116207 memcached\n'], []) 2022-02-08 03:42:36,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 03:42:37,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:42:37,118 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['116207 memcached\n'], []) 2022-02-08 03:42:37,118 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:42:37,121 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:42:38,043 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:42:38,594 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:42:38,796 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:42:38,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:42:38,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:42:38,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:42:38,885 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:42:38,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:42:38,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:42:39,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:42:39,036 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:42:39,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:42:39,108 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['26357 memcached\n'], []) 2022-02-08 03:42:39,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 03:42:39,142 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:42:39,171 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['26357 memcached\n'], []) 2022-02-08 03:42:39,171 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:42:39,174 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:42:40,028 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:42:40,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:42:41,026 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:42:41,028 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:42:41,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:42:41,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:42:41,102 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:42:41,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:42:41,142 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:42:41,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:42:41,194 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:42:41,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:42:41,308 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['14865 memcached\n'], []) 2022-02-08 03:42:41,308 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 03:42:41,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:42:41,388 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['14865 memcached\n'], []) 2022-02-08 03:42:41,388 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:42:41,391 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:42:42,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:42:42,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:42:43,046 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:42:43,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:42:43,063 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:42:43,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:42:43,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:42:43,119 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:42:43,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:42:43,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:42:43,197 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:42:43,198 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:42:43,278 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['50305 memcached\n'], []) 2022-02-08 03:42:43,278 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 03:42:43,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:42:43,375 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['50305 memcached\n'], []) 2022-02-08 03:42:43,375 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:42:43,377 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:42:44,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:42:44,640 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:42:44,851 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:42:44,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:42:44,885 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:42:44,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:42:44,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:42:45,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:42:45,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:42:45,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:42:45,071 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:42:45,071 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:42:45,138 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['44146 memcached\n'], []) 2022-02-08 03:42:45,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 03:42:45,170 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:42:45,194 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['44146 memcached\n'], []) 2022-02-08 03:42:45,194 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:42:45,197 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:42:46,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:42:46,801 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:42:47,003 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:42:47,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:42:47,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:42:47,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:42:47,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:42:47,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:42:47,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:42:47,155 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:42:47,187 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:42:47,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:42:47,246 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['85382 memcached\n'], []) 2022-02-08 03:42:47,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 03:42:47,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:42:47,293 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['85382 memcached\n'], []) 2022-02-08 03:42:47,293 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:42:47,296 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:42:48,142 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:42:48,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:42:48,959 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:42:48,959 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:42:48,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:42:48,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:42:49,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:42:49,040 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:42:49,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:42:49,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:42:49,101 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:42:49,102 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:42:49,155 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['79072 memcached\n'], []) 2022-02-08 03:42:49,155 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 03:42:49,191 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:42:49,217 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['79072 memcached\n'], []) 2022-02-08 03:42:49,217 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:42:49,220 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 5 queries submitted, 0 failed, 5283 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:42:50,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:42:50,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:42:50,950 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:42:50,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:42:50,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:42:50,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:42:51,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:42:51,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:42:51,061 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:42:51,094 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:42:51,108 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:42:51,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:42:51,157 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['12852 memcached\n'], []) 2022-02-08 03:42:51,158 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 03:42:51,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:42:51,210 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['12852 memcached\n'], []) 2022-02-08 03:42:51,210 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:42:51,211 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:42:51,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:42:51,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:42:52,153 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:42:52,153 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:42:52,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:42:52,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:42:52,206 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:42:52,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:42:52,234 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:42:52,260 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:42:52,273 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:42:52,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:42:52,342 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['92951 memcached\n'], []) 2022-02-08 03:42:52,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 03:42:52,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:42:52,388 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['92951 memcached\n'], []) 2022-02-08 03:42:52,388 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:42:52,391 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:42:52,471 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:42:52,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:42:52,782 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:42:52,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:42:52,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:42:52,816 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:42:52,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:42:52,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:42:52,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:42:52,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:42:52,895 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:42:52,911 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 03:42:52,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:42:52,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:42:55,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:42:55,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:42:56,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:42:56,191 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:42:56,209 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 03:42:56,210 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:42:56,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:42:56,265 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:42:56,279 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 03:42:56,280 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:42:56,282 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:42:56,382 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:42:56,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:42:56,654 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:42:56,655 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:42:56,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:42:56,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:42:56,693 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:42:56,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:42:56,721 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:42:56,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:42:56,747 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:42:56,762 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 03:42:56,788 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:42:56,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:00,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:43:00,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:00,553 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:00,739 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:43:00,753 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 03:43:00,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:43:00,765 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:00,943 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:43:00,956 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 03:43:00,956 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:43:00,957 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:43:01,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:43:01,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:43:01,299 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:43:01,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:43:01,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:43:01,326 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:43:01,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:43:01,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:43:01,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:43:01,390 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:43:01,408 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:43:01,423 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 03:43:01,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:43:01,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:04,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:43:04,839 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:05,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:05,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:43:05,236 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 03:43:05,236 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:43:05,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:05,315 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:43:05,328 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 03:43:05,329 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:43:05,332 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 03:43:05,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:43:05,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 03:43:05,697 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:43:05,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 03:43:05,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 03:43:05,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 03:43:05,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 03:43:05,752 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 03:43:05,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 03:43:05,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 03:43:05,790 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:43:05,805 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 03:43:05,831 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:43:05,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:05,859 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:43:05,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:05,890 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:05,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:43:05,918 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 03:43:05,918 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:43:05,933 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:05,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:43:05,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:05,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:43:06,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:06,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:06,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:06,094 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:06,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:06,147 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:06,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:06,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:06,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:06,236 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:43:06,255 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:06,272 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:06,288 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:06,303 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:06,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:06,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:06,354 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:06,369 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:06,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:06,403 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:43:06,420 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:06,436 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:06,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:06,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:06,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:06,502 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:06,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:06,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:06,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:06,563 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:43:06,581 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:06,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:06,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:06,632 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:06,650 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:06,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:06,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:06,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:06,720 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 03:43:06,723 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:43:06,869 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:43:07,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:43:07,262 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:43:07,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:43:07,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:43:07,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:43:07,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:43:07,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:43:07,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:43:07,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:43:07,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:43:07,384 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 03:43:07,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:43:07,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:09,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:43:09,423 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:09,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:09,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:43:09,775 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 03:43:09,775 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:43:09,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:09,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:43:09,855 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 03:43:09,857 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:43:09,858 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 03:43:09,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:43:10,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 03:43:10,240 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:43:10,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 03:43:10,255 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 03:43:10,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 03:43:10,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 03:43:10,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 03:43:10,318 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 03:43:10,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 03:43:10,346 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:43:10,364 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 03:43:10,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:43:10,418 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:10,436 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:43:10,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:10,467 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:10,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:43:10,499 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 03:43:10,499 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:43:10,515 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:10,530 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:43:10,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:10,573 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:43:10,604 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:10,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:10,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:10,681 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:10,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:10,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:10,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:10,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:10,808 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:10,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:43:10,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:10,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:10,880 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:10,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:10,915 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:10,933 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:10,948 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:10,966 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:10,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:11,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:43:11,017 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:11,033 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:11,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:11,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:11,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:11,099 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:11,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:11,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:11,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:11,176 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:43:11,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:11,229 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:11,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:11,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:11,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:11,303 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:11,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:11,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:11,351 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 03:43:11,354 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:43:11,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:43:11,665 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:43:11,736 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:43:11,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:43:11,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:43:11,765 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:43:11,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:43:11,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:43:11,811 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:43:11,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:43:11,839 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:43:11,855 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 03:43:11,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:43:11,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:14,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:43:14,959 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:15,170 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:15,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:43:15,371 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 03:43:15,372 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:43:15,390 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:15,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:43:15,447 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 03:43:15,447 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:43:15,450 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:43:15,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:43:15,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:43:15,894 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:43:15,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:43:15,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:43:15,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:43:15,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:43:15,961 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:43:15,979 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:43:15,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:43:16,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:43:16,036 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 03:43:16,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:43:16,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:19,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:43:19,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:19,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:19,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:43:19,736 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 03:43:19,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:43:19,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:19,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:43:19,825 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 03:43:19,825 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:43:19,828 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 03:43:19,915 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:43:20,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 03:43:20,229 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:43:20,229 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 03:43:20,244 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 03:43:20,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 03:43:20,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 03:43:20,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 03:43:20,321 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 03:43:20,336 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 03:43:20,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:43:20,374 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 03:43:20,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:43:20,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:20,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:43:20,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:20,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:20,502 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:43:20,519 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 03:43:20,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:43:20,535 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:20,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:43:20,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:20,591 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:43:20,622 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:20,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:20,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:20,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:20,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:20,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:20,785 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:20,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:20,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:20,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:43:20,882 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:20,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:20,928 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:20,948 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:20,966 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:20,984 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:21,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:21,020 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:21,039 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:21,061 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:43:21,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:21,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:21,140 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:21,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:21,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:21,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:21,227 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:21,244 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:21,262 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:21,280 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:43:21,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:21,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:21,339 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:21,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:21,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:21,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:21,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:21,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:21,464 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 03:43:21,467 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 03:43:21,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:43:21,785 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 03:43:21,848 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:43:21,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 03:43:21,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 03:43:21,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 03:43:21,890 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 03:43:21,908 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 03:43:21,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 03:43:21,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 03:43:21,951 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:43:21,967 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 03:43:22,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:43:22,017 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:22,033 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:43:22,048 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:22,061 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:22,076 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:43:22,092 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 03:43:22,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:43:22,114 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:22,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:43:22,145 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:22,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:43:22,201 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:22,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:22,255 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:22,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:22,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:22,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:22,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:22,392 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:22,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:22,436 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:43:22,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:22,492 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:22,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:22,528 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:22,543 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:22,561 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:22,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:22,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:22,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:22,630 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:43:22,647 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:22,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:22,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:22,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:22,709 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:22,724 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:22,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:22,757 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:22,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:22,790 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:43:22,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:22,826 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:22,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:22,861 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:22,880 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:22,895 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:22,913 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:22,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:22,946 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 03:43:22,948 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:43:23,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:43:23,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:43:23,905 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:43:23,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:43:23,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:43:24,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:43:24,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:43:24,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:43:24,094 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:43:24,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:43:24,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:43:24,157 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 03:43:24,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:43:24,229 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:27,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:43:27,482 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:27,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:28,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:43:28,099 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 03:43:28,099 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:43:28,119 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:28,181 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:43:28,206 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 03:43:28,207 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:43:28,209 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:43:28,966 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:43:29,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:43:29,404 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:43:29,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:43:29,420 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:43:29,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:43:29,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:43:29,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:43:29,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:43:29,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:43:29,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:43:29,556 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 03:43:29,581 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:43:29,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:31,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:43:32,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:32,288 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:32,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:43:32,470 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 03:43:32,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:43:32,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:32,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:43:32,533 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 03:43:32,533 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:43:32,536 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:43:32,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:43:32,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:43:32,930 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:43:32,931 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:43:32,948 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:43:32,964 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:43:32,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:43:33,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:43:33,020 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:43:33,038 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:43:33,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:43:33,071 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 03:43:33,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:43:33,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:36,145 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:43:36,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:36,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:36,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:43:36,967 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 03:43:36,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:43:36,997 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:37,040 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:43:37,056 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 03:43:37,058 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:43:37,059 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 03:43:37,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:43:37,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 03:43:37,434 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:43:37,436 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 03:43:37,451 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 03:43:37,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 03:43:37,477 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 03:43:37,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 03:43:37,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 03:43:37,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 03:43:37,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:43:37,558 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 03:43:37,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:43:37,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:37,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:43:37,640 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:37,654 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:43:37,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:43:37,684 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 03:43:37,686 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:43:37,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:43:37,714 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:43:37,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:37,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:43:37,788 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:37,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:37,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:37,869 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:37,900 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:37,926 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:37,964 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:37,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:43:38,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:38,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:43:38,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:38,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:38,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:38,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:38,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:38,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:38,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:38,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:43:38,183 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:38,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:43:38,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:38,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:38,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:38,265 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:38,280 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:38,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:38,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:38,331 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:43:38,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:38,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:43:38,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:38,413 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:38,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:38,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:38,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:38,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:38,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:38,520 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:43:38,539 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 03:43:38,539 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:43:38,540 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:43:38,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:43:38,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:43:38,937 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:43:38,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:43:38,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:43:38,983 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:43:39,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:43:39,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:43:39,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:43:39,079 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:43:39,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:43:39,581 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:43:39,582 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.04299998283. 2022-02-08 03:43:39,582 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:43:39,584 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:43:39,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:43:40,102 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:43:40,266 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:43:40,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:43:40,301 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:43:40,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:43:40,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:43:40,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:43:40,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:43:40,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:43:40,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:43:41,088 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:43:41,089 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:43:43,089 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:43:43,282 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:43:43,282 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:43:45,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:43:45,469 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:43:45,469 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:43:47,469 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:43:47,767 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:43:47,767 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 5308 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:43:49,767 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:43:49,983 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:43:49,983 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:43:51,984 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:43:52,260 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:43:52,262 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:43:54,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:43:54,496 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:43:54,497 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:43:56,497 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:43:56,713 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:43:56,713 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:43:58,714 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:43:58,894 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:43:58,894 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:44:00,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:01,147 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:44:01,147 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:44:03,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:03,385 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:44:03,385 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:44:05,385 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:05,578 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:44:05,578 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:44:07,578 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:07,765 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:44:07,766 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:44:09,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:09,980 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:44:09,980 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 30.3989999294. 2022-02-08 03:44:09,982 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:44:09,983 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:44:10,709 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:44:11,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:44:11,279 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:44:11,280 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:44:11,303 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:44:11,339 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:44:11,369 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:44:11,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:44:11,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:44:11,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:44:11,474 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:12,088 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:44:12,088 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 03:44:14,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:14,265 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:44:14,266 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.28399991989. 2022-02-08 03:44:14,266 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:44:14,267 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:44:15,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:44:15,382 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:44:15,516 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:44:15,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:44:15,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:44:15,571 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:44:15,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:44:15,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:44:15,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:44:15,670 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:44:15,688 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:16,487 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:44:16,493 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.22699999809. 2022-02-08 03:44:16,493 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:44:16,494 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:44:17,418 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:44:17,749 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:44:17,855 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:44:17,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:44:17,872 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:44:17,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:44:17,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:44:17,969 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:44:17,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:44:18,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:44:18,089 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:18,917 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:44:18,917 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 03:44:20,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:21,098 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:44:21,098 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.60599994659. 2022-02-08 03:44:21,098 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:44:21,101 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:44:22,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:44:22,676 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:44:23,055 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:44:23,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:44:23,196 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:44:23,339 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:44:23,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:44:23,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:44:23,772 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:44:23,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:44:23,872 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:25,098 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:44:25,099 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 03:44:27,099 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:27,309 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:44:27,311 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.21200013161. 2022-02-08 03:44:27,311 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:44:27,312 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:44:27,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:44:28,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:44:28,447 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:44:28,448 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:44:28,467 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:44:28,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:44:28,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:44:28,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:44:28,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:44:28,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:44:28,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:29,184 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:44:29,184 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.87399983406. 2022-02-08 03:44:29,186 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:44:29,187 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:44:29,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:44:30,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:44:30,147 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:44:30,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:44:30,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:44:30,184 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:44:30,198 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:44:30,216 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:44:30,229 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:44:30,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:44:30,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:30,795 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:44:30,796 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.6099998951. 2022-02-08 03:44:30,796 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:44:30,798 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:44:31,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:44:31,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:44:31,809 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:44:31,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:44:31,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:44:31,959 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:44:32,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:44:32,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:44:32,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:44:32,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:44:32,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:33,239 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:44:33,239 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:44:35,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:35,691 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:44:35,691 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:44:37,693 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:38,109 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:44:38,109 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:44:40,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:40,707 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:44:40,707 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:44:42,709 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:43,193 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:44:43,193 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:44:45,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:45,655 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:44:45,657 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:44:47,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:48,165 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:44:48,167 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 5 queries submitted, 0 failed, 5332 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:44:50,168 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:44:50,486 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:44:50,486 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 19.6900000572. ############################################################ # # RAM FootPrint: 4166.87109375 # ############################################################ 2022-02-08 03:44:50,523 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 03:44:50,525 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 78 seconds. Reason: Iteration:19 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 5355 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:46:08,530 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 03:46:08,532 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 03:46:13,536 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:46:14,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:15,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:46:15,405 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:15,405 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:46:15,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:46:15,548 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:46:15,618 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:46:15,658 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:46:15,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:46:15,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:46:15,749 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:46:15,749 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:46:15,911 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['116914 memcached\n'], []) 2022-02-08 03:46:15,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 03:46:15,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:46:15,963 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['116914 memcached\n'], []) 2022-02-08 03:46:15,963 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:46:15,967 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:46:17,020 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:17,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:46:17,622 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:17,622 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:46:17,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:46:17,678 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:46:17,698 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:46:17,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:46:17,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:46:17,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:46:17,835 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:46:17,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:46:17,891 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['27152 memcached\n'], []) 2022-02-08 03:46:17,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 03:46:17,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:46:17,967 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['27152 memcached\n'], []) 2022-02-08 03:46:17,967 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:46:17,970 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:46:18,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:19,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:46:19,572 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:19,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:46:19,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:46:19,645 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:46:19,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:46:19,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:46:19,710 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:46:19,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:46:19,763 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:46:19,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:46:19,836 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['15628 memcached\n'], []) 2022-02-08 03:46:19,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 03:46:19,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:46:19,914 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['15628 memcached\n'], []) 2022-02-08 03:46:19,914 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:46:19,917 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:46:20,753 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:21,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:46:21,509 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:21,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:46:21,530 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:46:21,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:46:21,576 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:46:21,598 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:46:21,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:46:21,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:46:21,664 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:46:21,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:46:21,734 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['51037 memcached\n'], []) 2022-02-08 03:46:21,734 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 03:46:21,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:46:21,790 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['51037 memcached\n'], []) 2022-02-08 03:46:21,790 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:46:21,792 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:46:22,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:23,311 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:46:23,605 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:23,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:46:23,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:46:23,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:46:23,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:46:23,678 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:46:23,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:46:23,724 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:46:23,759 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:46:23,760 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:46:23,875 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['44810 memcached\n'], []) 2022-02-08 03:46:23,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 03:46:23,917 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:46:23,940 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['44810 memcached\n'], []) 2022-02-08 03:46:23,941 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:46:23,943 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:46:24,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:25,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:46:25,457 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:25,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:46:25,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:46:25,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:46:25,507 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:46:25,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:46:25,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:46:25,586 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:46:25,604 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:46:25,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:46:25,664 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['86120 memcached\n'], []) 2022-02-08 03:46:25,665 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 03:46:25,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:46:25,716 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['86120 memcached\n'], []) 2022-02-08 03:46:25,716 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:46:25,717 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:46:26,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:26,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:46:26,992 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:26,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:46:27,007 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:46:27,030 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:46:27,048 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:46:27,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:46:27,104 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:46:27,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:46:27,197 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:46:27,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:46:27,256 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['79746 memcached\n'], []) 2022-02-08 03:46:27,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 03:46:27,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:46:27,312 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['79746 memcached\n'], []) 2022-02-08 03:46:27,312 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:46:27,315 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:46:28,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:28,762 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:46:28,921 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:28,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:46:28,936 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:46:28,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:46:28,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:46:28,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:46:29,002 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:46:29,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:46:29,051 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:46:29,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:46:29,119 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['13567 memcached\n'], []) 2022-02-08 03:46:29,119 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 03:46:29,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:46:29,171 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['13567 memcached\n'], []) 2022-02-08 03:46:29,173 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:46:29,174 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:46:29,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:30,326 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:46:30,497 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:30,499 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:46:30,522 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:46:30,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:46:30,553 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:46:30,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:46:30,586 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:46:30,614 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:46:30,628 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:46:30,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:46:30,681 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['93656 memcached\n'], []) 2022-02-08 03:46:30,683 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 03:46:30,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:46:30,729 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['93656 memcached\n'], []) 2022-02-08 03:46:30,729 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:46:30,730 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:46:30,822 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:31,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:46:31,104 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:31,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:46:31,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:46:31,131 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:46:31,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:46:31,161 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:46:31,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:46:31,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:46:31,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:46:31,224 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 03:46:31,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:46:31,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:33,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:46:33,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:34,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:34,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:46:34,276 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 03:46:34,278 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:46:34,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:34,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:46:34,411 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 03:46:34,411 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:46:34,413 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:46:34,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:34,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:46:34,816 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:34,816 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:46:34,831 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:46:34,844 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:46:34,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:46:34,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:46:34,882 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:46:34,895 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:46:34,910 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:46:34,924 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 03:46:34,951 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:46:34,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:38,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:46:38,397 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:38,607 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:38,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:46:38,796 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 03:46:38,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:46:38,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:38,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:46:38,880 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 03:46:38,880 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:46:38,881 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:46:38,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:39,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:46:39,219 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:39,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:46:39,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:46:39,244 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:46:39,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:46:39,272 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:46:39,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:46:39,298 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:46:39,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:46:39,325 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 03:46:39,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:46:39,362 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:42,426 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:46:42,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:42,959 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:43,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:46:43,164 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 03:46:43,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:46:43,178 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:43,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:46:43,266 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 03:46:43,266 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:46:43,269 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 03:46:43,372 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:43,594 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 03:46:43,660 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:43,660 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 03:46:43,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 03:46:43,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 03:46:43,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 03:46:43,721 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 03:46:43,736 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 03:46:43,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 03:46:43,767 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:46:43,785 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 03:46:43,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:46:43,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:43,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:46:43,857 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:43,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:43,884 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:46:43,901 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 03:46:43,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:46:43,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:43,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:46:43,946 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:43,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:46:44,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:44,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:44,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:44,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:44,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:44,142 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:44,170 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:44,198 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:44,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:44,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:46:44,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:44,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:44,292 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:44,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:44,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:44,342 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:44,359 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:44,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:44,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:44,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:46:44,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:44,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:44,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:44,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:44,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:44,525 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:44,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:44,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:44,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:44,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:46:44,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:44,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:44,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:44,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:44,707 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:44,724 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:44,742 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:44,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:44,775 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 03:46:44,776 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:46:44,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:45,119 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:46:45,213 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:45,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:46:45,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:46:45,265 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:46:45,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:46:45,298 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:46:45,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:46:45,331 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:46:45,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:46:45,367 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 03:46:45,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:46:45,405 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:47,260 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:46:47,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:47,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:48,140 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:46:48,154 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 03:46:48,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:46:48,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:48,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:46:48,288 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 03:46:48,289 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:46:48,290 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 03:46:48,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:48,609 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 03:46:48,674 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:48,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 03:46:48,688 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 03:46:48,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 03:46:48,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 03:46:48,744 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 03:46:48,765 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 03:46:48,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 03:46:48,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:46:48,812 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 03:46:48,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:46:48,854 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:48,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:46:48,887 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:48,900 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:48,913 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:46:48,926 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 03:46:48,926 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:46:48,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:48,954 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:46:48,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:48,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:46:49,028 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:49,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:49,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:49,109 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:49,140 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:49,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:49,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:49,221 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 5 queries submitted, 0 failed, 5375 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:46:49,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:49,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:46:49,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:49,293 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:49,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:49,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:49,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:49,354 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:49,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:49,382 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:49,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:49,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:46:49,428 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:49,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:49,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:49,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:49,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:49,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:49,522 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:49,538 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:49,553 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:49,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:46:49,607 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:49,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:49,645 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:49,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:49,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:49,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:49,720 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:49,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:49,755 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 03:46:49,757 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:46:49,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:50,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:46:50,084 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:50,084 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:46:50,099 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:46:50,112 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:46:50,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:46:50,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:46:50,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:46:50,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:46:50,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:46:50,194 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 03:46:50,219 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:46:50,236 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:53,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:46:53,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:53,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:53,816 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:46:53,831 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 03:46:53,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:46:53,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:53,891 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:46:53,905 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 03:46:53,905 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:46:53,907 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:46:54,016 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:54,265 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:46:54,342 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:54,344 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:46:54,359 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:46:54,375 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:46:54,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:46:54,410 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:46:54,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:46:54,451 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:46:54,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:46:54,507 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 03:46:54,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:46:54,558 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:57,362 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:46:57,697 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:57,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:58,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:46:58,130 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 03:46:58,130 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:46:58,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:58,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:46:58,293 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 03:46:58,293 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:46:58,296 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 03:46:58,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:46:58,630 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 03:46:58,723 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:46:58,724 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 03:46:58,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 03:46:58,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 03:46:58,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 03:46:58,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 03:46:58,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 03:46:58,816 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 03:46:58,836 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:46:58,857 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 03:46:58,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:46:58,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:58,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:46:58,941 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:58,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:46:58,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:46:58,992 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 03:46:58,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:46:59,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:46:59,026 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:46:59,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:59,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:46:59,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:59,130 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:59,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:59,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:59,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:59,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:59,282 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:59,312 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:46:59,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:59,354 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:46:59,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:59,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:59,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:59,413 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:59,428 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:59,443 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:59,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:59,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:46:59,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:59,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:46:59,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:59,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:59,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:59,586 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:59,607 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:59,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:59,642 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:59,660 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:46:59,676 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:59,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:46:59,711 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:59,729 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:59,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:59,767 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:59,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:59,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:59,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:59,844 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:46:59,865 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 03:46:59,867 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 03:46:59,954 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:47:00,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 03:47:00,243 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:47:00,244 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 03:47:00,260 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 03:47:00,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 03:47:00,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 03:47:00,308 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 03:47:00,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 03:47:00,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 03:47:00,349 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:47:00,367 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 03:47:00,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:47:00,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:47:00,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:47:00,438 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:47:00,454 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:47:00,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:47:00,483 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 03:47:00,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:47:00,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:47:00,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:47:00,530 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:00,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:47:00,594 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:00,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:00,655 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:00,686 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:00,716 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:00,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:00,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:00,798 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:00,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:00,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:47:00,865 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:00,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:00,898 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:00,915 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:00,933 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:00,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:00,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:00,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:01,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:01,023 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:47:01,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:01,058 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:01,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:01,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:01,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:01,127 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:01,141 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:01,157 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:01,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:01,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:47:01,204 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:01,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:01,236 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:01,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:01,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:01,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:01,302 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:01,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:01,334 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 03:47:01,336 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:47:01,510 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:47:01,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:47:01,803 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:47:01,803 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:47:01,816 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:47:01,831 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:47:01,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:47:01,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:47:01,874 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:47:01,887 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:47:01,913 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:47:01,943 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 03:47:01,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:47:01,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:47:04,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:47:05,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:47:05,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:47:05,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:47:05,653 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 03:47:05,653 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:47:05,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:47:05,765 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:47:05,835 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 03:47:05,836 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:47:05,838 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:47:06,545 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:47:06,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:47:07,094 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:47:07,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:47:07,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:47:07,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:47:07,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:47:07,224 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:47:07,242 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:47:07,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:47:07,289 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:47:07,306 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 03:47:07,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:47:07,346 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:47:09,345 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:47:09,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:47:09,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:47:09,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:47:09,940 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 03:47:09,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:47:09,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:47:10,003 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:47:10,025 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 03:47:10,026 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:47:10,028 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:47:10,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:47:10,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:47:10,407 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:47:10,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:47:10,424 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:47:10,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:47:10,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:47:10,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:47:10,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:47:10,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:47:10,525 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:47:10,542 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 03:47:10,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:47:10,594 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:47:13,585 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:47:13,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:47:14,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:47:14,407 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:47:14,426 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 03:47:14,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:47:14,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:47:14,493 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:47:14,509 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 03:47:14,509 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:47:14,510 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 03:47:14,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:47:14,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 03:47:14,905 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:47:14,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 03:47:14,921 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 03:47:14,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 03:47:14,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 03:47:14,966 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 03:47:14,979 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 03:47:14,992 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 03:47:15,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:47:15,023 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 03:47:15,053 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:47:15,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:47:15,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:47:15,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:47:15,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:47:15,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:47:15,153 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 03:47:15,153 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:47:15,170 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:47:15,188 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:47:15,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:15,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:47:15,266 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:15,293 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:15,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:15,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:15,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:15,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:15,448 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:15,479 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:47:15,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:15,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:47:15,540 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:15,558 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:15,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:15,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:15,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:15,624 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:15,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:15,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:47:15,673 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:15,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:47:15,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:15,721 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:15,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:15,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:15,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:15,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:15,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:15,823 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:47:15,839 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:15,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:47:15,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:15,892 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:15,908 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:15,930 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:15,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:15,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:15,986 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:16,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:47:16,023 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 03:47:16,025 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:47:16,026 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:47:16,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:47:16,341 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:47:16,420 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:47:16,421 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:47:16,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:47:16,447 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:47:16,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:47:16,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:47:16,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:47:16,517 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:47:16,532 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:17,138 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:47:17,140 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.11500000954. 2022-02-08 03:47:17,140 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:47:17,141 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:47:17,242 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:47:17,581 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:47:17,661 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:47:17,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:47:17,683 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:47:17,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:47:17,723 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:47:17,749 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:47:17,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:47:17,786 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:47:17,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:18,066 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:47:18,066 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:47:20,068 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:20,408 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:47:20,410 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:47:22,410 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:22,565 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:47:22,565 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:47:24,565 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:24,776 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:47:24,776 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:47:26,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:27,197 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:47:27,197 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:47:29,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:29,434 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:47:29,434 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:47:31,436 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:31,625 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:47:31,625 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:47:33,628 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:33,937 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:47:33,937 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:47:35,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:36,121 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:47:36,121 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:47:38,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:38,348 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:47:38,348 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:47:40,348 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:40,536 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:47:40,536 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:47:42,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:42,776 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:47:42,776 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:47:44,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:44,973 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:47:44,973 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:47:46,974 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:47,229 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:47:47,229 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:47:49,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 5399 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:47:49,463 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:47:49,463 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:47:51,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:51,674 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:47:51,676 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:47:53,676 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:53,897 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:47:53,897 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 36.7569999695. 2022-02-08 03:47:53,898 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:47:53,900 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:47:54,609 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:47:54,989 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:47:55,161 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:47:55,163 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:47:55,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:47:55,237 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:47:55,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:47:55,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:47:55,318 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:47:55,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:47:55,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:56,056 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:47:56,056 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 03:47:58,058 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:47:58,256 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:47:58,256 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.35899996758. 2022-02-08 03:47:58,256 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:47:58,257 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:47:59,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:47:59,384 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:47:59,515 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:47:59,516 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:47:59,538 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:47:59,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:47:59,591 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:47:59,615 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:47:59,655 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:47:59,691 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:47:59,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:48:00,421 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:48:00,421 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 03:48:02,426 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:48:02,621 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:48:02,621 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.36599993706. 2022-02-08 03:48:02,622 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:48:02,624 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:48:03,374 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:48:03,713 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:48:03,875 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:48:03,877 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:48:03,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:48:03,931 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:48:03,959 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:48:03,987 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:48:04,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:48:04,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:48:04,079 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:48:04,829 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:48:04,829 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.20700001717. 2022-02-08 03:48:04,829 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:48:04,832 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:48:05,867 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:48:06,381 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:48:06,704 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:48:06,706 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:48:06,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:48:06,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:48:06,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:48:07,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:48:07,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:48:07,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:48:07,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:48:08,371 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:48:08,371 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 03:48:10,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:48:10,802 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:48:10,802 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.128 2022-02-08 03:48:12,802 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:48:13,217 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:48:13,217 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 8.38700008392. 2022-02-08 03:48:13,217 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:48:13,220 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:48:13,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:48:13,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:48:13,881 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:48:13,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:48:13,908 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:48:13,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:48:14,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:48:14,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:48:14,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:48:14,091 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:48:14,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:48:14,888 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:48:14,890 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.67300009727. 2022-02-08 03:48:14,890 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:48:14,891 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:48:15,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:48:15,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:48:16,040 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:48:16,042 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:48:16,063 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:48:16,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:48:16,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:48:16,121 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:48:16,135 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:48:16,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:48:16,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:48:16,835 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:48:16,835 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.94599986076. 2022-02-08 03:48:16,836 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:48:16,838 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:48:18,025 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:48:18,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:48:18,607 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:48:18,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:48:18,634 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:48:18,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:48:18,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:48:18,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:48:18,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:48:18,789 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:48:18,806 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:48:19,997 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:48:19,999 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.16300010681. ############################################################ # # RAM FootPrint: 4164.70703125 # ############################################################ 2022-02-08 03:48:20,036 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 03:48:20,036 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 116 seconds. Reason: Iteration:20 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 5424 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 5448 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:50:16,040 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 03:50:16,042 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 03:50:21,045 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:50:22,102 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:50:22,782 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:50:23,108 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:50:23,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:50:23,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:50:23,217 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:50:23,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:50:23,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:50:23,346 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:50:23,382 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:50:23,401 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:50:23,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:50:23,471 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['117621 memcached\n'], []) 2022-02-08 03:50:23,471 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 03:50:23,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 03:50:23,572 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['117621 memcached\n'], []) 2022-02-08 03:50:23,572 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:50:23,575 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:50:24,571 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:50:25,190 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:50:25,434 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:50:25,436 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:50:25,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:50:25,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:50:25,496 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:50:25,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:50:25,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:50:25,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:50:25,612 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:50:25,612 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:50:25,683 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['28017 memcached\n'], []) 2022-02-08 03:50:25,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 03:50:25,716 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 03:50:25,744 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['28017 memcached\n'], []) 2022-02-08 03:50:25,744 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:50:25,746 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:50:26,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:50:26,793 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:50:27,253 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:50:27,253 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:50:27,275 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:50:27,377 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:50:27,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:50:27,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:50:27,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:50:27,487 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:50:27,509 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:50:27,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:50:27,605 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['16444 memcached\n'], []) 2022-02-08 03:50:27,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 03:50:27,687 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 03:50:27,717 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['16444 memcached\n'], []) 2022-02-08 03:50:27,717 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:50:27,720 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:50:28,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:50:29,263 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:50:29,466 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:50:29,466 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:50:29,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:50:29,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:50:29,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:50:29,665 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:50:29,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:50:29,732 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:50:29,783 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:50:29,783 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:50:29,848 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['51744 memcached\n'], []) 2022-02-08 03:50:29,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 03:50:29,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 03:50:29,947 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['51744 memcached\n'], []) 2022-02-08 03:50:29,947 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:50:29,950 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:50:30,885 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:50:31,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:50:31,835 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:50:31,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:50:31,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:50:31,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:50:31,887 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:50:31,903 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:50:31,915 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:50:31,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:50:31,967 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:50:31,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:50:32,036 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['45533 memcached\n'], []) 2022-02-08 03:50:32,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 03:50:32,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 03:50:32,092 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['45533 memcached\n'], []) 2022-02-08 03:50:32,092 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:50:32,095 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:50:32,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:50:33,694 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:50:34,154 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:50:34,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:50:34,173 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:50:34,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:50:34,322 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:50:34,352 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:50:34,367 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:50:34,392 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:50:34,411 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:50:34,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:50:34,473 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['86797 memcached\n'], []) 2022-02-08 03:50:34,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 03:50:34,499 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 03:50:34,522 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['86797 memcached\n'], []) 2022-02-08 03:50:34,523 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:50:34,525 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:50:35,305 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:50:36,290 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:50:36,479 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:50:36,479 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:50:36,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:50:36,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:50:36,546 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:50:36,566 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:50:36,605 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:50:36,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:50:36,651 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:50:36,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:50:36,710 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['80454 memcached\n'], []) 2022-02-08 03:50:36,711 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 03:50:36,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 03:50:36,762 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['80454 memcached\n'], []) 2022-02-08 03:50:36,763 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:50:36,765 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:50:37,584 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:50:38,056 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:50:38,198 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:50:38,200 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:50:38,214 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:50:38,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:50:38,249 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:50:38,267 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:50:38,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:50:38,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:50:38,336 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:50:38,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:50:38,398 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['14274 memcached\n'], []) 2022-02-08 03:50:38,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 03:50:38,426 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 03:50:38,450 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['14274 memcached\n'], []) 2022-02-08 03:50:38,450 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:50:38,451 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:50:38,933 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:50:39,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:50:39,338 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:50:39,338 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:50:39,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:50:39,387 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:50:39,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:50:39,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:50:39,492 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:50:39,542 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:50:39,559 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 03:50:39,559 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:50:39,634 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['94451 memcached\n'], []) 2022-02-08 03:50:39,635 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 03:50:39,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 03:50:39,680 | infra | DEBUG | Thread-4596 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['94451 memcached\n'], []) 2022-02-08 03:50:39,681 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:50:39,684 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:50:39,760 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:50:39,984 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:50:40,066 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:50:40,068 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:50:40,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:50:40,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:50:40,108 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:50:40,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:50:40,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:50:40,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:50:40,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:50:40,181 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 03:50:40,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:50:40,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:50:42,963 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:50:43,276 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:50:43,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:50:43,647 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:50:43,661 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 03:50:43,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:50:43,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:50:43,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:50:43,744 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 03:50:43,744 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:50:43,746 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:50:43,818 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:50:44,035 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:50:44,099 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:50:44,099 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:50:44,114 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:50:44,125 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:50:44,137 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:50:44,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:50:44,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:50:44,176 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:50:44,188 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:50:44,204 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 03:50:44,232 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:50:44,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:50:47,255 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:50:47,611 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:50:47,809 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:50:47,990 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:50:48,003 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 03:50:48,005 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:50:48,017 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:50:48,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:50:48,085 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 03:50:48,085 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:50:48,088 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:50:48,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:50:48,374 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:50:48,463 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:50:48,463 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:50:48,474 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:50:48,486 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:50:48,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:50:48,517 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:50:48,533 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:50:48,552 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:50:48,568 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:50:48,582 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 03:50:48,614 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:50:48,630 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 5472 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:50:51,885 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:50:52,236 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:50:52,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:50:52,638 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:50:52,651 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 03:50:52,651 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:50:52,664 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:50:52,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:50:52,747 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 03:50:52,747 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:50:52,750 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 03:50:52,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:50:53,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 03:50:53,144 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:50:53,144 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 03:50:53,158 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 03:50:53,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 03:50:53,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 03:50:53,206 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 03:50:53,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 03:50:53,233 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 03:50:53,252 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:50:53,269 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 03:50:53,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:50:53,316 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:50:53,335 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:50:53,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:50:53,365 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:50:53,380 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:50:53,394 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 03:50:53,394 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:50:53,411 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:50:53,427 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:50:53,441 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:50:53,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:50:53,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:50:53,536 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:50:53,569 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:50:53,596 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:50:53,630 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:50:53,658 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:50:53,688 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:50:53,719 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:50:53,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:50:53,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:50:53,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:50:53,798 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:50:53,813 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:50:53,831 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:50:53,846 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:50:53,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:50:53,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:50:53,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:50:53,911 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:50:53,927 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:50:53,944 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:50:53,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:50:53,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:50:53,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:50:54,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:50:54,030 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:50:54,049 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:50:54,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:50:54,081 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:50:54,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:50:54,117 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:50:54,134 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:50:54,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:50:54,170 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:50:54,187 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:50:54,203 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:50:54,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:50:54,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:50:54,260 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 03:50:54,263 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:50:54,595 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:50:55,244 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:50:55,772 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:50:55,772 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:50:55,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:50:56,051 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:50:56,252 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:50:56,500 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:50:56,648 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:50:56,756 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:50:56,857 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:50:56,990 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 03:50:57,015 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:50:57,194 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:50:59,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:50:59,765 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:00,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:00,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:51:00,255 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 03:51:00,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:51:00,270 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:00,319 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:51:00,453 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 03:51:00,453 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:51:00,457 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 03:51:00,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:51:00,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 03:51:00,835 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:51:00,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 03:51:00,849 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 03:51:00,862 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 03:51:00,875 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 03:51:00,892 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 03:51:00,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 03:51:00,920 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 03:51:00,934 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:51:00,950 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 03:51:00,977 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:51:00,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:01,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:51:01,030 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:01,045 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:01,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:51:01,072 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 03:51:01,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:51:01,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:01,104 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:51:01,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:01,147 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:51:01,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:01,211 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:01,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:01,273 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:01,303 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:01,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:01,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:01,391 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:01,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:01,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:51:01,456 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:01,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:01,489 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:01,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:01,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:01,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:01,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:01,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:01,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:01,604 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:51:01,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:01,637 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:01,653 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:01,668 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:01,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:01,698 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:01,714 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:01,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:01,744 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:01,762 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:51:01,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:01,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:01,819 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:01,835 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:01,852 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:01,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:01,888 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:01,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:01,921 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 03:51:01,923 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:51:02,020 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:51:02,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:51:02,357 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:51:02,357 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:51:02,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:51:02,387 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:51:02,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:51:02,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:51:02,431 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:51:02,450 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:51:02,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:51:02,489 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 03:51:02,513 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:51:02,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:05,467 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:51:05,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:06,076 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:06,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:51:06,299 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 03:51:06,299 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:51:06,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:06,357 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:51:06,375 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 03:51:06,375 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:51:06,378 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:51:06,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:51:06,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:51:06,828 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:51:06,828 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:51:06,845 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:51:06,864 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:51:06,881 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:51:06,907 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:51:06,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:51:06,980 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:51:07,000 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:51:07,019 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 03:51:07,048 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:51:07,065 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:09,848 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:51:10,180 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:10,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:10,549 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:51:10,566 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 03:51:10,566 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:51:10,582 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:10,632 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:51:10,651 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 03:51:10,653 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:51:10,654 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 03:51:10,734 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:51:10,967 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 03:51:11,033 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:51:11,033 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 03:51:11,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 03:51:11,059 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 03:51:11,072 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 03:51:11,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 03:51:11,102 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 03:51:11,115 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 03:51:11,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:51:11,142 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 03:51:11,167 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:51:11,183 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:11,197 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:51:11,213 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:11,230 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:11,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:51:11,260 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 03:51:11,260 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:51:11,275 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:11,290 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:51:11,303 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:11,332 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:51:11,368 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:11,398 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:11,431 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:11,461 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:11,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:11,529 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:11,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:11,592 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:11,617 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:11,631 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:51:11,650 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:11,667 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:11,684 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:11,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:11,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:11,733 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:11,750 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:11,769 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:11,785 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:11,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:51:11,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:11,842 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:11,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:11,874 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:11,892 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:11,908 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:11,924 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:11,938 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:11,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:11,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:51:11,987 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:12,005 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:12,020 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:12,036 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:12,052 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:12,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:12,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:12,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:12,135 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 03:51:12,138 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 03:51:12,221 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:51:12,440 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 03:51:12,503 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:51:12,503 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 03:51:12,517 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 03:51:12,530 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 03:51:12,543 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 03:51:12,562 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 03:51:12,575 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 03:51:12,589 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 03:51:12,604 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:51:12,618 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 03:51:12,642 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:51:12,657 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:12,674 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:51:12,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:12,701 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:12,716 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:51:12,729 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 03:51:12,729 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:51:12,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:12,757 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:51:12,775 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:12,803 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:51:12,839 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:12,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:12,904 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:12,933 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:12,966 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:12,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:13,028 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:13,058 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:13,085 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:13,101 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:51:13,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:13,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:13,154 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:13,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:13,186 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:13,201 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:13,220 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:13,234 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:13,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:13,265 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:51:13,280 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:13,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:13,311 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:13,325 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:13,341 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:13,355 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:13,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:13,385 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:13,401 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:13,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:51:13,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:13,453 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:13,470 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:13,484 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:13,502 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:13,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:13,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:13,558 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:13,575 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 03:51:13,578 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:51:13,690 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:51:13,950 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:51:14,017 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:51:14,019 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:51:14,033 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:51:14,048 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:51:14,061 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:51:14,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:51:14,092 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:51:14,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:51:14,118 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:51:14,141 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 03:51:14,168 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:51:14,186 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:16,914 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:51:17,275 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:17,483 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:17,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:51:17,677 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 03:51:17,677 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:51:17,693 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:17,739 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:51:17,755 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 03:51:17,756 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:51:17,757 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:51:18,415 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:51:18,773 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:51:18,938 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:51:18,940 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:51:18,957 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:51:18,979 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:51:18,993 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:51:19,032 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:51:19,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:51:19,073 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:51:19,088 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:51:19,108 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 03:51:19,132 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:51:19,151 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:21,361 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:51:21,644 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:21,795 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:21,941 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:51:21,960 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 03:51:21,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:51:21,973 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:22,009 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:51:22,029 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 03:51:22,029 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:51:22,030 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:51:22,138 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:51:22,378 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:51:22,457 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:51:22,457 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:51:22,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:51:22,490 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:51:22,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:51:22,526 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:51:22,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:51:22,553 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:51:22,571 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:51:22,591 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 03:51:22,625 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:51:22,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:25,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:51:26,226 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:26,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:26,680 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:51:26,698 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 03:51:26,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:51:26,717 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:26,757 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:51:26,776 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 03:51:26,776 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:51:26,778 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 03:51:26,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:51:27,107 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 03:51:27,177 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:51:27,177 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 03:51:27,193 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 03:51:27,207 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 03:51:27,223 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 03:51:27,240 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 03:51:27,256 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 03:51:27,272 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 03:51:27,286 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 03:51:27,303 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 03:51:27,328 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 03:51:27,351 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:27,371 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 03:51:27,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:27,403 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 03:51:27,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 03:51:27,434 | test | DEBUG | Thread-4596 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 03:51:27,434 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 03:51:27,448 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 03:51:27,464 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 03:51:27,480 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:27,509 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 03:51:27,543 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:27,572 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:27,608 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:27,637 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:27,671 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:27,703 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:27,734 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:27,767 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 03:51:27,796 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:27,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 03:51:27,832 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:27,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:27,868 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:27,885 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:27,901 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:27,920 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:27,937 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:27,953 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 03:51:27,970 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:27,996 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 03:51:28,013 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:28,029 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:28,046 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:28,062 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:28,078 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:28,095 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:28,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:28,130 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 03:51:28,153 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:28,171 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 03:51:28,188 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:28,206 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:28,224 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:28,243 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:28,259 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:28,279 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:28,296 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:28,313 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 03:51:28,329 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 03:51:28,331 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:51:28,332 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 03:51:28,444 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:51:28,673 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 03:51:28,743 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:51:28,743 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 03:51:28,759 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 03:51:28,776 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 03:51:28,790 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 03:51:28,808 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 03:51:28,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 03:51:28,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 03:51:28,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:51:29,311 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 03:51:29,311 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.981000185013. 2022-02-08 03:51:29,311 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:51:29,312 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 03:51:29,430 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:51:29,691 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 03:51:29,763 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:51:29,763 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 03:51:29,778 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 03:51:29,792 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 03:51:29,815 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 03:51:29,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 03:51:29,851 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 03:51:29,871 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 03:51:29,894 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:51:30,548 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:51:30,549 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:51:32,555 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:51:32,857 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:51:32,858 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:51:34,858 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:51:35,052 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:51:35,053 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:51:37,055 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:51:37,345 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:51:37,345 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:51:39,349 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:51:39,740 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:51:39,740 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:51:41,740 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:51:41,976 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:51:41,976 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:51:43,976 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:51:44,200 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:51:44,200 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:51:46,201 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:51:46,403 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:51:46,404 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:51:48,404 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:51:48,588 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:51:48,588 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 5500 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:51:50,588 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:51:50,802 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:51:50,803 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:51:52,803 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:51:53,098 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:51:53,098 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:51:55,098 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:51:55,275 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:51:55,275 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:51:57,275 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:51:57,460 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:51:57,460 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 03:51:59,460 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:51:59,664 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 03:51:59,664 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 30.3539998531. 2022-02-08 03:51:59,665 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:51:59,667 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 03:52:00,621 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:52:00,947 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 03:52:01,082 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:52:01,082 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 03:52:01,104 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 03:52:01,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 03:52:01,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 03:52:01,174 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 03:52:01,193 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 03:52:01,227 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 03:52:01,250 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:52:01,865 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 03:52:01,868 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.20399999619. 2022-02-08 03:52:01,869 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:52:01,871 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 03:52:02,641 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:52:02,960 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 03:52:03,105 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:52:03,105 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 03:52:03,128 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 03:52:03,148 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 03:52:03,168 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 03:52:03,191 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 03:52:03,219 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 03:52:03,283 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 03:52:03,303 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:52:03,953 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 03:52:03,954 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.08599996567. 2022-02-08 03:52:03,954 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:52:03,957 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 03:52:04,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:52:05,150 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 03:52:05,309 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:52:05,309 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 03:52:05,331 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 03:52:05,358 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 03:52:05,382 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 03:52:05,417 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 03:52:05,437 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 03:52:05,473 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 03:52:05,494 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:52:06,160 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 03:52:06,160 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.20500016212. 2022-02-08 03:52:06,160 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:52:06,161 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 03:52:07,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:52:07,539 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 03:52:07,661 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:52:07,661 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 03:52:07,681 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 03:52:07,730 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 03:52:07,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 03:52:07,805 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 03:52:07,825 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 03:52:07,855 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 03:52:07,878 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:52:08,553 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 03:52:08,555 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.39499998093. 2022-02-08 03:52:08,555 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:52:08,556 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 03:52:09,698 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:52:10,269 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 03:52:10,746 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:52:10,746 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 03:52:10,838 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 03:52:10,928 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 03:52:11,040 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 03:52:11,069 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 03:52:11,181 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 03:52:11,397 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 03:52:11,523 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:52:12,607 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:52:12,607 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 03:52:14,607 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:52:15,033 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:52:15,033 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 03:52:17,033 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:52:17,555 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:52:17,555 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 03:52:19,556 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:52:19,766 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 03:52:19,766 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 11.2119998932. 2022-02-08 03:52:19,767 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:52:19,769 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 03:52:20,298 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:52:20,584 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 03:52:20,700 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:52:20,700 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 03:52:20,714 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 03:52:20,727 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 03:52:20,747 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 03:52:20,766 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 03:52:20,779 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 03:52:20,808 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 03:52:20,822 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:52:21,364 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 03:52:21,364 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.59600019455. 2022-02-08 03:52:21,364 | test | DEBUG | Thread-4596 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 03:52:21,365 | infra | DEBUG | Thread-4596 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 03:52:21,545 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 03:52:22,111 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 03:52:22,474 | infra | DEBUG | Thread-4596 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 03:52:22,476 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 03:52:22,519 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 03:52:22,601 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 03:52:22,775 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 03:52:22,903 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 03:52:23,006 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 03:52:23,075 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 03:52:23,246 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:52:24,414 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:52:24,414 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:52:26,414 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:52:26,736 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:52:26,737 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:52:28,737 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:52:29,161 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:52:29,163 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:52:31,164 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:52:31,505 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:52:31,506 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:52:33,506 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:52:34,059 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:52:34,059 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:52:36,061 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:52:36,385 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 03:52:36,385 | test | INFO | Thread-4596 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 03:52:38,388 | infra | DEBUG | Thread-4596 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 03:52:38,569 | infra | DEBUG | Thread-4596 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 03:52:38,569 | test | INFO | Thread-4596 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 17.2049999237. 2022-02-08 03:52:38,572 | test | DEBUG | Thread-4596 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L836 2022-02-08 03:52:38,572 | test | DEBUG | Thread-4596 | [common_lib:sleep:24] Sleep 300 seconds. Reason: None 5 queries submitted, 0 failed, 5525 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 5550 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 5576 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 5598 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 5622 passed, 0 rejected, 0 cancelled, 0 timeout ############################################################ # # Step 8: Rebalance In_Out with Loading of docs # ############################################################ KV nodes in cluster: ['172.23.121.116', '172.23.121.123', '172.23.121.124', '172.23.121.126', '172.23.121.127', '172.23.121.128', '172.23.121.129', '172.23.121.135'] CBAS nodes in cluster: [] INDEX nodes in cluster: ['172.23.121.130', '172.23.121.131', '172.23.121.132', '172.23.121.133', '172.23.121.134'] FTS nodes in cluster: [] QUERY nodes in cluster: ['172.23.121.130', '172.23.121.131', '172.23.121.132', '172.23.121.133', '172.23.121.134'] EVENTING nodes in cluster: [] AVAILABLE nodes for cluster: ['172.23.121.136', '172.23.121.115'] Servers coming in : ['172.23.121.115', '172.23.121.136'] with services: ['kv', 'kv'] Servers going out : ['172.23.121.128'] NEW AVAILABLE nodes for cluster: ['172.23.121.128'] 2022-02-08 03:57:38,674 | infra | DEBUG | MainThread | [task:__init__:249] Last known rebalance status_id: 5b2ed63791e9ba8c1825b8e4ae0b9ba4 2022-02-08 03:57:38,697 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[172.23.121.115,172.23.121.136]_OUT=[172.23.121.128]_1644321458.57 2022-02-08 03:57:38,697 | infra | INFO | pool-3-thread-11 | [task:start_task:91] Thread 'Rebalance_task_IN=[172.23.121.115,172.23.121.136]_OUT=[172.23.121.128]_1644321458.57' started 2022-02-08 03:57:38,703 | test | DEBUG | pool-3-thread-11 | [rest_client:add_node:1199] Adding remote node 172.23.121.115:8091 to cluster 172.23.120.170:8091 2022-02-08 03:57:38,724 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/Hospital.py -> SystemTestMagma():L541 2022-02-08 03:57:38,726 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep for 60s for rebalance to start 2022-02-08 03:57:43,867 | test | DEBUG | pool-3-thread-11 | [rest_client:add_node:1199] Adding remote node 172.23.121.136:8091 to cluster 172.23.120.170:8091 2022-02-08 03:57:47,654 | test | DEBUG | pool-3-thread-11 | [task:start_rebalance:384] Removing node 172.23.121.128:8091 from cluster 2022-02-08 03:57:47,655 | test | DEBUG | pool-3-thread-11 | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.128,ns_1@172.23.121.115,ns_1@172.23.121.132,ns_1@172.23.121.136,ns_1@172.23.121.135,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.134,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123,ns_1@172.23.121.131,ns_1@172.23.121.130,ns_1@172.23.121.133', 'ejectedNodes': u'ns_1@172.23.121.128', 'user': 'Administrator'} 2022-02-08 03:57:47,739 | test | DEBUG | pool-3-thread-11 | [rest_client:rebalance:1459] Rebalance operation started 2022-02-08 03:57:47,749 | test | INFO | pool-3-thread-11 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.121.128 | kv | 7.1.0-2255-enterprise | 85.0455303322 | --- OUT ---> | | 172.23.121.132 | index, n1ql | 7.1.0-2255-enterprise | 11.0522233713 | Cluster node | | 172.23.121.135 | kv | 7.1.0-2255-enterprise | 56.8745808182 | Cluster node | | 172.23.120.170 | kv | 7.1.0-2255-enterprise | 81.4810045074 | Cluster node | | 172.23.121.129 | kv | 7.1.0-2255-enterprise | 89.7653772987 | Cluster node | | 172.23.121.124 | kv | 7.1.0-2255-enterprise | 52.1289295663 | Cluster node | | 172.23.121.126 | kv | 7.1.0-2255-enterprise | 62.0084455001 | Cluster node | | 172.23.121.134 | index, n1ql | 7.1.0-2255-enterprise | 21.2042899599 | Cluster node | | 172.23.121.116 | kv | 7.1.0-2255-enterprise | 56.6524178466 | Cluster node | | 172.23.121.127 | kv | 7.1.0-2255-enterprise | 89.199540406 | Cluster node | | 172.23.121.123 | kv | 7.1.0-2255-enterprise | 60.0368324125 | Cluster node | | 172.23.121.131 | index, n1ql | 7.1.0-2255-enterprise | 10.6380241832 | Cluster node | | 172.23.121.130 | index, n1ql | 7.1.0-2255-enterprise | 14.0671983266 | Cluster node | | 172.23.121.133 | index, n1ql | 7.1.0-2255-enterprise | 11.5240191635 | Cluster node | | 172.23.121.115 | ['kv'] | | | <--- IN --- | | 172.23.121.136 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2022-02-08 03:57:47,759 | infra | DEBUG | pool-3-thread-11 | [task:call:316] New rebalance status_id: 561b9bd957c92caaedf4fae8efca7808 2022-02-08 03:57:47,780 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 03:57:47,780 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 03:57:47,825 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 03:57:47,828 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 03:57:47,828 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 5648 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:57:57,872 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 03:57:57,872 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 03:57:57,933 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 03:57:57,936 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 03:57:57,936 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 03:58:07,967 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.81 % 2022-02-08 03:58:07,967 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 0.81 2022-02-08 03:58:07,996 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 03:58:07,999 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 03:58:07,999 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 03:58:18,015 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.69 % 2022-02-08 03:58:18,015 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 1.69 2022-02-08 03:58:18,043 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 03:58:18,046 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 03:58:18,046 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 03:58:28,059 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.73 % 2022-02-08 03:58:28,059 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 2.73 2022-02-08 03:58:28,089 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 03:58:28,092 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 03:58:28,092 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 03:58:38,121 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.99 % 2022-02-08 03:58:38,122 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 3.99 2022-02-08 03:58:38,163 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 03:58:38,165 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 03:58:38,165 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 03:58:38,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L766 2022-02-08 03:58:38,733 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 30 seconds. Reason: Let the rebalance begin! 2022-02-08 03:58:48,181 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.14 % 2022-02-08 03:58:48,181 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 5.14 2022-02-08 03:58:48,221 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 03:58:48,224 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 03:58:48,224 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 5671 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:58:58,247 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.12 % 2022-02-08 03:58:58,249 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 6.12 2022-02-08 03:58:58,292 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 03:58:58,296 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 03:58:58,296 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 03:59:08,308 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.44 % 2022-02-08 03:59:08,309 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 7.44 2022-02-08 03:59:08,342 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 03:59:08,345 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 03:59:08,345 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 03:59:08,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.44 % 2022-02-08 03:59:08,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 03:59:08,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 03:59:18,359 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.16 % 2022-02-08 03:59:18,359 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 8.16 2022-02-08 03:59:18,391 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 03:59:18,394 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 03:59:18,394 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 03:59:18,852 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.16 % 2022-02-08 03:59:18,859 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 03:59:18,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 03:59:28,407 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.17 % 2022-02-08 03:59:28,407 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 9.17 2022-02-08 03:59:28,438 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 03:59:28,441 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 03:59:28,441 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 03:59:28,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.34 % 2022-02-08 03:59:28,882 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 03:59:28,884 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 03:59:38,461 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.13 % 2022-02-08 03:59:38,461 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 10.13 2022-02-08 03:59:38,496 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 03:59:38,500 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 03:59:38,500 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 03:59:38,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.13 % 2022-02-08 03:59:38,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 03:59:38,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 03:59:48,513 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.87 % 2022-02-08 03:59:48,513 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 11.87 2022-02-08 03:59:48,546 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 03:59:48,549 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 03:59:48,549 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 03:59:48,923 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.87 % 2022-02-08 03:59:48,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 03:59:48,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 5696 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 03:59:58,569 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.21 % 2022-02-08 03:59:58,571 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 12.21 2022-02-08 03:59:58,618 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 03:59:58,621 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 03:59:58,621 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 03:59:58,946 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.21 % 2022-02-08 03:59:58,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 03:59:58,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:00:08,634 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.16 % 2022-02-08 04:00:08,634 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 13.16 2022-02-08 04:00:08,674 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:00:08,677 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:00:08,677 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:00:08,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.16 % 2022-02-08 04:00:08,974 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:00:08,974 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:00:18,694 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.88 % 2022-02-08 04:00:18,694 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 13.88 2022-02-08 04:00:18,739 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:00:18,742 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:00:18,742 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:00:18,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.88 % 2022-02-08 04:00:19,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:00:19,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:00:28,760 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.72 % 2022-02-08 04:00:28,762 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 14.72 2022-02-08 04:00:28,802 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:00:28,805 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:00:28,805 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:00:29,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.72 % 2022-02-08 04:00:29,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:00:29,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:00:38,819 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.05 % 2022-02-08 04:00:38,819 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 16.05 2022-02-08 04:00:38,848 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:00:38,851 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:00:38,851 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:00:39,045 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.05 % 2022-02-08 04:00:39,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:00:39,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:00:48,874 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.61 % 2022-02-08 04:00:48,874 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 16.61 2022-02-08 04:00:48,914 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:00:48,918 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:00:48,918 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:00:49,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.61 % 2022-02-08 04:00:49,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:00:49,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 5720 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:00:58,938 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.18 % 2022-02-08 04:00:58,938 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 17.18 2022-02-08 04:00:58,986 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:00:58,990 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:00:58,990 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:00:59,094 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.18 % 2022-02-08 04:00:59,104 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:00:59,104 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:01:09,003 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.07 % 2022-02-08 04:01:09,005 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 18.07 2022-02-08 04:01:09,036 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:01:09,039 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:01:09,039 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:01:09,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.07 % 2022-02-08 04:01:09,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:01:09,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:01:19,059 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.64 % 2022-02-08 04:01:19,059 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 18.64 2022-02-08 04:01:19,101 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:01:19,105 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:01:19,105 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:01:19,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.64 % 2022-02-08 04:01:19,153 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:01:19,154 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:01:29,121 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.58 % 2022-02-08 04:01:29,121 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 19.58 2022-02-08 04:01:29,151 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:01:29,153 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:01:29,154 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:01:29,165 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.58 % 2022-02-08 04:01:29,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:01:29,173 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:01:39,177 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.65 % 2022-02-08 04:01:39,178 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 20.65 2022-02-08 04:01:39,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.65 % 2022-02-08 04:01:39,194 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:01:39,196 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:01:39,224 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:01:39,226 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:01:39,226 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:01:49,196 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >20.65% in 160.400000095 seconds 2022-02-08 04:01:49,217 | test | DEBUG | MainThread | [rest_client:diag_eval:1476] /diag/eval status on 172.23.120.170:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2022-02-08 04:01:49,219 | test | INFO | MainThread | [opd:abort_rebalance:780] Abort rebalance 2022-02-08 04:01:49,220 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:01:49,250 | test | DEBUG | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.82 % 2022-02-08 04:01:49,250 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance - status: running, progress: 20.82 5 queries submitted, 0 failed, 5744 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:01:49,296 | infra | DEBUG | pool-3-thread-11 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:01:49,299 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:01:49,299 | infra | DEBUG | pool-3-thread-11 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:01:49,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:01:50,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:01:50,691 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:01:50,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:01:50,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:01:50,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:01:50,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:01:50,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:01:50,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:01:50,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:01:50,865 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:01:50,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:01:50,950 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['95191 memcached\n'], []) 2022-02-08 04:01:50,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 04:01:51,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:01:51,042 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['95191 memcached\n'], []) 2022-02-08 04:01:51,043 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:01:51,046 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:01:51,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:01:51,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:01:51,434 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:01:51,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:01:51,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:01:51,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:01:51,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:01:51,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:01:51,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:01:51,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:01:51,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:01:51,562 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 04:01:51,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:01:51,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:01:54,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:01:54,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:01:54,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:01:54,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:01:54,871 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 04:01:54,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:01:54,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:01:54,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:01:54,940 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 04:01:54,940 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:01:54,943 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:01:55,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:01:55,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:01:55,322 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:01:55,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:01:55,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:01:55,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:01:55,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:01:55,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:01:55,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:01:55,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:01:55,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:01:55,430 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 04:01:55,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:01:55,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:01:55,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:01:55,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:01:55,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:01:55,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:01:55,540 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 04:01:55,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:01:55,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:01:55,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:01:55,586 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 04:01:55,588 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:01:55,589 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:01:56,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:01:56,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:01:56,770 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:01:56,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:01:56,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:01:56,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:01:56,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:01:56,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:01:56,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:01:56,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:01:56,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:01:56,898 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 04:01:56,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:01:56,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:01:59,354 | test | ERROR | pool-3-thread-11 | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'a69bd93d976ba37350a3f67a10ae2d23', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=5ca59fc024dad2696fa5fd1995b9ad31', u'status': u'notRunning'} - rebalance failed 2022-02-08 04:01:59,474 | test | INFO | pool-3-thread-11 | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.120.170: 2022-02-08 04:01:59,476 | test | ERROR | pool-3-thread-11 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644321712019L, u'shortText': u'message', u'serverTime': u'2022-02-08T04:01:52.019Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.28432.81>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,stats_recv,4,\n [{file,"src/mc_client_binary.erl"},\n {line,163}]},\n {mc_client_binary,stats,4,\n [{file,"src/mc_client_binary.erl"},\n {line,455}]},\n {ns_memcached,do_handle_call,3,\n [{file,"src/ns_memcached.erl"},\n {line,508}]},\n {ns_memcached,worker_loop,3,\n [{file,"src/ns_memcached.erl"},\n {line,227}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,226}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,119,\n "replication:ns_1@172.23.120.170->ns_1@172.23.121.115:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.120.170\'},\n {if_rebalance,<0.934.81>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.121.115\'],\n 119}},\n infinity]}}}}}.\nRebalance Operation Id = e99352b42373454ddf279886adfb4d4e'} 2022-02-08 04:01:59,476 | test | ERROR | pool-3-thread-11 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644321711990L, u'shortText': u'message', u'serverTime': u'2022-02-08T04:01:51.990Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW0208 03:51:42.160712 95256 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2022-02-08 04:01:59,476 | test | ERROR | pool-3-thread-11 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644321711977L, u'shortText': u'message', u'serverTime': u'2022-02-08T04:01:51.977Z', u'text': u'Worker <0.28195.81> (for action {move,{119,\n [\'ns_1@172.23.120.170\',\n \'ns_1@172.23.121.124\'],\n [\'ns_1@172.23.121.115\',\n \'ns_1@172.23.120.170\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.28432.81>,\n {{{{badmatch,\n {error,\n closed}},\n [{mc_client_binary,\n stats_recv,\n 4,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 163}]},\n {mc_client_binary,\n stats,\n 4,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 455}]},\n {ns_memcached,\n do_handle_call,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 508}]},\n {ns_memcached,\n worker_loop,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 227}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 226}]}]},\n {gen_server,\n call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,\n 119,\n "replication:ns_1@172.23.120.170->ns_1@172.23.121.115:GleamBookUsers0"},\n 300000]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.120.170\'},\n {if_rebalance,\n <0.934.81>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.121.115\'],\n 119}},\n infinity]}}}}'} 2022-02-08 04:01:59,476 | test | ERROR | pool-3-thread-11 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644321711969L, u'shortText': u'message', u'serverTime': u'2022-02-08T04:01:51.969Z', u'text': u"Control connection to memcached on 'ns_1@172.23.120.170' disconnected. Check logs for details."} 2022-02-08 04:01:59,476 | test | ERROR | pool-3-thread-11 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644321481749L, u'shortText': u'message', u'serverTime': u'2022-02-08T03:58:01.749Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2022-02-08 04:01:59,476 | test | ERROR | pool-3-thread-11 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.115', u'tstamp': 1644321468414L, u'shortText': u'message', u'serverTime': u'2022-02-08T03:57:48.414Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.121.115\' in 0 seconds.'} 2022-02-08 04:01:59,476 | test | ERROR | pool-3-thread-11 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.136', u'tstamp': 1644321468311L, u'shortText': u'message', u'serverTime': u'2022-02-08T03:57:48.311Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.121.136\' in 0 seconds.'} 2022-02-08 04:01:59,476 | test | ERROR | pool-3-thread-11 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644321468134L, u'shortText': u'message', u'serverTime': u'2022-02-08T03:57:48.134Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2022-02-08 04:01:59,476 | test | ERROR | pool-3-thread-11 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_storage_conf', u'type': u'info', u'node': u'ns_1@172.23.121.115', u'tstamp': 1644321468055L, u'shortText': u'message', u'serverTime': u'2022-02-08T03:57:48.055Z', u'text': u'Deleting old data files of bucket "GleamBookUsers0"'} 2022-02-08 04:01:59,477 | test | ERROR | pool-3-thread-11 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644321467677L, u'shortText': u'message', u'serverTime': u'2022-02-08T03:57:47.677Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.121.115','ns_1@172.23.121.132',\n 'ns_1@172.23.121.136','ns_1@172.23.121.135',\n 'ns_1@172.23.120.170','ns_1@172.23.121.129',\n 'ns_1@172.23.121.124','ns_1@172.23.121.126',\n 'ns_1@172.23.121.134','ns_1@172.23.121.116',\n 'ns_1@172.23.121.127','ns_1@172.23.121.123',\n 'ns_1@172.23.121.131','ns_1@172.23.121.130',\n 'ns_1@172.23.121.133'], EjectNodes = ['ns_1@172.23.121.128'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = e99352b42373454ddf279886adfb4d4e"} 2022-02-08 04:01:59,477 | test | ERROR | pool-3-thread-11 | [task:call:325] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'a69bd93d976ba37350a3f67a10ae2d23', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=5ca59fc024dad2696fa5fd1995b9ad31', u'status': u'notRunning'} - rebalance failed 2022-02-08 04:02:00,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:02:00,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:00,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:00,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:02:00,802 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 04:02:00,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:02:00,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:00,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:02:00,872 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 04:02:00,872 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:02:00,875 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 04:02:00,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:02:01,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 04:02:01,219 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:02:01,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 04:02:01,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 04:02:01,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 04:02:01,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 04:02:01,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 04:02:01,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 04:02:01,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 04:02:01,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:02:01,326 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 04:02:01,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:02:01,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:01,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:02:01,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:01,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:01,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:02:01,433 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 04:02:01,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:02:01,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:01,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:02:01,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:01,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:02:01,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:01,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:01,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:01,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:01,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:01,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:01,711 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:01,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:01,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:01,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:02:01,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:01,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:01,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:01,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:01,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:01,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:01,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:01,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:01,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:01,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:02:01,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:01,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:01,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:01,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:02,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:02,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:02,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:02,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:02,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:02,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:02:02,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:02,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:02,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:02,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:02,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:02,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:02,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:02,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:02,210 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 04:02:02,213 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:02:02,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:02:03,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:02:03,345 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:02:03,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:02:03,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:02:03,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:02:03,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:02:03,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:02:03,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:02:03,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:02:03,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:02:03,469 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 04:02:03,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:02:03,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:05,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:02:05,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:05,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:05,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:02:05,829 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 04:02:05,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:02:05,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:05,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:02:05,911 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 04:02:05,911 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:02:05,914 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 04:02:05,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:02:06,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 04:02:06,253 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:02:06,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 04:02:06,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 04:02:06,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 04:02:06,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 04:02:06,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 04:02:06,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 04:02:06,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 04:02:06,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:02:06,355 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 04:02:06,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:02:06,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:06,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:02:06,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:06,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:06,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:02:06,459 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 04:02:06,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:02:06,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:06,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:02:06,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:06,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:02:06,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:06,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:06,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:06,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:06,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:06,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:06,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:06,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:06,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:06,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:02:06,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:06,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:06,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:06,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:06,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:06,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:06,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:06,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:06,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:06,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:02:06,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:06,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:07,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:07,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:07,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:07,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:07,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:07,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:07,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:07,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:02:07,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:07,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:07,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:07,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:07,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:07,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:07,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:07,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:07,247 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 04:02:07,250 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:02:07,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:02:08,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:02:08,424 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:02:08,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:02:08,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:02:08,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:02:08,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:02:08,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:02:08,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:02:08,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:02:08,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:02:08,559 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 04:02:08,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:02:08,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:11,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:02:11,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:12,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:12,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:02:12,306 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 04:02:12,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:02:12,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:12,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:02:12,371 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 04:02:12,371 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:02:12,374 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:02:12,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:02:13,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:02:13,289 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:02:13,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:02:13,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:02:13,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:02:13,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:02:13,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:02:13,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:02:13,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:02:13,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:02:13,426 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 04:02:13,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:02:13,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:16,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:02:16,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:16,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:17,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:02:17,096 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 04:02:17,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:02:17,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:17,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:02:17,186 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 04:02:17,186 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:02:17,187 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 04:02:17,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:02:17,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 04:02:17,536 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:02:17,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 04:02:17,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 04:02:17,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 04:02:17,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 04:02:17,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 04:02:17,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 04:02:17,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 04:02:17,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:02:17,641 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 04:02:17,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:02:17,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:17,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:02:17,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:17,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:17,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:02:17,763 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 04:02:17,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:02:17,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:17,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:02:17,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:17,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:02:17,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:17,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:17,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:17,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:17,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:18,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:18,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:18,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:18,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:18,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:02:18,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:18,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:18,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:18,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:18,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:18,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:18,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:18,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:18,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:18,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:02:18,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:18,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:18,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:18,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:18,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:18,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:18,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:18,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:18,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:18,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:02:18,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:18,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:18,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:18,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:18,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:18,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:18,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:18,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:18,561 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 04:02:18,562 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 04:02:18,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:02:18,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 04:02:18,938 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:02:18,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 04:02:18,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 04:02:18,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 04:02:18,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 04:02:18,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 04:02:19,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 04:02:19,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 04:02:19,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:02:19,043 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 04:02:19,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:02:19,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:19,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:02:19,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:19,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:19,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:02:19,142 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 04:02:19,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:02:19,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:19,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:02:19,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:19,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:02:19,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:19,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:19,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:19,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:19,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:19,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:19,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:19,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:19,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:19,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:02:19,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:19,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:19,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:19,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:19,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:19,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:19,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:19,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:19,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:19,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:02:19,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:19,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:19,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:19,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:19,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:19,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:19,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:19,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:19,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:19,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:02:19,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:19,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:19,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:19,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:19,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:19,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:19,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:19,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:19,908 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 04:02:19,911 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:02:20,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:02:20,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:02:20,986 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:02:20,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:02:21,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:02:21,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:02:21,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:02:21,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:02:21,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:02:21,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:02:21,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:02:21,127 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 04:02:21,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:02:21,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:23,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:02:24,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:24,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:24,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:02:24,684 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 04:02:24,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:02:24,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:24,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:02:24,756 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 04:02:24,756 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:02:24,757 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:02:25,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:02:25,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:02:25,528 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:02:25,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:02:25,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:02:25,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:02:25,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:02:25,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:02:25,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:02:25,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:02:25,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:02:25,667 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 04:02:25,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:02:25,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:28,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:02:28,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:28,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:28,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:02:28,579 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 04:02:28,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:02:28,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:28,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:02:28,644 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 04:02:28,644 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:02:28,647 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:02:29,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:02:29,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:02:29,584 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:02:29,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:02:29,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:02:29,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:02:29,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:02:29,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:02:29,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:02:29,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:02:29,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:02:29,704 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 04:02:29,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:02:29,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:31,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:02:32,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:32,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:32,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:02:32,582 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 04:02:32,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:02:32,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:32,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:02:32,647 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 04:02:32,647 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:02:32,648 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:02:33,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:02:33,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:02:33,796 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:02:33,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:02:33,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:02:33,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:02:33,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:02:33,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:02:33,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:02:33,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:02:33,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:02:33,928 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 04:02:33,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:02:33,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:37,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:02:37,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:37,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:38,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:02:38,102 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 04:02:38,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:02:38,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:38,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:02:38,186 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 04:02:38,186 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:02:38,188 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 04:02:38,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:02:38,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 04:02:38,529 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:02:38,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 04:02:38,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 04:02:38,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 04:02:38,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 04:02:38,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 04:02:38,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 04:02:38,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 04:02:38,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:02:38,635 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 04:02:38,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:02:38,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:38,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:02:38,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:38,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:02:38,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:02:38,749 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 04:02:38,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:02:38,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:02:38,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:02:38,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:38,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:02:38,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:38,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:38,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:38,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:38,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:39,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:39,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:39,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:02:39,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:39,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:02:39,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:39,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:39,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:39,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:39,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:39,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:39,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:39,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:02:39,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:39,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:02:39,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:39,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:39,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:39,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:39,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:39,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:39,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:39,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:02:39,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:39,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:02:39,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:39,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:39,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:39,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:39,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:39,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:39,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:39,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:02:39,530 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 04:02:39,538 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L789 2022-02-08 04:02:39,538 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 5 queries submitted, 0 failed, 5763 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:03:39,562 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.120.170:8091 2022-02-08 04:03:39,563 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:03:39,588 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.120.170:8091 is running 2022-02-08 04:03:39,611 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.136:8091 2022-02-08 04:03:39,611 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:03:39,628 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.136:8091 is running 2022-02-08 04:03:39,660 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.126:8091 2022-02-08 04:03:39,660 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:03:39,687 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.126:8091 is running 2022-02-08 04:03:39,711 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.133:8091 2022-02-08 04:03:39,711 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:03:39,749 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.133:8091 is running 2022-02-08 04:03:39,769 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.135:8091 2022-02-08 04:03:39,769 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:03:39,792 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.135:8091 is running 2022-02-08 04:03:39,809 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.134:8091 2022-02-08 04:03:39,809 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:03:39,831 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.134:8091 is running 2022-02-08 04:03:39,855 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.127:8091 2022-02-08 04:03:39,855 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:03:39,871 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.127:8091 is running 2022-02-08 04:03:39,891 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.124:8091 2022-02-08 04:03:39,892 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:03:39,915 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.124:8091 is running 2022-02-08 04:03:39,964 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.130:8091 2022-02-08 04:03:39,964 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:03:39,982 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.130:8091 is running 2022-02-08 04:03:39,999 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.132:8091 2022-02-08 04:03:39,999 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:03:40,016 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.132:8091 is running 2022-02-08 04:03:40,038 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.123:8091 2022-02-08 04:03:40,038 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:03:40,069 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.123:8091 is running 2022-02-08 04:03:40,091 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.115:8091 2022-02-08 04:03:40,091 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:03:40,121 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.115:8091 is running 2022-02-08 04:03:40,151 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.116:8091 2022-02-08 04:03:40,151 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:03:40,168 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.116:8091 is running 2022-02-08 04:03:40,190 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.129:8091 2022-02-08 04:03:40,191 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:03:40,220 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.129:8091 is running 2022-02-08 04:03:40,236 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.131:8091 2022-02-08 04:03:40,236 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:03:40,253 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.131:8091 is running 2022-02-08 04:03:40,256 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:03:41,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:03:41,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:03:41,901 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:03:41,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:03:41,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:03:42,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:03:42,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:03:42,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:03:42,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:03:42,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:03:42,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:03:42,424 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 04:03:42,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:03:42,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:03:43,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:03:43,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:03:44,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:03:44,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:03:44,420 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 04:03:44,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:03:44,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:03:44,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:03:44,608 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 04:03:44,609 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:03:44,615 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:03:44,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:03:44,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:03:45,039 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:03:45,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:03:45,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:03:45,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:03:45,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:03:45,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:03:45,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:03:45,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:03:45,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:03:45,183 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 04:03:45,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:03:45,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:03:45,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:03:45,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:03:45,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:03:45,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:03:45,309 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 04:03:45,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:03:45,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:03:45,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:03:45,359 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 04:03:45,359 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:03:45,361 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:03:45,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:03:46,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:03:46,394 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:03:46,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:03:46,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:03:46,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:03:46,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:03:46,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:03:46,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:03:46,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:03:46,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:03:46,799 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 04:03:46,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:03:46,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:03:47,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:03:47,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:03:48,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:03:48,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:03:48,552 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 04:03:48,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:03:48,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:03:48,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:03:48,811 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 04:03:48,811 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:03:48,812 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 04:03:48,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:03:49,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 04:03:49,224 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:03:49,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 04:03:49,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 5 queries submitted, 0 failed, 5787 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:03:49,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 04:03:49,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 04:03:49,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 04:03:49,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 04:03:49,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 04:03:49,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:03:49,368 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 04:03:49,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:03:49,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:03:49,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:03:49,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:03:49,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:03:49,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:03:49,517 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 04:03:49,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:03:49,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:03:49,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:03:49,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:49,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:03:49,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:49,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:49,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:49,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:49,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:49,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:49,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:49,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:49,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:49,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:03:49,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:49,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:50,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:50,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:50,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:50,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:50,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:50,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:50,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:50,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:03:50,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:50,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:50,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:50,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:50,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:50,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:50,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:50,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:50,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:50,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:03:50,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:50,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:50,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:50,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:50,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:50,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:50,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:50,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:50,486 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 04:03:50,489 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:03:51,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:03:51,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:03:51,898 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:03:51,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:03:51,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:03:51,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:03:52,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:03:52,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:03:52,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:03:52,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:03:52,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:03:52,180 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 04:03:52,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:03:52,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:03:52,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:03:52,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:03:53,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:03:53,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:03:53,308 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 04:03:53,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:03:53,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:03:53,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:03:53,407 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 04:03:53,407 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:03:53,408 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 04:03:53,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:03:53,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 04:03:53,832 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:03:53,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 04:03:53,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 04:03:53,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 04:03:53,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 04:03:53,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 04:03:53,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 04:03:53,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 04:03:53,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:03:53,980 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 04:03:54,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:03:54,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:03:54,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:03:54,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:03:54,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:03:54,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:03:54,114 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 04:03:54,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:03:54,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:03:54,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:03:54,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:54,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:03:54,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:54,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:54,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:54,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:54,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:54,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:54,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:54,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:03:54,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:54,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:03:54,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:54,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:54,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:54,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:54,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:54,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:54,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:54,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:03:54,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:54,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:03:54,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:54,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:54,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:54,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:54,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:54,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:54,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:54,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:03:54,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:54,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:03:54,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:54,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:54,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:54,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:54,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:54,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:55,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:55,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:03:55,052 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 04:03:55,053 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:03:55,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:03:56,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:03:56,796 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:03:56,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:03:56,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:03:56,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:03:56,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:03:56,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:03:57,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:03:57,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:03:57,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:03:57,138 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 04:03:57,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:03:57,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:03:57,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:03:58,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:03:58,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:03:58,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:03:58,970 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 04:03:58,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:03:59,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:03:59,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:03:59,114 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 04:03:59,115 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:03:59,117 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:04:00,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:04:00,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:04:00,917 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:04:00,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:04:00,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:04:01,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:04:01,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:04:01,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:04:01,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:04:01,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:04:01,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:04:01,433 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 04:04:01,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:04:01,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:04:01,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:04:02,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:04:02,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:04:03,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:04:03,046 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 04:04:03,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:04:03,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:04:03,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:04:03,269 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 04:04:03,269 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:04:03,272 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 04:04:03,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:04:03,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 04:04:03,831 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:04:03,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 04:04:03,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 04:04:03,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 04:04:03,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 04:04:03,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 04:04:03,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 04:04:03,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 04:04:03,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:04:04,019 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 04:04:04,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:04:04,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:04:04,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:04:04,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:04:04,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:04:04,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:04:04,196 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 04:04:04,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:04:04,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:04:04,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:04:04,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:04,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:04:04,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:04,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:04,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:04,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:04,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:04,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:04,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:04,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:04,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:04,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:04:04,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:04,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:04,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:04,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:04,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:04,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:04,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:04,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:05,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:05,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:04:05,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:05,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:05,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:05,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:05,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:05,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:05,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:05,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:05,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:05,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:04:05,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:05,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:05,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:05,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:05,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:05,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:05,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:05,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:05,572 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 04:04:05,575 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 04:04:05,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:04:05,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 04:04:05,973 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:04:05,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 04:04:05,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 04:04:06,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 04:04:06,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 04:04:06,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 04:04:06,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 04:04:06,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 04:04:06,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:04:06,118 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 04:04:06,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:04:06,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:04:06,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:04:06,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:04:06,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:04:06,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:04:06,240 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 04:04:06,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:04:06,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:04:06,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:04:06,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:06,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:04:06,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:06,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:06,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:06,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:06,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:06,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:06,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:06,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:06,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:06,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:04:06,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:06,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:06,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:06,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:06,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:06,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:06,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:06,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:06,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:06,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:04:06,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:06,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:06,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:06,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:06,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:06,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:06,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:07,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:07,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:07,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:04:07,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:07,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:07,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:07,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:07,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:07,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:07,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:07,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:07,206 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 04:04:07,207 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:04:08,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:04:08,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:04:09,078 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:04:09,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:04:09,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:04:09,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:04:09,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:04:09,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:04:09,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:04:09,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:04:09,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:04:09,414 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 04:04:09,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:04:09,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:04:09,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:04:10,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:04:10,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:04:11,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:04:11,104 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 04:04:11,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:04:11,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:04:11,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:04:11,308 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 04:04:11,308 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:04:11,311 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:04:11,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:04:11,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:04:11,865 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:04:11,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:04:11,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:04:11,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:04:11,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:04:12,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:04:12,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:04:12,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:04:12,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:04:12,138 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 04:04:12,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:04:12,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:04:12,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:04:12,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:04:13,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:04:13,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:04:13,299 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 04:04:13,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:04:13,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:04:13,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:04:13,381 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 04:04:13,381 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:04:13,384 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:04:14,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:04:14,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:04:14,632 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:04:14,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:04:14,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:04:14,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:04:14,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:04:14,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:04:14,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:04:14,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:04:14,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:04:14,930 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 04:04:14,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:04:15,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:04:15,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:04:15,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:04:16,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:04:16,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:04:16,230 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 04:04:16,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:04:16,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:04:16,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:04:16,371 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 04:04:16,371 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:04:16,374 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:04:17,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:04:17,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:04:18,178 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:04:18,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:04:18,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:04:18,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:04:18,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:04:18,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:04:18,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:04:18,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:04:18,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:04:18,516 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 04:04:18,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:04:18,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:04:19,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:04:19,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:04:20,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:04:20,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:04:20,461 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 04:04:20,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:04:20,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:04:20,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:04:20,621 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 04:04:20,621 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:04:20,624 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 04:04:20,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:04:20,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 04:04:21,017 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:04:21,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 04:04:21,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 04:04:21,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 04:04:21,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 04:04:21,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 04:04:21,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 04:04:21,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 04:04:21,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:04:21,141 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 04:04:21,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:04:21,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:04:21,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:04:21,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:04:21,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:04:21,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:04:21,259 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 04:04:21,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:04:21,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:04:21,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:04:21,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:21,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:04:21,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:21,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:21,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:21,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:21,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:21,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:21,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:21,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:04:21,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:21,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:04:21,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:21,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:21,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:21,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:21,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:21,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:21,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:21,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:04:21,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:21,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:04:21,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:21,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:21,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:21,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:21,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:21,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:21,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:22,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:04:22,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:22,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:04:22,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:22,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:22,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:22,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:22,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:22,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:22,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:22,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:04:22,227 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 04:04:22,229 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[172.23.121.115,172.23.121.136]_OUT=[172.23.121.128]_1644321458.57 2022-02-08 04:04:22,229 | test | INFO | MainThread | [opd:abort_rebalance:808] Restarting Rebalance after killing at 20 2022-02-08 04:04:22,332 | infra | DEBUG | MainThread | [task:__init__:249] Last known rebalance status_id: a69bd93d976ba37350a3f67a10ae2d23 2022-02-08 04:04:22,332 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.121.128]_1644321862.23 2022-02-08 04:04:22,335 | infra | INFO | pool-3-thread-14 | [task:start_task:91] Thread 'Rebalance_task_IN=[]_OUT=[172.23.121.128]_1644321862.23' started 2022-02-08 04:04:22,344 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L813 2022-02-08 04:04:22,344 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2022-02-08 04:04:22,413 | test | DEBUG | pool-3-thread-14 | [task:start_rebalance:384] Removing node 172.23.121.128:8091 from cluster 2022-02-08 04:04:22,413 | test | DEBUG | pool-3-thread-14 | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.128,ns_1@172.23.121.115,ns_1@172.23.121.132,ns_1@172.23.121.136,ns_1@172.23.121.135,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.134,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123,ns_1@172.23.121.131,ns_1@172.23.121.130,ns_1@172.23.121.133', 'ejectedNodes': u'ns_1@172.23.121.128', 'user': 'Administrator'} 2022-02-08 04:04:22,602 | test | DEBUG | pool-3-thread-14 | [rest_client:rebalance:1459] Rebalance operation started 2022-02-08 04:04:22,615 | test | INFO | pool-3-thread-14 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.121.128 | kv | 7.1.0-2255-enterprise | 86.2485688844 | --- OUT ---> | | 172.23.121.115 | kv | 7.1.0-2255-enterprise | 84.2105263158 | Cluster node | | 172.23.121.132 | index, n1ql | 7.1.0-2255-enterprise | 14.7032082089 | Cluster node | | 172.23.121.136 | kv | 7.1.0-2255-enterprise | 10.9222661397 | Cluster node | | 172.23.121.135 | kv | 7.1.0-2255-enterprise | 87.4952247549 | Cluster node | | 172.23.120.170 | kv | 7.1.0-2255-enterprise | 88.204606184 | Cluster node | | 172.23.121.129 | kv | 7.1.0-2255-enterprise | 88.4253370644 | Cluster node | | 172.23.121.124 | kv | 7.1.0-2255-enterprise | 88.476686571 | Cluster node | | 172.23.121.126 | kv | 7.1.0-2255-enterprise | 88.4322678843 | Cluster node | | 172.23.121.134 | index, n1ql | 7.1.0-2255-enterprise | 19.928232731 | Cluster node | | 172.23.121.116 | kv | 7.1.0-2255-enterprise | 88.6219171116 | Cluster node | | 172.23.121.127 | kv | 7.1.0-2255-enterprise | 88.5166856998 | Cluster node | | 172.23.121.123 | kv | 7.1.0-2255-enterprise | 90.5088186778 | Cluster node | | 172.23.121.131 | index, n1ql | 7.1.0-2255-enterprise | 14.6778350515 | Cluster node | | 172.23.121.130 | index, n1ql | 7.1.0-2255-enterprise | 31.8674232654 | Cluster node | | 172.23.121.133 | index, n1ql | 7.1.0-2255-enterprise | 12.3168676261 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2022-02-08 04:04:22,628 | infra | DEBUG | pool-3-thread-14 | [task:call:316] New rebalance status_id: 44156d6c660340b3fdfc735a21d8b096 2022-02-08 04:04:22,644 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 04:04:22,644 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 04:04:22,703 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:04:22,706 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:04:22,707 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:04:32,744 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 04:04:32,744 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 04:04:32,792 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:04:32,795 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:04:32,795 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:04:42,813 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.84 % 2022-02-08 04:04:42,813 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 0.84 2022-02-08 04:04:42,871 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:04:42,874 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:04:42,875 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 5812 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:04:52,904 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.84 % 2022-02-08 04:04:52,904 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 0.84 2022-02-08 04:04:52,946 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:04:52,948 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:04:52,948 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:05:02,986 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.41 % 2022-02-08 04:05:02,986 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 1.41 2022-02-08 04:05:03,029 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:05:03,030 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:05:03,030 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:05:13,061 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.83 % 2022-02-08 04:05:13,061 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 1.83 2022-02-08 04:05:13,119 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:05:13,122 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:05:13,124 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:05:23,145 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.36 % 2022-02-08 04:05:23,145 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 2.36 2022-02-08 04:05:23,197 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:05:23,200 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:05:23,200 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:05:33,230 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.67 % 2022-02-08 04:05:33,230 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 2.67 2022-02-08 04:05:33,272 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:05:33,273 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:05:33,275 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:05:43,293 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.70 % 2022-02-08 04:05:43,293 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 3.7 2022-02-08 04:05:43,346 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:05:43,349 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:05:43,349 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 5838 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:05:53,374 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.43 % 2022-02-08 04:05:53,374 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 4.43 2022-02-08 04:05:53,431 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:05:53,434 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:05:53,434 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:06:03,461 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.43 % 2022-02-08 04:06:03,463 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 4.43 2022-02-08 04:06:03,506 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:06:03,509 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:06:03,509 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:06:13,523 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.70 % 2022-02-08 04:06:13,538 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 4.7 2022-02-08 04:06:13,585 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:06:13,588 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:06:13,588 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:06:22,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.40 % 2022-02-08 04:06:22,369 | test | INFO | MainThread | [opd:abort_rebalance:814] Rebalance % = 5.4 2022-02-08 04:06:22,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.40 % 2022-02-08 04:06:22,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:06:22,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:06:23,631 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.40 % 2022-02-08 04:06:23,632 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 5.4 2022-02-08 04:06:23,700 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:06:23,703 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:06:23,703 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:06:32,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.20 % 2022-02-08 04:06:32,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:06:32,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:06:33,739 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.46 % 2022-02-08 04:06:33,740 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 6.46 2022-02-08 04:06:33,786 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:06:33,789 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:06:33,789 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:06:42,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.46 % 2022-02-08 04:06:42,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:06:42,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:06:43,802 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.46 % 2022-02-08 04:06:43,803 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 6.46 2022-02-08 04:06:43,842 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:06:43,845 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:06:43,845 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 5864 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:06:52,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.04 % 2022-02-08 04:06:52,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:06:52,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:06:53,878 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.04 % 2022-02-08 04:06:53,878 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 7.04 2022-02-08 04:06:53,940 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:06:53,944 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:06:53,944 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:07:02,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.33 % 2022-02-08 04:07:02,540 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:07:02,540 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:07:03,959 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.33 % 2022-02-08 04:07:03,959 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 7.33 2022-02-08 04:07:04,005 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:07:04,052 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:07:04,052 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:07:12,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.72 % 2022-02-08 04:07:12,573 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:07:12,573 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:07:14,068 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.72 % 2022-02-08 04:07:14,069 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 7.72 2022-02-08 04:07:14,114 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:07:14,117 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:07:14,117 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:07:22,594 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.29 % 2022-02-08 04:07:22,602 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:07:22,602 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:07:24,148 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.29 % 2022-02-08 04:07:24,150 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 8.29 2022-02-08 04:07:24,197 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:07:24,200 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:07:24,200 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:07:32,627 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.56 % 2022-02-08 04:07:32,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:07:32,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:07:34,217 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.56 % 2022-02-08 04:07:34,217 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 8.56 2022-02-08 04:07:34,260 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:07:34,263 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:07:34,263 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:07:42,655 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.56 % 2022-02-08 04:07:42,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:07:42,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:07:44,282 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.56 % 2022-02-08 04:07:44,282 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 8.56 2022-02-08 04:07:44,328 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:07:44,331 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:07:44,331 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 5888 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:07:52,683 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.48 % 2022-02-08 04:07:52,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:07:52,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:07:54,352 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.48 % 2022-02-08 04:07:54,352 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 9.48 2022-02-08 04:07:54,401 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:07:54,404 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:07:54,404 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:08:02,717 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.75 % 2022-02-08 04:08:02,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:08:02,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:08:04,421 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.75 % 2022-02-08 04:08:04,421 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 9.75 2022-02-08 04:08:04,466 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:08:04,469 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:08:04,469 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:08:12,740 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.11 % 2022-02-08 04:08:12,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:08:12,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:08:14,486 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.37 % 2022-02-08 04:08:14,486 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 10.37 2022-02-08 04:08:14,517 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:08:14,519 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:08:14,520 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:08:22,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.74 % 2022-02-08 04:08:22,770 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:08:22,772 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:08:24,539 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.74 % 2022-02-08 04:08:24,539 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 10.74 2022-02-08 04:08:24,585 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:08:24,588 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:08:24,588 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:08:32,796 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.01 % 2022-02-08 04:08:32,803 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:08:32,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:08:34,605 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.27 % 2022-02-08 04:08:34,605 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 11.27 2022-02-08 04:08:34,650 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:08:34,651 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:08:34,651 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:08:42,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.59 % 2022-02-08 04:08:42,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:08:42,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:08:44,665 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.59 % 2022-02-08 04:08:44,665 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 11.59 2022-02-08 04:08:44,700 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:08:44,701 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:08:44,701 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 5914 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:08:52,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.86 % 2022-02-08 04:08:52,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:08:52,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:08:54,717 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.86 % 2022-02-08 04:08:54,719 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 11.86 2022-02-08 04:08:54,766 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:08:54,769 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:08:54,769 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:09:02,874 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.86 % 2022-02-08 04:09:02,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:09:02,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:09:04,792 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.86 % 2022-02-08 04:09:04,792 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 11.86 2022-02-08 04:09:04,829 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:09:04,832 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:09:04,832 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:09:12,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.41 % 2022-02-08 04:09:12,910 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:09:12,910 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:09:14,890 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.41 % 2022-02-08 04:09:14,890 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 12.41 2022-02-08 04:09:14,948 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:09:14,953 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:09:14,953 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:09:22,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.73 % 2022-02-08 04:09:22,928 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:09:22,928 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:09:24,970 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.73 % 2022-02-08 04:09:24,970 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 12.73 2022-02-08 04:09:25,010 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:09:25,013 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:09:25,013 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:09:32,948 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.66 % 2022-02-08 04:09:33,045 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:09:33,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:09:35,026 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.66 % 2022-02-08 04:09:35,028 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 13.66 2022-02-08 04:09:35,125 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:09:35,128 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:09:35,128 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:09:43,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.16 % 2022-02-08 04:09:43,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:09:43,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:09:45,142 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.16 % 2022-02-08 04:09:45,142 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 14.16 2022-02-08 04:09:45,188 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:09:45,194 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:09:45,194 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 5938 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:09:53,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.75 % 2022-02-08 04:09:53,181 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:09:53,181 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:09:55,213 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.75 % 2022-02-08 04:09:55,213 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 14.75 2022-02-08 04:09:55,266 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:09:55,269 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:09:55,269 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:10:03,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.14 % 2022-02-08 04:10:03,211 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:10:03,213 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:10:05,293 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.43 % 2022-02-08 04:10:05,293 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 15.43 2022-02-08 04:10:05,335 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:10:05,338 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:10:05,338 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:10:13,232 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.95 % 2022-02-08 04:10:13,242 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:10:13,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:10:15,364 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.21 % 2022-02-08 04:10:15,364 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 16.21 2022-02-08 04:10:15,414 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:10:15,417 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:10:15,417 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:10:23,270 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.21 % 2022-02-08 04:10:23,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:10:23,280 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:10:25,434 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.60 % 2022-02-08 04:10:25,434 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 16.6 2022-02-08 04:10:25,469 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:10:25,471 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:10:25,471 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:10:33,308 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.73 % 2022-02-08 04:10:33,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:10:33,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:10:35,492 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.00 % 2022-02-08 04:10:35,492 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 17.0 2022-02-08 04:10:35,538 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:10:35,542 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:10:35,542 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:10:43,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.50 % 2022-02-08 04:10:43,351 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:10:43,352 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:10:45,562 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.50 % 2022-02-08 04:10:45,562 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 17.5 2022-02-08 04:10:45,618 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:10:45,621 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:10:45,621 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 5964 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:10:53,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.84 % 2022-02-08 04:10:53,394 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:10:53,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:10:55,645 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.08 % 2022-02-08 04:10:55,645 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 19.08 2022-02-08 04:10:55,684 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:10:55,687 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:10:55,687 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:11:03,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.34 % 2022-02-08 04:11:03,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:11:03,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:11:05,701 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.47 % 2022-02-08 04:11:05,701 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 19.47 2022-02-08 04:11:05,736 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:11:05,740 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:11:05,740 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:11:13,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.59 % 2022-02-08 04:11:13,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:11:13,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:11:15,755 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.83 % 2022-02-08 04:11:15,756 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 19.83 2022-02-08 04:11:15,792 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:11:15,795 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:11:15,795 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:11:23,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.39 % 2022-02-08 04:11:23,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:11:23,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:11:25,834 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.39 % 2022-02-08 04:11:25,835 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 20.39 2022-02-08 04:11:25,891 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:11:25,894 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:11:25,894 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:11:33,494 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.44 % 2022-02-08 04:11:33,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:11:33,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:11:35,913 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.68 % 2022-02-08 04:11:35,913 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 21.68 2022-02-08 04:11:35,957 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:11:35,960 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:11:35,960 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:11:43,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.68 % 2022-02-08 04:11:43,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:11:43,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:11:45,974 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.68 % 2022-02-08 04:11:45,976 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 21.68 2022-02-08 04:11:46,019 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:11:46,022 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:11:46,022 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 5990 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:11:53,551 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.07 % 2022-02-08 04:11:53,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:11:53,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:11:56,038 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.31 % 2022-02-08 04:11:56,038 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 22.31 2022-02-08 04:11:56,078 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:11:56,082 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:11:56,082 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:12:03,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.50 % 2022-02-08 04:12:03,579 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:12:03,579 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:12:06,101 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.50 % 2022-02-08 04:12:06,101 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 23.5 2022-02-08 04:12:06,134 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:12:06,137 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:12:06,137 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:12:13,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 24.01 % 2022-02-08 04:12:13,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:12:13,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:12:16,150 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 24.58 % 2022-02-08 04:12:16,150 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 24.58 2022-02-08 04:12:16,183 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:12:16,186 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:12:16,186 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:12:23,612 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.15 % 2022-02-08 04:12:23,619 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:12:23,619 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:12:26,210 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.15 % 2022-02-08 04:12:26,210 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 25.15 2022-02-08 04:12:26,280 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:12:26,283 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:12:26,283 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:12:33,635 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.52 % 2022-02-08 04:12:33,642 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:12:33,642 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:12:36,296 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.89 % 2022-02-08 04:12:36,298 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 25.89 2022-02-08 04:12:36,328 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:12:36,332 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:12:36,332 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:12:43,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.97 % 2022-02-08 04:12:43,670 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:12:43,670 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:12:46,346 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.97 % 2022-02-08 04:12:46,348 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 26.97 2022-02-08 04:12:46,391 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:12:46,394 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:12:46,394 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 6014 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:12:53,686 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.97 % 2022-02-08 04:12:53,693 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:12:53,693 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:12:56,414 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.97 % 2022-02-08 04:12:56,415 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 26.97 2022-02-08 04:12:56,460 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:12:56,463 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:12:56,463 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:13:03,704 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.84 % 2022-02-08 04:13:03,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:13:03,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:13:06,489 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.08 % 2022-02-08 04:13:06,489 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 28.08 2022-02-08 04:13:06,529 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:13:06,532 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:13:06,532 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:13:13,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.59 % 2022-02-08 04:13:13,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:13:13,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:13:16,546 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.59 % 2022-02-08 04:13:16,548 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 28.59 2022-02-08 04:13:16,592 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:13:16,595 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:13:16,595 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:13:23,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.59 % 2022-02-08 04:13:23,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:13:23,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:13:26,618 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.59 % 2022-02-08 04:13:26,618 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 28.59 2022-02-08 04:13:26,661 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:13:26,664 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:13:26,664 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:13:33,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.34 % 2022-02-08 04:13:33,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:13:33,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:13:36,683 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.34 % 2022-02-08 04:13:36,684 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 29.34 2022-02-08 04:13:36,717 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:13:36,720 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:13:36,720 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:13:43,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.71 % 2022-02-08 04:13:43,818 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:13:43,818 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:13:46,733 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.71 % 2022-02-08 04:13:46,733 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 29.71 2022-02-08 04:13:46,767 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:13:46,769 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:13:46,770 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 6036 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:13:53,834 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.27 % 2022-02-08 04:13:53,841 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:13:53,841 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:13:56,782 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.51 % 2022-02-08 04:13:56,782 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 30.51 2022-02-08 04:13:56,813 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:13:56,816 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:13:56,816 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:14:03,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 31.09 % 2022-02-08 04:14:03,869 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:14:03,869 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:14:06,829 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 31.58 % 2022-02-08 04:14:06,829 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 31.58 2022-02-08 04:14:06,859 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:14:06,861 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:14:06,861 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:14:13,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.98 % 2022-02-08 04:14:13,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:14:13,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:14:16,875 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.98 % 2022-02-08 04:14:16,875 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 33.98 2022-02-08 04:14:16,915 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:14:16,917 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:14:16,918 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:14:23,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.98 % 2022-02-08 04:14:23,928 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:14:23,928 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:14:26,937 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.22 % 2022-02-08 04:14:26,937 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 34.22 2022-02-08 04:14:26,996 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:14:27,000 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:14:27,000 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:14:33,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.79 % 2022-02-08 04:14:33,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:14:33,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:14:37,059 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.11 % 2022-02-08 04:14:37,059 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 35.11 2022-02-08 04:14:37,108 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:14:37,111 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:14:37,111 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:14:43,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.04 % 2022-02-08 04:14:43,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:14:43,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:14:47,131 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.04 % 2022-02-08 04:14:47,132 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 36.04 2022-02-08 04:14:47,180 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:14:47,183 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:14:47,184 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 6061 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:14:54,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.29 % 2022-02-08 04:14:54,010 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:14:54,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:14:57,200 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.66 % 2022-02-08 04:14:57,200 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 36.66 2022-02-08 04:14:57,242 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:14:57,244 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:14:57,244 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:15:04,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.36 % 2022-02-08 04:15:04,030 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:15:04,030 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:15:07,260 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.36 % 2022-02-08 04:15:07,262 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 37.36 2022-02-08 04:15:07,302 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:15:07,305 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:15:07,305 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:15:14,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.68 % 2022-02-08 04:15:14,053 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:15:14,053 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:15:17,319 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.36 % 2022-02-08 04:15:17,319 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 38.36 2022-02-08 04:15:17,355 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:15:17,358 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:15:17,358 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:15:24,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.36 % 2022-02-08 04:15:24,076 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:15:24,076 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:15:27,375 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.62 % 2022-02-08 04:15:27,375 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 38.62 2022-02-08 04:15:27,424 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:15:27,427 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:15:27,427 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:15:34,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.62 % 2022-02-08 04:15:34,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:15:34,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:15:37,440 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.20 % 2022-02-08 04:15:37,440 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 39.2 2022-02-08 04:15:37,476 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:15:37,480 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:15:37,480 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:15:44,132 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.71 % 2022-02-08 04:15:44,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:15:44,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:15:47,500 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.71 % 2022-02-08 04:15:47,500 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 39.71 2022-02-08 04:15:47,552 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:15:47,555 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:15:47,555 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 6085 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:15:54,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.03 % 2022-02-08 04:15:54,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:15:54,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:15:57,575 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.03 % 2022-02-08 04:15:57,575 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 40.03 2022-02-08 04:15:57,625 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:15:57,628 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:15:57,628 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:16:04,165 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >40.03% in 581.796000004 seconds 2022-02-08 04:16:04,187 | test | DEBUG | MainThread | [rest_client:diag_eval:1476] /diag/eval status on 172.23.120.170:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2022-02-08 04:16:04,187 | test | INFO | MainThread | [opd:abort_rebalance:780] Abort rebalance 2022-02-08 04:16:04,203 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:16:05,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:16:05,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:16:05,746 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:16:05,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:16:05,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:16:05,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:16:05,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:16:05,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:16:05,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:16:05,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:16:05,895 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:16:05,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:16:05,953 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['96659 memcached\n'], []) 2022-02-08 04:16:05,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 04:16:05,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:16:06,017 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['96659 memcached\n'], []) 2022-02-08 04:16:06,017 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:16:06,020 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:16:06,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:16:06,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:16:06,415 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:16:06,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:16:06,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:16:06,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:16:06,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:16:06,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:16:06,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:16:06,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:16:06,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:16:06,529 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 04:16:06,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:16:06,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:07,644 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.40 % 2022-02-08 04:16:07,644 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 40.4 2022-02-08 04:16:07,678 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:16:07,681 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:16:07,681 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:16:09,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:16:09,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:09,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:10,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:16:10,181 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 04:16:10,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:16:10,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:10,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:16:10,257 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 04:16:10,257 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:16:10,259 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:16:10,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:16:10,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:16:11,056 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:16:11,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:16:11,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:16:11,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:16:11,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:16:11,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:16:11,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:16:11,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:16:11,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:16:11,197 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 04:16:11,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:16:11,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:11,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:16:11,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:11,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:11,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:16:11,322 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 04:16:11,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:16:11,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:11,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:16:11,371 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 04:16:11,371 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:16:11,372 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:16:12,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:16:12,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:16:12,911 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:16:12,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:16:12,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:16:12,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:16:12,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:16:12,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:16:12,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:16:13,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:16:13,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:16:13,043 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 04:16:13,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:16:13,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:16,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:16:16,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:16,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:17,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:16:17,183 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 04:16:17,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:16:17,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:17,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:16:17,266 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 04:16:17,266 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:16:17,269 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 04:16:17,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:16:17,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 04:16:17,607 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:16:17,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 04:16:17,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 04:16:17,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 04:16:17,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 04:16:17,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 04:16:17,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 04:16:17,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 04:16:17,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:16:17,694 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 41.79 % 2022-02-08 04:16:17,694 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 41.79 2022-02-08 04:16:17,707 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 04:16:17,723 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:16:17,726 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:16:17,726 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:16:17,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:16:17,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:17,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:16:17,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:17,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:17,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:16:17,806 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 04:16:17,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:16:17,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:17,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:16:17,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:17,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:16:17,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:17,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:17,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:18,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:18,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:18,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:18,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:18,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:18,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:18,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:16:18,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:18,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:18,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:18,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:18,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:18,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:18,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:18,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:18,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:18,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:16:18,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:18,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:18,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:18,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:18,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:18,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:18,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:18,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:18,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:18,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:16:18,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:18,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:18,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:18,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:18,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:18,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:18,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:18,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:18,605 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 04:16:18,608 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:16:18,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:16:19,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:16:19,512 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:16:19,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:16:19,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:16:19,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:16:19,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:16:19,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:16:19,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:16:19,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:16:19,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:16:19,631 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 04:16:19,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:16:19,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:21,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:16:21,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:21,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:21,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:16:21,924 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 04:16:21,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:16:21,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:21,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:16:22,003 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 04:16:22,003 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:16:22,005 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 04:16:22,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:16:22,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 04:16:22,345 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:16:22,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 04:16:22,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 04:16:22,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 04:16:22,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 04:16:22,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 04:16:22,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 04:16:22,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 04:16:22,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:16:22,460 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 04:16:22,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:16:22,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:22,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:16:22,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:22,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:22,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:16:22,569 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 04:16:22,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:16:22,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:22,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:16:22,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:22,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:16:22,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:22,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:22,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:22,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:22,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:22,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:22,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:22,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:22,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:22,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:16:22,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:22,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:22,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:22,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:23,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:23,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:23,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:23,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:23,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:23,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:16:23,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:23,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:23,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:23,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:23,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:23,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:23,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:23,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:23,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:23,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:16:23,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:23,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:23,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:23,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:23,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:23,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:23,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:23,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:23,349 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 04:16:23,351 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:16:24,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:16:24,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:16:24,898 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:16:24,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:16:24,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:16:24,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:16:24,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:16:24,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:16:24,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:16:24,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:16:25,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:16:25,026 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 04:16:25,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:16:25,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:27,742 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 41.91 % 2022-02-08 04:16:27,742 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 41.91 2022-02-08 04:16:27,773 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:16:27,776 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:16:27,776 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:16:28,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:16:28,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:29,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:29,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:16:29,296 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 04:16:29,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:16:29,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:29,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:16:29,361 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 04:16:29,362 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:16:29,365 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:16:29,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:16:30,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:16:30,594 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:16:30,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:16:30,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:16:30,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:16:30,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:16:30,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:16:30,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:16:30,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:16:30,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:16:30,762 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 04:16:30,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:16:30,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:34,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:16:34,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:34,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:34,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:16:34,878 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 04:16:34,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:16:34,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:34,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:16:34,957 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 04:16:34,957 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:16:34,960 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 04:16:35,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:16:35,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 04:16:35,321 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:16:35,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 04:16:35,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 04:16:35,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 04:16:35,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 04:16:35,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 04:16:35,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 04:16:35,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 04:16:35,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:16:35,434 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 04:16:35,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:16:35,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:35,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:16:35,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:35,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:35,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:16:35,539 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 04:16:35,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:16:35,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:35,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:16:35,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:35,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:16:35,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:35,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:35,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:35,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:35,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:35,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:35,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:35,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:35,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:35,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:16:35,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:35,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:35,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:35,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:35,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:36,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:36,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:36,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:36,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:36,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:16:36,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:36,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:36,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:36,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:36,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:36,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:36,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:36,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:36,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:36,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:16:36,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:36,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:36,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:36,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:36,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:36,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:36,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:36,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:36,345 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 04:16:36,348 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 04:16:36,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:16:36,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 04:16:36,703 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:16:36,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 04:16:36,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 04:16:36,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 04:16:36,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 04:16:36,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 04:16:36,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 04:16:36,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 04:16:36,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:16:36,815 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 04:16:36,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:16:36,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:36,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:16:36,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:36,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:36,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:16:36,921 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 04:16:36,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:16:36,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:36,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:16:36,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:37,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:16:37,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:37,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:37,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:37,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:37,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:37,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:37,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:37,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:37,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:37,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:16:37,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:37,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:37,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:37,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:37,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:37,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:37,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:37,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:37,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:37,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:16:37,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:37,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:37,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:37,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:37,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:37,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:37,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:37,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:37,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:37,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:16:37,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:37,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:37,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:37,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:37,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:37,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:37,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:37,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:37,746 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 04:16:37,750 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:16:37,790 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 43.11 % 2022-02-08 04:16:37,790 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 43.11 2022-02-08 04:16:37,826 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:16:37,828 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:16:37,829 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:16:38,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:16:38,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:16:39,105 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:16:39,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:16:39,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:16:39,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:16:39,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:16:39,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:16:39,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:16:39,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:16:39,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:16:39,252 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 04:16:39,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:16:39,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:42,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:16:42,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:42,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:43,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:16:43,140 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 04:16:43,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:16:43,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:43,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:16:43,213 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 04:16:43,213 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:16:43,216 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:16:43,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:16:44,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:16:44,147 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:16:44,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:16:44,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:16:44,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:16:44,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:16:44,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:16:44,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:16:44,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:16:44,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:16:44,290 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 04:16:44,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:16:44,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:44,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:16:44,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:44,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:45,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:16:45,145 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 04:16:45,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:16:45,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:45,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:16:45,187 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 04:16:45,187 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:16:45,191 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:16:45,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:16:46,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:16:46,440 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:16:46,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:16:46,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:16:46,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:16:46,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:16:46,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:16:46,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:16:46,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:16:46,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:16:46,565 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 04:16:46,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:16:46,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:47,842 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 43.87 % 2022-02-08 04:16:47,844 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 43.87 2022-02-08 04:16:47,872 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:16:47,875 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:16:47,875 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:16:48,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:16:49,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 6106 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:16:49,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:49,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:16:49,440 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 04:16:49,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:16:49,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:49,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:16:49,497 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 04:16:49,499 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:16:49,500 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:16:50,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:16:50,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:16:50,944 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:16:50,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:16:50,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:16:50,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:16:50,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:16:51,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:16:51,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:16:51,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:16:51,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:16:51,081 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 04:16:51,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:16:51,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:54,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:16:55,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:55,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:55,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:16:55,654 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 04:16:55,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:16:55,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:55,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:16:55,717 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 04:16:55,719 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:16:55,720 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 04:16:55,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:16:56,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 04:16:56,089 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:16:56,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 04:16:56,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 04:16:56,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 04:16:56,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 04:16:56,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 04:16:56,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 04:16:56,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 04:16:56,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:16:56,210 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 04:16:56,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:16:56,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:56,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:16:56,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:56,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:16:56,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:16:56,328 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 04:16:56,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:16:56,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:16:56,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:16:56,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:56,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:16:56,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:56,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:56,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:56,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:56,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:56,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:56,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:56,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:16:56,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:56,711 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:16:56,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:56,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:56,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:56,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:56,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:56,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:56,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:56,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:16:56,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:56,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:16:56,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:56,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:56,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:56,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:56,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:56,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:56,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:56,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:16:56,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:56,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:16:57,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:57,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:57,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:57,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:57,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:57,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:57,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:57,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:16:57,128 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 04:16:57,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L789 2022-02-08 04:16:57,134 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2022-02-08 04:16:57,885 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.76 % 2022-02-08 04:16:57,887 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 44.76 2022-02-08 04:16:57,917 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:16:57,921 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:16:57,921 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:17:07,931 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.64 % 2022-02-08 04:17:07,931 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 45.64 2022-02-08 04:17:07,964 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:17:07,969 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:17:07,969 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:17:17,984 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 46.60 % 2022-02-08 04:17:17,984 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 46.6 2022-02-08 04:17:18,015 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:17:18,016 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:17:18,017 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:17:28,030 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.55 % 2022-02-08 04:17:28,032 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 47.55 2022-02-08 04:17:28,065 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:17:28,068 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:17:28,069 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:17:38,088 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.24 % 2022-02-08 04:17:38,088 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 49.24 2022-02-08 04:17:38,119 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:17:38,122 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:17:38,122 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:17:48,134 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.65 % 2022-02-08 04:17:48,134 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 51.65 2022-02-08 04:17:48,167 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:17:48,168 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:17:48,170 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 6130 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:17:57,157 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.120.170:8091 2022-02-08 04:17:57,157 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:17:57,197 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:17:57,206 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:17:57,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:17:58,184 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 53.53 % 2022-02-08 04:17:58,184 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 53.53 2022-02-08 04:17:58,214 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:17:58,217 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:17:58,217 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:17:59,226 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:17:59,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:17:59,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:01,249 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:01,257 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:01,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:03,275 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:03,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:03,283 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:05,298 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:05,305 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:05,305 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:07,321 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:07,331 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:07,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:08,230 | test | DEBUG | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.67 % 2022-02-08 04:18:08,230 | test | INFO | pool-3-thread-14 | [task:check:414] Rebalance - status: running, progress: 56.67 2022-02-08 04:18:08,266 | infra | DEBUG | pool-3-thread-14 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:18:08,269 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:18:08,269 | infra | DEBUG | pool-3-thread-14 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:18:09,346 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:09,357 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:09,357 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:11,384 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:11,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:11,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:13,407 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:13,415 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:13,415 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:15,428 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:15,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:15,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:17,456 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:17,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:17,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:18,280 | test | ERROR | pool-3-thread-14 | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'edba486ed5342596b33fc705b9a9ee2a', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=066302813d269f4dad4d699b29bf63a8', u'status': u'notRunning'} - rebalance failed 2022-02-08 04:18:18,309 | test | INFO | pool-3-thread-14 | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.120.170: 2022-02-08 04:18:18,311 | test | ERROR | pool-3-thread-14 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644322690044L, u'shortText': u'message', u'serverTime': u'2022-02-08T04:18:10.044Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.10783.85>,\n {{wait_seqno_persisted_failed,\n "GleamBookUsers0",242,348382,\n [{\'ns_1@172.23.121.115\',\n {\'EXIT\',\n {{{{{badmatch,\n [{<32772.8438.59>,\n {done,exit,\n {{{badmatch,\n {memcached_error,key_enoent,\n undefined}},\n [{dcp_proxy,connect,5,\n [{file,"src/dcp_proxy.erl"},\n {line,257}]},\n {dcp_proxy,maybe_connect,2,\n [{file,"src/dcp_proxy.erl"},\n {line,236}]},\n {dcp_producer_conn,\n handle_call,4,\n [{file,\n "src/dcp_producer_conn.erl"},\n {line,50}]},\n {dcp_proxy,handle_call,3,\n [{file,"src/dcp_proxy.erl"},\n {line,155}]},\n {gen_server,try_handle_call,\n 4,\n [{file,"gen_server.erl"},\n {line,721}]},\n {gen_server,handle_msg,6,\n [{file,"gen_server.erl"},\n {line,750}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},\n {line,226}]}]},\n {gen_server,call,\n [<32772.8441.59>,\n {connect,\n [collections,del_times,\n del_user_xattr,json,\n set_consumer_name,snappy,\n xattr]},\n infinity]}},\n [{gen_server,call,3,\n [{file,"gen_server.erl"},\n {line,247}]},\n {dcp_replicator,\n connect_to_producer,3,\n [{file,\n "src/dcp_replicator.erl"},\n {line,76}]},\n {dcp_replicator,\n \'-spawn_and_wait/1-fun-0-\',1,\n [{file,\n "src/dcp_replicator.erl"},\n {line,309}]}]}}]},\n [{misc,\n sync_shutdown_many_i_am_trapping_exits,\n 1,\n [{file,"src/misc.erl"},\n {line,1454}]},\n {dcp_replicator,spawn_and_wait,1,\n [{file,"src/dcp_replicator.erl"},\n {line,330}]},\n {dcp_replicator,handle_info,2,\n [{file,"src/dcp_replicator.erl"},\n {line,137}]},\n {gen_server,try_dispatch,4,\n [{file,"gen_server.erl"},\n {line,695}]},\n {gen_server,handle_msg,6,\n [{file,"gen_server.erl"},\n {line,771}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},\n {line,226}]}]},\n {gen_server,call,\n [<32772.8280.59>,get_partitions,\n infinity]}},\n {gen_server,call,\n [\'dcp_replication_manager-GleamBookUsers0\',\n {get_replicator_pid,242},\n infinity]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.121.115\'},\n {if_rebalance,<0.15566.82>,\n {wait_seqno_persisted,242,348382}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,474}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,211}]}]}}}}.\nRebalance Operation Id = 38f30cb03967062970c55c63aa830424'} 2022-02-08 04:18:18,312 | test | ERROR | pool-3-thread-14 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644322690010L, u'shortText': u'message', u'serverTime': u'2022-02-08T04:18:10.010Z', u'text': u'Worker <0.10541.85> (for action {move,{242,\n [\'ns_1@172.23.121.116\',\n \'ns_1@172.23.121.127\'],\n [\'ns_1@172.23.121.116\',\n \'ns_1@172.23.121.115\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.10783.85>,\n {{wait_seqno_persisted_failed,\n "GleamBookUsers0",\n 242,\n 348382,\n [{\'ns_1@172.23.121.115\',\n {\'EXIT\',\n {{{{{badmatch,\n [{<32772.8438.59>,\n {done,\n exit,\n {{{badmatch,\n {memcached_error,\n key_enoent,\n undefined}},\n [{dcp_proxy,\n connect,\n 5,\n [{file,\n "src/dcp_proxy.erl"},\n {line,\n 257}]},\n {dcp_proxy,\n maybe_connect,\n 2,\n [{file,\n "src/dcp_proxy.erl"},\n {line,\n 236}]},\n {dcp_producer_conn,\n handle_call,\n 4,\n [{file,\n "src/dcp_producer_conn.erl"},\n {line,\n 50}]},\n {dcp_proxy,\n handle_call,\n 3,\n [{file,\n "src/dcp_proxy.erl"},\n {line,\n 155}]},\n {gen_server,\n try_handle_call,\n 4,\n [{file,\n "gen_server.erl"},\n {line,\n 721}]},\n {gen_server,\n handle_msg,\n 6,\n [{file,\n "gen_server.erl"},\n {line,\n 750}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 226}]}]},\n {gen_server,\n call,\n [<32772.8441.59>,\n {connect,\n [collections,\n del_times,\n del_user_xattr,\n json,\n set_consumer_name,\n snappy,\n xattr]},\n infinity]}},\n [{gen_server,\n call,\n 3,\n [{file,\n "gen_server.erl"},\n {line,\n 247}]},\n {dcp_replicator,\n connect_to_producer,\n 3,\n [{file,\n "src/dcp_replicator.erl"},\n {line,\n 76}]},\n {dcp_replicator,\n \'-spawn_and_wait/1-fun-0-\',\n 1,\n [{file,\n "src/dcp_replicator.erl"},\n {line,\n 309}]}]}}]},\n [{misc,\n sync_shutdown_many_i_am_trapping_exits,\n 1,\n [{file,\n "src/misc.erl"},\n {line,\n 1454}]},\n {dcp_replicator,\n spawn_and_wait,\n 1,\n [{file,\n "src/dcp_replicator.erl"},\n {line,\n 330}]},\n {dcp_replicator,\n handle_info,\n 2,\n [{file,\n "src/dcp_replicator.erl"},\n {line,\n 137}]},\n {gen_server,\n try_dispatch,\n 4,\n [{file,\n "gen_server.erl"},\n {line,\n 695}]},\n {gen_server,\n handle_msg,\n 6,\n [{file,\n "gen_server.erl"},\n {line,\n 771}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 226}]}]},\n {gen_server,\n call,\n [<32772.8280.59>,\n get_partitions,\n infinity]}},\n {gen_server,\n call,\n [\'dcp_replication_manager-GleamBookUsers0\',\n {get_replicator_pid,\n 242},\n infinity]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.121.115\'},\n {if_rebalance,\n <0.15566.82>,\n {wait_seqno_persisted,\n 242,\n 348382}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 474}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 211}]}]}}}'} 2022-02-08 04:18:18,312 | test | ERROR | pool-3-thread-14 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644322567803L, u'shortText': u'message', u'serverTime': u'2022-02-08T04:16:07.803Z', u'text': u"Control connection to memcached on 'ns_1@172.23.120.170' disconnected. Check logs for details."} 2022-02-08 04:18:18,313 | test | ERROR | pool-3-thread-14 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644322567389L, u'shortText': u'message', u'serverTime': u'2022-02-08T04:16:07.389Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW0208 04:02:54.269289 96715 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2022-02-08 04:18:18,313 | test | ERROR | pool-3-thread-14 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644321863630L, u'shortText': u'message', u'serverTime': u'2022-02-08T04:04:23.630Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2022-02-08 04:18:18,313 | test | ERROR | pool-3-thread-14 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644321863165L, u'shortText': u'message', u'serverTime': u'2022-02-08T04:04:23.165Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2022-02-08 04:18:18,313 | test | ERROR | pool-3-thread-14 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644321862419L, u'shortText': u'message', u'serverTime': u'2022-02-08T04:04:22.419Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.121.115','ns_1@172.23.121.132',\n 'ns_1@172.23.121.136','ns_1@172.23.121.135',\n 'ns_1@172.23.120.170','ns_1@172.23.121.129',\n 'ns_1@172.23.121.124','ns_1@172.23.121.126',\n 'ns_1@172.23.121.134','ns_1@172.23.121.116',\n 'ns_1@172.23.121.127','ns_1@172.23.121.123',\n 'ns_1@172.23.121.131','ns_1@172.23.121.130',\n 'ns_1@172.23.121.133'], EjectNodes = ['ns_1@172.23.121.128'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 38f30cb03967062970c55c63aa830424"} 2022-02-08 04:18:18,313 | test | ERROR | pool-3-thread-14 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644321770631L, u'shortText': u'message', u'serverTime': u'2022-02-08T04:02:50.631Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.120.170\' in 19 seconds.'} 2022-02-08 04:18:18,315 | test | ERROR | pool-3-thread-14 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644321712019L, u'shortText': u'message', u'serverTime': u'2022-02-08T04:01:52.019Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.28432.81>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,stats_recv,4,\n [{file,"src/mc_client_binary.erl"},\n {line,163}]},\n {mc_client_binary,stats,4,\n [{file,"src/mc_client_binary.erl"},\n {line,455}]},\n {ns_memcached,do_handle_call,3,\n [{file,"src/ns_memcached.erl"},\n {line,508}]},\n {ns_memcached,worker_loop,3,\n [{file,"src/ns_memcached.erl"},\n {line,227}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,226}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,119,\n "replication:ns_1@172.23.120.170->ns_1@172.23.121.115:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.120.170\'},\n {if_rebalance,<0.934.81>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.121.115\'],\n 119}},\n infinity]}}}}}.\nRebalance Operation Id = e99352b42373454ddf279886adfb4d4e'} 2022-02-08 04:18:18,315 | test | ERROR | pool-3-thread-14 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644321711990L, u'shortText': u'message', u'serverTime': u'2022-02-08T04:01:51.990Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW0208 03:51:42.160712 95256 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2022-02-08 04:18:18,315 | test | ERROR | pool-3-thread-14 | [task:call:325] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'edba486ed5342596b33fc705b9a9ee2a', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=066302813d269f4dad4d699b29bf63a8', u'status': u'notRunning'} - rebalance failed 2022-02-08 04:18:19,479 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:19,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:19,487 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:21,503 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:21,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:21,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:23,542 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:23,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:23,551 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:25,579 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:25,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:25,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:27,615 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:27,627 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:27,627 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:29,642 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:29,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:29,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:31,664 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:31,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:31,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:33,694 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:33,703 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:33,703 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:35,719 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:35,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:35,729 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:37,743 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:37,752 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:37,752 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:39,765 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:39,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:39,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:41,813 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:41,822 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:41,822 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:43,835 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:43,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:43,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:45,859 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:45,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:45,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 04:18:47,888 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 04:18:47,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 04:18:47,900 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 5 queries submitted, 0 failed, 6156 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:18:49,915 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.120.170:8091 is running 2022-02-08 04:18:49,934 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.136:8091 2022-02-08 04:18:49,936 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:18:49,964 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.136:8091 is running 2022-02-08 04:18:49,979 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.126:8091 2022-02-08 04:18:49,979 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:18:49,990 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.126:8091 is running 2022-02-08 04:18:50,003 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.133:8091 2022-02-08 04:18:50,003 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:18:50,015 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.133:8091 is running 2022-02-08 04:18:50,029 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.135:8091 2022-02-08 04:18:50,029 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:18:50,042 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.135:8091 is running 2022-02-08 04:18:50,055 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.134:8091 2022-02-08 04:18:50,055 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:18:50,068 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.134:8091 is running 2022-02-08 04:18:50,084 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.127:8091 2022-02-08 04:18:50,084 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:18:50,111 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.127:8091 is running 2022-02-08 04:18:50,127 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.124:8091 2022-02-08 04:18:50,128 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:18:50,148 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.124:8091 is running 2022-02-08 04:18:50,167 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.130:8091 2022-02-08 04:18:50,167 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:18:50,186 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.130:8091 is running 2022-02-08 04:18:50,203 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.132:8091 2022-02-08 04:18:50,203 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:18:50,214 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.132:8091 is running 2022-02-08 04:18:50,233 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.123:8091 2022-02-08 04:18:50,233 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:18:50,249 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.123:8091 is running 2022-02-08 04:18:50,266 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.115:8091 2022-02-08 04:18:50,266 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:18:50,293 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.115:8091 is running 2022-02-08 04:18:50,312 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.116:8091 2022-02-08 04:18:50,312 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:18:50,332 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.116:8091 is running 2022-02-08 04:18:50,349 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.129:8091 2022-02-08 04:18:50,351 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:18:50,362 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.129:8091 is running 2022-02-08 04:18:50,377 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.131:8091 2022-02-08 04:18:50,378 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 04:18:50,390 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.131:8091 is running 2022-02-08 04:18:50,391 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:18:51,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:18:51,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:18:51,661 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:18:51,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:18:51,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:18:51,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:18:51,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:18:51,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:18:51,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:18:51,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:18:51,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:18:51,811 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 04:18:51,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:18:51,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:18:52,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:18:52,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:18:52,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:18:52,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:18:52,941 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 04:18:52,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:18:52,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:18:52,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:18:53,013 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 04:18:53,013 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:18:53,016 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:18:53,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:18:53,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:18:53,372 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:18:53,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:18:53,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:18:53,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:18:53,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:18:53,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:18:53,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:18:53,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:18:53,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:18:53,486 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 04:18:53,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:18:53,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:18:53,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:18:53,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:18:53,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:18:53,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:18:53,608 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 04:18:53,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:18:53,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:18:53,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:18:53,651 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 04:18:53,651 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:18:53,653 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:18:53,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:18:53,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:18:53,999 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:18:53,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:18:54,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:18:54,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:18:54,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:18:54,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:18:54,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:18:54,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:18:54,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:18:54,115 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 04:18:54,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:18:54,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:18:54,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:18:54,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:18:55,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:18:55,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:18:55,309 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 04:18:55,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:18:55,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:18:55,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:18:55,371 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 04:18:55,371 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:18:55,375 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 04:18:55,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:18:55,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 04:18:55,742 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:18:55,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 04:18:55,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 04:18:55,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 04:18:55,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 04:18:55,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 04:18:55,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 04:18:55,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 04:18:55,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:18:55,848 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 04:18:55,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:18:55,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:18:55,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:18:55,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:18:55,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:18:55,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:18:55,960 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 04:18:55,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:18:55,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:18:55,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:18:56,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:56,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:18:56,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:56,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:56,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:56,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:56,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:56,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:56,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:56,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:56,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:56,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:18:56,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:56,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:56,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:56,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:56,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:56,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:56,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:56,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:56,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:56,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:18:56,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:56,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:56,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:56,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:56,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:56,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:56,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:56,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:56,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:56,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:18:56,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:56,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:56,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:56,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:56,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:56,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:56,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:56,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:56,796 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 04:18:56,801 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:18:56,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:18:57,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:18:57,178 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:18:57,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:18:57,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:18:57,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:18:57,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:18:57,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:18:57,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:18:57,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:18:57,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:18:57,296 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 04:18:57,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:18:57,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:18:57,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:18:57,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:18:57,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:18:58,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:18:58,061 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 04:18:58,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:18:58,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:18:58,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:18:58,101 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 04:18:58,101 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:18:58,104 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 04:18:58,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:18:58,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 04:18:58,401 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:18:58,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 04:18:58,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 04:18:58,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 04:18:58,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 04:18:58,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 04:18:58,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 04:18:58,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 04:18:58,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:18:58,513 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 04:18:58,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:18:58,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:18:58,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:18:58,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:18:58,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:18:58,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:18:58,627 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 04:18:58,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:18:58,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:18:58,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:18:58,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:58,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:18:58,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:58,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:58,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:58,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:58,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:58,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:58,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:58,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:18:58,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:58,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:18:59,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:59,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:59,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:59,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:59,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:59,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:59,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:59,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:18:59,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:59,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:18:59,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:59,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:59,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:59,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:59,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:59,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:59,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:59,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:18:59,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:59,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:18:59,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:59,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:59,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:59,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:59,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:59,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:59,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:59,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:18:59,456 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 04:18:59,460 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:18:59,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:18:59,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:18:59,855 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:18:59,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:18:59,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:18:59,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:18:59,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:18:59,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:18:59,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:18:59,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:18:59,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:18:59,960 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 04:18:59,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:19:00,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:00,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:19:00,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:00,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:01,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:19:01,203 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 04:19:01,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:19:01,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:01,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:19:01,270 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 04:19:01,270 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:19:01,273 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:19:01,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:19:01,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:19:01,651 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:19:01,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:19:01,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:19:01,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:19:01,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:19:01,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:19:01,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:19:01,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:19:01,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:19:01,789 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 04:19:01,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:19:01,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:02,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:19:02,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:02,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:02,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:19:02,990 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 04:19:02,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:19:03,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:03,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:19:03,043 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 04:19:03,045 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:19:03,046 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 04:19:03,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:19:03,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 04:19:03,460 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:19:03,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 04:19:03,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 04:19:03,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 04:19:03,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 04:19:03,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 04:19:03,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 04:19:03,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 04:19:03,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:19:03,598 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 04:19:03,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:19:03,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:03,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:19:03,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:03,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:03,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:19:03,720 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 04:19:03,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:19:03,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:03,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:19:03,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:03,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:19:03,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:03,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:03,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:03,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:04,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:04,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:04,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:04,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:04,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:04,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:19:04,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:04,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:04,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:04,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:04,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:04,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:04,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:04,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:04,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:04,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:19:04,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:04,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:04,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:04,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:04,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:04,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:04,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:04,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:04,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:04,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:19:04,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:04,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:04,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:04,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:04,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:04,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:04,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:04,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:04,681 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 04:19:04,684 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 04:19:04,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:19:04,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 04:19:05,043 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:19:05,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 04:19:05,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 04:19:05,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 04:19:05,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 04:19:05,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 04:19:05,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 04:19:05,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 04:19:05,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:19:05,150 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 04:19:05,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:19:05,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:05,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:19:05,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:05,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:05,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:19:05,262 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 04:19:05,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:19:05,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:05,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:19:05,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:05,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:19:05,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:05,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:05,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:05,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:05,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:05,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:05,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:05,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:05,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:05,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:19:05,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:05,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:05,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:05,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:05,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:05,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:05,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:05,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:05,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:05,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:19:05,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:05,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:05,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:05,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:05,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:05,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:05,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:06,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:06,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:06,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:19:06,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:06,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:06,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:06,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:06,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:06,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:06,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:06,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:06,209 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 04:19:06,210 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:19:06,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:19:06,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:19:06,601 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:19:06,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:19:06,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:19:06,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:19:06,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:19:06,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:19:06,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:19:06,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:19:06,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:19:06,720 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 04:19:06,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:19:06,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:07,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:19:07,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:07,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:07,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:19:07,887 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 04:19:07,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:19:07,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:07,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:19:07,937 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 04:19:07,937 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:19:07,938 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:19:08,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:19:08,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:19:08,434 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:19:08,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:19:08,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:19:08,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:19:08,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:19:08,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:19:08,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:19:08,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:19:08,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:19:08,585 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 04:19:08,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:19:08,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:08,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:19:09,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:09,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:09,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:19:09,625 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 04:19:09,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:19:09,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:09,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:19:09,703 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 04:19:09,704 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:19:09,707 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:19:09,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:19:10,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:19:10,193 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:19:10,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:19:10,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:19:10,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:19:10,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:19:10,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:19:10,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:19:10,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:19:10,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:19:10,325 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 04:19:10,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:19:10,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:10,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:19:10,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:11,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:11,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:19:11,329 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 04:19:11,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:19:11,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:11,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:19:11,398 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 04:19:11,398 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:19:11,401 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:19:11,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:19:11,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:19:11,813 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:19:11,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:19:11,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:19:11,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:19:11,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:19:11,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:19:11,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:19:11,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:19:11,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:19:11,961 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 04:19:11,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:19:12,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:12,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:19:12,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:13,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:13,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:19:13,540 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 04:19:13,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:19:13,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:13,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:19:13,588 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 04:19:13,588 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:19:13,589 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 04:19:13,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:19:13,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 04:19:14,036 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:19:14,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 04:19:14,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 04:19:14,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 04:19:14,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 04:19:14,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 04:19:14,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 04:19:14,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 04:19:14,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:19:14,203 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 04:19:14,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:19:14,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:14,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:19:14,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:14,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:19:14,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:19:14,388 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 04:19:14,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:19:14,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:19:14,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:19:14,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:14,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:19:14,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:14,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:14,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:14,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:14,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:14,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:14,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:14,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:19:14,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:14,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:19:14,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:14,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:14,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:14,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:14,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:15,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:15,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:15,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:19:15,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:15,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:19:15,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:15,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:15,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:15,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:15,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:15,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:15,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:15,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:19:15,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:15,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:19:15,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:15,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:15,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:15,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:15,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:15,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:15,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:15,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:19:15,398 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 04:19:15,401 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.121.128]_1644321862.23 2022-02-08 04:19:15,401 | test | INFO | MainThread | [opd:abort_rebalance:808] Restarting Rebalance after killing at 40 2022-02-08 04:19:15,502 | infra | DEBUG | MainThread | [task:__init__:249] Last known rebalance status_id: edba486ed5342596b33fc705b9a9ee2a 2022-02-08 04:19:15,502 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.121.128]_1644322755.4 2022-02-08 04:19:15,502 | infra | INFO | pool-3-thread-22 | [task:start_task:91] Thread 'Rebalance_task_IN=[]_OUT=[172.23.121.128]_1644322755.4' started 2022-02-08 04:19:15,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L813 2022-02-08 04:19:15,519 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2022-02-08 04:19:15,562 | test | DEBUG | pool-3-thread-22 | [task:start_rebalance:384] Removing node 172.23.121.128:8091 from cluster 2022-02-08 04:19:15,562 | test | DEBUG | pool-3-thread-22 | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.128,ns_1@172.23.121.115,ns_1@172.23.121.132,ns_1@172.23.121.136,ns_1@172.23.121.135,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.134,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123,ns_1@172.23.121.131,ns_1@172.23.121.130,ns_1@172.23.121.133', 'ejectedNodes': u'ns_1@172.23.121.128', 'user': 'Administrator'} 2022-02-08 04:19:15,638 | test | DEBUG | pool-3-thread-22 | [rest_client:rebalance:1459] Rebalance operation started 2022-02-08 04:19:15,640 | test | INFO | pool-3-thread-22 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.121.128 | kv | 7.1.0-2255-enterprise | 30.1562701795 | --- OUT ---> | | 172.23.121.115 | kv | 7.1.0-2255-enterprise | 49.7306310929 | Cluster node | | 172.23.121.132 | index, n1ql | 7.1.0-2255-enterprise | 10.2829091849 | Cluster node | | 172.23.121.136 | kv | 7.1.0-2255-enterprise | 49.2015706806 | Cluster node | | 172.23.121.135 | kv | 7.1.0-2255-enterprise | 33.2261091007 | Cluster node | | 172.23.120.170 | kv | 7.1.0-2255-enterprise | 31.8733239688 | Cluster node | | 172.23.121.129 | kv | 7.1.0-2255-enterprise | 26.3254352664 | Cluster node | | 172.23.121.124 | kv | 7.1.0-2255-enterprise | 38.0546955624 | Cluster node | | 172.23.121.126 | kv | 7.1.0-2255-enterprise | 23.0244158572 | Cluster node | | 172.23.121.134 | index, n1ql | 7.1.0-2255-enterprise | 14.8124357657 | Cluster node | | 172.23.121.116 | kv | 7.1.0-2255-enterprise | 29.0434782609 | Cluster node | | 172.23.121.127 | kv | 7.1.0-2255-enterprise | 23.7858358424 | Cluster node | | 172.23.121.123 | kv | 7.1.0-2255-enterprise | 37.2664314648 | Cluster node | | 172.23.121.131 | index, n1ql | 7.1.0-2255-enterprise | 4.63128349069 | Cluster node | | 172.23.121.130 | index, n1ql | 7.1.0-2255-enterprise | 11.118285124 | Cluster node | | 172.23.121.133 | index, n1ql | 7.1.0-2255-enterprise | 6.43140263393 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2022-02-08 04:19:15,651 | infra | DEBUG | pool-3-thread-22 | [task:call:316] New rebalance status_id: 7c2f7ecf59bd476297411cab8e5bc2d7 2022-02-08 04:19:15,661 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 04:19:15,661 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 04:19:15,719 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:19:15,721 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:19:15,721 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:19:25,739 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 04:19:25,739 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 04:19:25,790 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:19:25,793 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:19:25,793 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:19:35,812 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.58 % 2022-02-08 04:19:35,812 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 0.58 2022-02-08 04:19:35,859 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:19:35,862 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:19:35,862 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:19:45,875 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.58 % 2022-02-08 04:19:45,875 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 0.58 2022-02-08 04:19:45,907 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:19:45,908 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:19:45,908 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 6180 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:19:55,924 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.17 % 2022-02-08 04:19:55,924 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 1.17 2022-02-08 04:19:55,960 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:19:56,062 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:19:56,062 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:20:06,081 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.45 % 2022-02-08 04:20:06,081 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 2.45 2022-02-08 04:20:06,128 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:20:06,131 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:20:06,132 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:20:16,158 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.45 % 2022-02-08 04:20:16,158 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 2.45 2022-02-08 04:20:16,191 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:20:16,194 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:20:16,194 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:20:26,213 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.53 % 2022-02-08 04:20:26,213 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 10.53 2022-02-08 04:20:26,252 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:20:26,255 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:20:26,255 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:20:36,296 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.12 % 2022-02-08 04:20:36,296 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 11.12 2022-02-08 04:20:36,346 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:20:36,349 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:20:36,349 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:20:46,361 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.10 % 2022-02-08 04:20:46,362 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 12.1 2022-02-08 04:20:46,398 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:20:46,400 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:20:46,400 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 6204 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:20:56,608 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.94 % 2022-02-08 04:20:56,658 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 13.94 2022-02-08 04:20:56,700 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:20:56,948 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:20:56,948 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:21:06,963 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.94 % 2022-02-08 04:21:06,963 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 13.94 2022-02-08 04:21:06,993 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:21:07,049 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:21:07,049 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:21:15,530 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.94 % 2022-02-08 04:21:15,530 | test | INFO | MainThread | [opd:abort_rebalance:814] Rebalance % = 13.94 2022-02-08 04:21:15,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.94 % 2022-02-08 04:21:15,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:21:15,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:21:17,063 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.94 % 2022-02-08 04:21:17,063 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 13.94 2022-02-08 04:21:17,101 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:21:17,104 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:21:17,104 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:21:25,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.31 % 2022-02-08 04:21:25,673 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:21:25,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:21:27,119 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.31 % 2022-02-08 04:21:27,119 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 22.31 2022-02-08 04:21:27,167 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:21:27,170 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:21:27,170 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:21:35,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.31 % 2022-02-08 04:21:35,698 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:21:35,698 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:21:37,190 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.31 % 2022-02-08 04:21:37,190 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 22.31 2022-02-08 04:21:37,233 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:21:37,234 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:21:37,234 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:21:45,721 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.61 % 2022-02-08 04:21:45,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:21:45,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:21:47,249 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.61 % 2022-02-08 04:21:47,249 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 22.61 2022-02-08 04:21:47,292 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:21:47,295 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:21:47,295 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 6223 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:21:55,749 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.19 % 2022-02-08 04:21:55,757 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:21:55,757 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:21:57,309 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.19 % 2022-02-08 04:21:57,309 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 23.19 2022-02-08 04:21:57,355 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:21:57,358 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:21:57,358 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:22:05,789 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.89 % 2022-02-08 04:22:05,798 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:22:05,799 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:22:07,371 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.14 % 2022-02-08 04:22:07,371 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 30.14 2022-02-08 04:22:07,404 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:22:07,407 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:22:07,407 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:22:15,813 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.50 % 2022-02-08 04:22:15,821 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:22:15,822 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:22:17,420 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.79 % 2022-02-08 04:22:17,421 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 33.79 2022-02-08 04:22:17,453 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:22:17,456 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:22:17,456 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:22:25,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.08 % 2022-02-08 04:22:25,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:22:25,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:22:27,471 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.08 % 2022-02-08 04:22:27,471 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 34.08 2022-02-08 04:22:27,503 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:22:27,506 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:22:27,506 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:22:35,864 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.08 % 2022-02-08 04:22:35,871 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:22:35,872 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:22:37,555 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.08 % 2022-02-08 04:22:37,556 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 34.08 2022-02-08 04:22:37,602 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:22:37,605 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:22:37,605 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:22:45,907 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.65 % 2022-02-08 04:22:45,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:22:45,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:22:47,617 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.65 % 2022-02-08 04:22:47,618 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 35.65 2022-02-08 04:22:47,665 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:22:47,668 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:22:47,668 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 6246 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:22:55,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.65 % 2022-02-08 04:22:55,984 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:22:55,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:22:57,681 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.65 % 2022-02-08 04:22:57,681 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 35.65 2022-02-08 04:22:57,719 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:22:57,720 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:22:57,721 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:23:06,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.24 % 2022-02-08 04:23:06,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:23:06,020 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:23:07,736 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.10 % 2022-02-08 04:23:07,736 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 37.1 2022-02-08 04:23:07,767 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:23:07,769 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:23:07,770 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:23:16,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.23 % 2022-02-08 04:23:16,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:23:16,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:23:17,795 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.23 % 2022-02-08 04:23:17,795 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 38.23 2022-02-08 04:23:17,838 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:23:17,839 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:23:17,839 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:23:26,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.23 % 2022-02-08 04:23:26,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:23:26,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:23:27,858 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.23 % 2022-02-08 04:23:27,858 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 38.23 2022-02-08 04:23:27,907 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:23:27,911 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:23:27,911 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:23:36,094 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.93 % 2022-02-08 04:23:36,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:23:36,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:23:37,924 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.93 % 2022-02-08 04:23:37,924 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 38.93 2022-02-08 04:23:37,973 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:23:37,976 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:23:37,976 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:23:46,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.80 % 2022-02-08 04:23:46,154 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:23:46,154 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:23:47,989 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.80 % 2022-02-08 04:23:47,990 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 39.8 2022-02-08 04:23:48,022 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:23:48,026 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:23:48,026 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 6273 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:23:56,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.80 % 2022-02-08 04:23:56,181 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:23:56,183 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:23:58,043 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.80 % 2022-02-08 04:23:58,043 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 39.8 2022-02-08 04:23:58,088 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:23:58,092 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:23:58,092 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:24:06,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.68 % 2022-02-08 04:24:06,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:24:06,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:24:08,115 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.68 % 2022-02-08 04:24:08,115 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 40.68 2022-02-08 04:24:08,171 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:24:08,174 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:24:08,176 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:24:16,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 41.25 % 2022-02-08 04:24:16,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:24:16,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:24:18,187 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 41.25 % 2022-02-08 04:24:18,187 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 41.25 2022-02-08 04:24:18,217 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:24:18,220 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:24:18,220 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:24:26,252 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 41.82 % 2022-02-08 04:24:26,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:24:26,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:24:28,232 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 41.82 % 2022-02-08 04:24:28,232 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 41.82 2022-02-08 04:24:28,273 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:24:28,276 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:24:28,278 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:24:36,298 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 41.82 % 2022-02-08 04:24:36,305 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:24:36,305 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:24:38,292 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 41.82 % 2022-02-08 04:24:38,292 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 41.82 2022-02-08 04:24:38,325 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:24:38,328 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:24:38,328 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:24:46,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 41.82 % 2022-02-08 04:24:46,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:24:46,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:24:48,339 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.11 % 2022-02-08 04:24:48,339 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 42.11 2022-02-08 04:24:48,375 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:24:48,378 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:24:48,378 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 6297 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:24:56,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.11 % 2022-02-08 04:24:56,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:24:56,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:24:58,394 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.11 % 2022-02-08 04:24:58,394 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 42.11 2022-02-08 04:24:58,456 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:24:58,457 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:24:58,457 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:25:06,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.68 % 2022-02-08 04:25:06,411 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:25:06,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:25:08,470 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.68 % 2022-02-08 04:25:08,470 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 42.68 2022-02-08 04:25:08,505 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:25:08,509 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:25:08,509 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:25:16,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 43.94 % 2022-02-08 04:25:16,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:25:16,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:25:18,523 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 43.94 % 2022-02-08 04:25:18,523 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 43.94 2022-02-08 04:25:18,559 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:25:18,565 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:25:18,565 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:25:26,466 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.23 % 2022-02-08 04:25:26,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:25:26,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:25:28,585 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.23 % 2022-02-08 04:25:28,585 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 44.23 2022-02-08 04:25:28,638 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:25:28,641 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:25:28,641 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:25:36,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.23 % 2022-02-08 04:25:36,497 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:25:36,497 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:25:38,658 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.52 % 2022-02-08 04:25:38,658 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 44.52 2022-02-08 04:25:38,756 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:25:38,759 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:25:38,759 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:25:46,512 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.52 % 2022-02-08 04:25:46,522 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:25:46,522 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:25:48,773 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.09 % 2022-02-08 04:25:48,773 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 45.09 2022-02-08 04:25:48,809 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:25:48,812 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:25:48,812 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 6321 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:25:56,538 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.09 % 2022-02-08 04:25:56,545 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:25:56,545 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:25:58,826 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.09 % 2022-02-08 04:25:58,826 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 45.09 2022-02-08 04:25:58,881 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:25:58,882 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:25:58,882 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:26:06,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.09 % 2022-02-08 04:26:06,566 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:26:06,566 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:26:08,910 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.09 % 2022-02-08 04:26:08,910 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 45.09 2022-02-08 04:26:08,947 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:26:08,950 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:26:08,950 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:26:16,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.09 % 2022-02-08 04:26:16,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:26:16,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:26:18,963 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.09 % 2022-02-08 04:26:18,963 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 45.09 2022-02-08 04:26:19,013 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:26:19,016 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:26:19,016 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:26:26,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.09 % 2022-02-08 04:26:26,612 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:26:26,612 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:26:29,052 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.09 % 2022-02-08 04:26:29,052 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 45.09 2022-02-08 04:26:29,107 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:26:29,109 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:26:29,109 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:26:36,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.91 % 2022-02-08 04:26:36,686 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:26:36,686 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:26:39,122 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.91 % 2022-02-08 04:26:39,122 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 47.91 2022-02-08 04:26:39,181 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:26:39,184 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:26:39,184 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:26:46,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.91 % 2022-02-08 04:26:46,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:26:46,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:26:49,213 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.91 % 2022-02-08 04:26:49,213 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 47.91 2022-02-08 04:26:49,266 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:26:49,273 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:26:49,273 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 6345 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:26:56,744 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.91 % 2022-02-08 04:26:56,752 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:26:56,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:26:59,285 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.91 % 2022-02-08 04:26:59,285 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 47.91 2022-02-08 04:26:59,315 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:26:59,318 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:26:59,318 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:27:06,821 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.91 % 2022-02-08 04:27:06,829 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:27:06,829 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:27:09,332 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.91 % 2022-02-08 04:27:09,332 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 47.91 2022-02-08 04:27:09,367 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:27:09,369 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:27:09,369 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:27:16,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.30 % 2022-02-08 04:27:16,849 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:27:16,849 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:27:19,387 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.30 % 2022-02-08 04:27:19,388 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 49.3 2022-02-08 04:27:19,427 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:27:19,431 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:27:19,431 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:27:26,862 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.30 % 2022-02-08 04:27:26,872 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:27:26,872 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:27:29,444 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.30 % 2022-02-08 04:27:29,446 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 49.3 2022-02-08 04:27:29,490 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:27:29,493 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:27:29,493 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:27:36,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.30 % 2022-02-08 04:27:36,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:27:36,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:27:39,505 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.30 % 2022-02-08 04:27:39,506 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 49.3 2022-02-08 04:27:39,542 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:27:39,546 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:27:39,546 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:27:46,920 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 50.00 % 2022-02-08 04:27:46,928 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:27:46,928 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 6367 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:27:49,559 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 50.00 % 2022-02-08 04:27:49,559 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: running, progress: 50.0 2022-02-08 04:27:49,599 | infra | DEBUG | pool-3-thread-22 | [task:check:441] Wait before next rebalance progress check 2022-02-08 04:27:49,601 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 04:27:49,602 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:27:56,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 84.38 % 2022-02-08 04:27:56,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:27:56,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:27:59,617 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-08 04:27:59,618 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 04:28:04,644 | test | INFO | pool-3-thread-22 | [task:check:414] Rebalance - status: none, progress: 100 2022-02-08 04:28:04,717 | test | INFO | pool-3-thread-22 | [task:check:473] Rebalance completed with progress: 100% in 529.078999996 sec 2022-02-08 04:28:04,717 | infra | INFO | pool-3-thread-22 | [task:complete_task:106] Thread 'Rebalance_task_IN=[]_OUT=[172.23.121.128]_1644322755.4' completed 2022-02-08 04:28:06,967 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >84.38% in 411.436000109 seconds 2022-02-08 04:28:06,983 | test | DEBUG | MainThread | [rest_client:diag_eval:1476] /diag/eval status on 172.23.120.170:8091: 1 content: false command: ns_orchestrator:needs_rebalance() 2022-02-08 04:28:07,015 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-08 04:28:07,016 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 04:28:12,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 04:28:12,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 04:28:22,048 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >100% in 15.0650000572 seconds 2022-02-08 04:28:22,069 | test | DEBUG | MainThread | [rest_client:diag_eval:1476] /diag/eval status on 172.23.120.170:8091: 1 content: false command: ns_orchestrator:needs_rebalance() 2022-02-08 04:28:22,069 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.121.128]_1644322755.4 2022-02-08 04:28:22,155 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3970] Stats: {u'opsPerSec': 23840.2, u'diskFetches': 3, u'dataUsed': 89337784389L, u'itemCount': 77132120, u'quotaPercentUsed': 78.1694718895934, u'storageTotals': {u'hdd': {u'usedByData': 151703308226L, u'used': 264497507077L, u'quotaTotal': 1298666618880L, u'total': 1298666618880L, u'free': 888207963350L}, u'ram': {u'usedByData': 67630668192L, u'used': 117998379008L, u'quotaTotal': 87566581760L, u'total': 122941186048L, u'quotaUsed': 86518005760L, u'quotaUsedPerNode': 8651800576L, u'quotaTotalPerNode': 8756658176L}}, u'diskUsed': 151703308226L, u'vbActiveNumNonResident': 39367050, u'memUsed': 67630668192L} 2022-02-08 04:28:22,908 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+------------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 77132120 | 80.58 GiB | 62.99 GiB | 141.28 GiB | 48.9615350907 | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+------------+---------------+ 2022-02-08 04:28:22,910 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+------------------+------------------+-----------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+------------------+------------------+-----------------+---------------+ | 40000000 | 380000000 | 2000000-20000000 | 2000000-20000000 | 1000000-2000000 | 0-0 | +---------------+---------------+------------------+------------------+-----------------+---------------+ 2022-02-08 04:28:23,207 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 48.9615350907 2022-02-08 04:28:23,226 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:28:23,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:24,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:28:24,351 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:24,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:28:24,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:28:24,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:28:24,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:28:24,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:28:24,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:28:24,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:28:24,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 04:28:24,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 04:28:24,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 04:28:24,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 04:28:24,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 04:28:25,276 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:28:25,278 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:28:25,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:25,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:28:25,746 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:25,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:28:25,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:28:25,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:28:25,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:28:25,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:28:25,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:28:25,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:28:25,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 04:28:26,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 04:28:26,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 04:28:26,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 04:28:26,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 04:28:26,496 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:28:26,500 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:28:26,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:26,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:28:26,950 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:26,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:28:26,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:28:26,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:28:27,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:28:27,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:28:27,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:28:27,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:28:27,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 04:28:27,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 04:28:27,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 04:28:27,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 04:28:27,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 04:28:27,960 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:28:27,961 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 04:28:28,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:28,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 04:28:28,365 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:28,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 04:28:28,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 04:28:28,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 04:28:28,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 04:28:28,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 04:28:28,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 04:28:28,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 04:28:28,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 04:28:28,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 04:28:28,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 04:28:28,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 04:28:28,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 04:28:28,605 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 04:28:28,607 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:28:28,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:29,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:28:29,084 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:29,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:28:29,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:28:29,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:28:29,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:28:29,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:28:29,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:28:29,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:28:29,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 04:28:29,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 04:28:29,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 04:28:29,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 04:28:29,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 04:28:29,967 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:28:29,970 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 04:28:30,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:30,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 04:28:30,378 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:30,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 04:28:30,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 04:28:30,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 04:28:30,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 04:28:30,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 04:28:30,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 04:28:30,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 04:28:30,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 04:28:30,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 04:28:30,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 04:28:30,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 04:28:30,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 04:28:30,598 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 04:28:30,601 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:28:31,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:31,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:28:31,427 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:31,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:28:31,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:28:31,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:28:31,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:28:31,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:28:31,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:28:31,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:28:31,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 04:28:31,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 04:28:31,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 04:28:31,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 04:28:31,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 04:28:32,359 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:28:32,362 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:28:32,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:32,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:28:32,858 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:32,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:28:32,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:28:32,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:28:32,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:28:32,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:28:33,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:28:33,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:28:33,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 04:28:33,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 04:28:33,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 04:28:33,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 04:28:33,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 04:28:33,861 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:28:33,865 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 04:28:33,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:34,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 04:28:34,269 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:34,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 04:28:34,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 04:28:34,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 04:28:34,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 04:28:34,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 04:28:34,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 04:28:34,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 04:28:34,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 04:28:34,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 04:28:34,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 04:28:34,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 04:28:34,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 04:28:34,546 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 04:28:34,551 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 04:28:34,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:34,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 04:28:34,957 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:34,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 04:28:34,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 04:28:34,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 04:28:35,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 04:28:35,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 04:28:35,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 04:28:35,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 04:28:35,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 04:28:35,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 04:28:35,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 04:28:35,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 04:28:35,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 04:28:35,183 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 04:28:35,184 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:28:35,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:35,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:28:35,898 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:35,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:28:35,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:28:35,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:28:35,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:28:35,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:28:35,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:28:36,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:28:36,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 04:28:36,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 04:28:36,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 04:28:36,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 04:28:36,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 04:28:36,937 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:28:36,940 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:28:37,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:37,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:28:37,444 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:37,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:28:37,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:28:37,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:28:37,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:28:37,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:28:37,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:28:37,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:28:37,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 04:28:37,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 04:28:37,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 04:28:37,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 04:28:37,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 04:28:38,062 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:28:38,063 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:28:38,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:38,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:28:38,457 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:38,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:28:38,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:28:38,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:28:38,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:28:38,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:28:38,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:28:38,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:28:38,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 04:28:38,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 04:28:38,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 04:28:38,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 04:28:38,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 04:28:39,273 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:28:39,278 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:28:39,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:39,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:28:39,779 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:39,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:28:39,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:28:39,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:28:39,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:28:39,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:28:39,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:28:39,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:28:39,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 04:28:40,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 04:28:40,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 04:28:40,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 04:28:40,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 04:28:40,651 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:28:40,654 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 04:28:40,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:41,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 04:28:41,125 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:41,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 04:28:41,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 04:28:41,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 04:28:41,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 04:28:41,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 04:28:41,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 04:28:41,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 04:28:41,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 04:28:41,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 04:28:41,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 04:28:41,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 04:28:41,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 04:28:41,359 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 04:28:41,359 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for kvstore is 140897MB 2022-02-08 04:28:41,361 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for wal is 5492MB 2022-02-08 04:28:41,361 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:642] Total Disk usage for keyTree is 10012MB 2022-02-08 04:28:41,361 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:643] Total Disk usage for seqTree is 130582MB 2022-02-08 04:28:41,362 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:28:41,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:41,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:28:41,773 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:41,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:28:41,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:28:41,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:28:41,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:28:41,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:28:41,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:28:41,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:28:41,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 04:28:41,940 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.116 - core(s): 8 2022-02-08 04:28:41,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 04:28:42,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 04:28:42,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 04:28:42,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 04:28:43,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 04:28:43,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 04:28:43,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 04:28:43,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 04:28:43,670 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:28:43,673 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:28:43,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:44,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:28:44,217 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:44,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:28:44,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:28:44,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:28:44,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:28:44,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:28:44,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:28:44,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:28:44,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 04:28:44,466 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.123 - core(s): 8 2022-02-08 04:28:44,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 04:28:45,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 04:28:45,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 04:28:45,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 04:28:45,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 04:28:46,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 04:28:46,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 04:28:46,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 04:28:46,901 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:28:46,904 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:28:46,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:47,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:28:47,306 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:47,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:28:47,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:28:47,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:28:47,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:28:47,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:28:47,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:28:47,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:28:47,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 04:28:47,486 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.124 - core(s): 8 2022-02-08 04:28:47,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 04:28:47,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 04:28:48,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 04:28:48,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 04:28:48,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 04:28:48,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 04:28:48,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 04:28:48,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 04:28:49,088 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:28:49,089 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:28:49,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 6394 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:28:49,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:28:49,575 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:49,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:28:49,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:28:49,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:28:49,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:28:49,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:28:49,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:28:49,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:28:49,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 04:28:49,752 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.126 - core(s): 8 2022-02-08 04:28:49,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 04:28:50,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 04:28:50,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 04:28:51,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 04:28:51,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 04:28:51,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 04:28:51,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 04:28:51,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 04:28:51,878 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:28:51,881 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:28:51,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:52,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:28:52,247 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:52,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:28:52,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:28:52,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:28:52,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:28:52,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:28:52,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:28:52,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:28:52,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 04:28:52,463 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.127 - core(s): 8 2022-02-08 04:28:52,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 04:28:53,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 04:28:53,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 04:28:53,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 04:28:53,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 04:28:53,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 04:28:53,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 04:28:54,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 04:28:54,529 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:28:54,533 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:28:54,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:54,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:28:54,953 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:54,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:28:54,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:28:54,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:28:54,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:28:55,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:28:55,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:28:55,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:28:55,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 04:28:55,121 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.129 - core(s): 8 2022-02-08 04:28:55,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 04:28:55,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 04:28:55,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 04:28:56,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 04:28:56,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 04:28:56,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 04:28:56,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 04:28:56,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 04:28:56,885 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:28:56,890 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:28:56,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:57,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:28:57,451 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:57,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:28:57,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:28:57,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:28:57,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:28:57,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:28:57,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:28:57,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:28:57,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 04:28:57,714 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.135 - core(s): 8 2022-02-08 04:28:57,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 04:28:58,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 04:28:58,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 04:28:58,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 04:28:58,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 04:28:58,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 04:28:59,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 04:28:59,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 04:28:59,463 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:28:59,470 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:28:59,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:28:59,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:28:59,971 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:28:59,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:29:00,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:29:00,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:29:00,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:29:00,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:29:00,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:29:00,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:29:00,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 04:29:00,148 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.115 - core(s): 8 2022-02-08 04:29:00,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 04:29:00,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 04:29:00,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 04:29:01,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 04:29:01,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 04:29:01,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 04:29:01,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 04:29:01,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 04:29:02,032 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:29:02,033 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:29:02,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:29:02,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:29:02,457 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:29:02,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:29:02,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:29:02,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:29:02,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:29:02,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:29:02,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:29:02,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:29:02,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 04:29:02,693 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.136 - core(s): 8 2022-02-08 04:29:02,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 04:29:03,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 04:29:03,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 04:29:03,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 04:29:03,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 04:29:03,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 04:29:04,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 04:29:04,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 04:29:04,335 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:29:04,336 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:29:04,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:29:04,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:29:04,766 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:29:04,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:29:04,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:29:04,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:29:04,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:29:04,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:29:04,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:29:04,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:29:04,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 04:29:04,976 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.120.170 - core(s): 8 2022-02-08 04:29:04,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 04:29:05,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 04:29:05,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 04:29:06,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 04:29:06,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 04:29:06,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 04:29:06,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 04:29:06,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 04:29:06,891 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:29:06,891 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:723] {'172.23.121.116': {'rw_0:magma': {u'NReadBytesGetPerSec': 1395476.875, u'ReplayWALOffset': u'929:12779520', u'NTablesCreated': 14896, u'RecentReadIOAmpSet': 0.025628140196204185, u'ReadIOAmp': 2.5903408527374268, u'BlockCacheMisses': 335660, u'NFlushes': 3592, u'RecentWriteAmp': 2.239135503768921, u'NumSyncFlushes': 127, u'TotalMemUsed': 26884019, u'NBloomFilterHits': 7372285, u'Fragmentation': 0.40750613808631897, u'NumNormalFlushes': 1647, u'RecentBlockCacheHitRatio': 55.785125732421875, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 7392, u'ReadIOAmp': 52.971014492753625, u'MemtablesMemUsed': 7772312, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 16076555, u'CheckpointOverhead': 2056192, u'NItemsGet': 69, u'NWriteIOs': 9866, u'NReadBytes': 312436082, u'LevelStats': [{u'TargetSize': 113246208, u'LogicalSize': 31671458, u'SizeRatio': 0.2796690375716598, u'NWriteBytes': 33869824, u'BloomFilterMemUsed': 0, u'NumTables': 235, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2576384}, {u'TargetSize': 113246208, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 113246208, u'LogicalSize': 89647789, u'SizeRatio': 0.7916184619620994, u'NWriteBytes': 15745024, u'BloomFilterMemUsed': 0, u'NumTables': 54, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 1499136}, {u'TargetSize': 889041124, u'LogicalSize': 757529886, u'SizeRatio': 0.8520751915183622, u'NWriteBytes': 5603328, u'BloomFilterMemUsed': 0, u'NumTables': 342, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 12845056}, {u'TargetSize': 3463722946L, u'LogicalSize': 1778082255, u'SizeRatio': 0.5133442491563527, u'NWriteBytes': 679251968, u'BloomFilterMemUsed': 0, u'NumTables': 734, u'WastedSpace': 46220782, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 1778044928}], u'NItemsGarbageCollected': 733496, u'NInternalFragmentationCompacts': 25, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 12628752, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 165312, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 171155, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 485604, u'CompactWriter': {u'NDataBlockWrites': 72144, u'NWriteBytes': 212103168, u'NIndexBlockWrites': 1110, u'NBytesIncoming': 334291346, u'NSyncs': 0, u'NInserts': 1423116, u'NWriteIOs': 970}, u'NonResidentBloomFilterSize': 0, u'IteratorMemUsed': 0, u'NWriteBytes': 744091265, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 3655, u'NReadBytes': 649298, u'NDataBlockReads': 69, u'NBytesOutgoing': 0, u'NIndexBlockReads': 138}, u'NBloomFilterMisses': 0, u'NItemsMerged': 187338, u'LogicalDiskSize': 1778082255, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.5500107697970584, u'BufferMemUsed': 1149952, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 1020552369, u'NTableFiles': 1587, u'NFileCountCompacts': 195, u'TableObjectMemUsed': 565680, u'TreeSnapshotMemoryUsed': 257808, u'TotalDiskUsage': 2252603392L, u'BytesIncoming': 1352866718, u'NItemsDel': 0, u'PendingCheckpointBytes': 5025792, u'NCompacts': 406, u'NItemsPut': 2053209, u'NSyncs': 1953, u'CompactReader': {u'NReadIOs': 119228, u'NReadBytes': 291455655, u'NDataBlockReads': 114317, u'NBytesOutgoing': 0, u'NIndexBlockReads': 4913}, u'ActiveDiskUsage': 1794965504, u'NTablesDeleted': 4072, u'IterateReader': {u'NReadIOs': 48272, u'NReadBytes': 20331129, u'NDataBlockReads': 24500, u'NBytesOutgoing': 0, u'NIndexBlockReads': 23772}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 183960, u'NWriteBytes': 531988097, u'NIndexBlockWrites': 11230, u'NBytesIncoming': 844478818, u'NSyncs': 1953, u'NInserts': 1294111, u'NWriteIOs': 8896}}, u'NReadBytes': 2251217799L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 288.8999938964844, u'NInternalFragmentationCompacts': 25, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 13874494, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 4096, u'NTxns': 171048, u'NWriteBytes': 1957945344, u'NSegments': 167, u'NSyncs': 171047, u'NLoggedBytes': 1957986304, u'NActiveSegments': 5, u'NWrites': 171884, u'PreallocateSupported': True, u'NInserts': 2594343, u'DiskUsed': 83857408}, u'NGets': 175253, u'NReadBytesSet': 93276529, u'NReadBytesCompact': 850689217, u'IndexResidentRatio': 0.10880175232887268, u'CompactQueueSize': 0, u'NSyncsPerSec': 4.0, u'BlockCacheHitRatio': 57.16250991821289, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 1380197453, u'BloomFilterMemUsedRatio': 0.017546996474266052, u'NonResidentBloomFilterSize': 1095852, u'NWriteBytes': 3458319597L, u'BytesOutgoing': 61996, u'RecentReadIOAmp': 3.224724054336548, u'WriteAmp': 2.1977062225341797, u'NBlocksCached': 3036, u'LogicalDataSize': 1101411020, u'ActiveBloomFilterMemUsed': 1827504, u'NFileCountCompacts': 595, u'NTableFiles': 2759, u'FlushQueueSize': 8, u'TableObjectMemUsed': 846960, u'BlockCacheQuota': 83352118, u'TotalDiskUsage': 2422587392L, u'NPurgerRuns': 0, u'ReadIOAmpSet': 0.0, u'NSyncs': 5859, u'NReadIOsPerSec': 737.7333374023438, u'NReadIOsSetPerSec': 28.899999618530273, u'NTablesDeleted': 11499, u'BloomFilterFPR': 0.016599509865045547, u'NBloomFilterHitsPerSec': 6202.533203125, u'WriteCacheMemUsed': 10045896, u'RecentBloomFilterFPR': 0.016558408737182617, u'BytesPerRead': 7343.2177734375, u'BytesIncomingPerSec': 1151875.75, u'NReadIOsGetPerSec': 477.3666687011719, u'NumKVStores': 27, u'NGetStats': 2050, u'TotalIndexBlocksSize': 18186613, u'NGetStatsPerSec': 0.9666666388511658, u'NWriteBytesCompact': 856256512, u'NReadIOsSet': 32520, u'ActiveDataSize': 1101411020, u'NWriteBytesCompactPerSec': 824251.75, u'NBloomFilterMissesPerSec': 104.43333435058594, u'NCommitBatches': 171663, u'NGetStatsComputed': 2050, u'NWriteIOs': 194268, u'NReadIOsGet': 486485, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 20838030, u'LSMTreeObjectMemUsed': 465696, u'NSetsPerSec': 1127.6666259765625, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 2.5999999046325684, u'ManagedFileObjectMemUsed': 277824, u'ReadAmpGet': 20758.12890625, u'BlockCacheMemUsed': 10515560, u'NWriterCompacts': 0, u'NReadIOs': 856459, u'WALBufferMemUsed': 81920, u'ReadAmp': 34807.75, u'keyStats': {u'NItemsTransformed': 14352850, u'NTablesCreated': 3844, u'ReadIOAmp': 0.32930170983685525, u'MemtablesMemUsed': 825496, u'NBloomFilterHits': 7371566, u'TotalIndexBlocksSize': 2081313, u'CheckpointOverhead': 64303104, u'NItemsGet': 1459379, u'NWriteIOs': 6905, u'NReadBytes': 1931908378, u'LevelStats': [{u'TargetSize': 113246208, u'LogicalSize': 4580685, u'SizeRatio': 0.040448904037475586, u'NWriteBytes': 79720448, u'BloomFilterMemUsed': 57672, u'NumTables': 190, u'WastedSpace': 76, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 4214784}, {u'TargetSize': 113246208, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 113246208, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 0}, {u'TargetSize': 113246208, u'LogicalSize': 13238618, u'SizeRatio': 0.11690120343808774, u'NWriteBytes': 66400256, u'BloomFilterMemUsed': 253952, u'NumTables': 8, u'WastedSpace': 32422, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 13271040}, {u'TargetSize': 113246208, u'LogicalSize': 79472631, u'SizeRatio': 0.7017685837215848, u'NWriteBytes': 564592640, u'BloomFilterMemUsed': 1511424, u'NumTables': 46, u'WastedSpace': 170294, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 79642624}], u'NItemsGarbageCollected': 952923, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 1233301, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 55872, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 676270, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 345481, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 177387, u'CompactWriter': {u'NDataBlockWrites': 220459, u'NWriteBytes': 642498560, u'NIndexBlockWrites': 3131, u'NBytesIncoming': 832851880, u'NSyncs': 0, u'NInserts': 14352850, u'NWriteIOs': 1508}, u'NonResidentBloomFilterSize': 1091029, u'IteratorMemUsed': 0, u'NWriteBytes': 725328116, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 480576, u'NReadBytes': 1378762123, u'NDataBlockReads': 2203220, u'NBytesOutgoing': 0, u'NIndexBlockReads': 2806886}, u'NBloomFilterMisses': 124146, u'NItemsMerged': 0, u'LogicalDiskSize': 80572253, u'TotalBloomFilterMemUsed': 1893208, u'WriteAmp': 10.083680543922089, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1823048, u'LogicalDataSize': 80572253, u'NTableFiles': 582, u'NFileCountCompacts': 198, u'TableObjectMemUsed': 139680, u'TreeSnapshotMemoryUsed': 151184, u'TotalDiskUsage': 165150720, u'BytesIncoming': 71930890, u'NItemsDel': 0, u'PendingCheckpointBytes': 7368704, u'NCompacts': 200, u'NItemsPut': 1441012, u'NSyncs': 1953, u'CompactReader': {u'NReadIOs': 195694, u'NReadBytes': 553146255, u'NDataBlockReads': 228818, u'NBytesOutgoing': 0, u'NIndexBlockReads': 6442}, u'ActiveDiskUsage': 97128448, u'NTablesDeleted': 3790, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.016562269201378068, u'Writer': {u'NDataBlockWrites': 15712, u'NWriteBytes': 82829556, u'NIndexBlockWrites': 3448, u'NBytesIncoming': 52242062, u'NSyncs': 1953, u'NInserts': 887493, u'NWriteIOs': 5397}}, u'NReadBytesCompactPerSec': 711454.1875, u'TableMetaMemUsed': 832283, u'NDeletes': 0, u'BlockCacheHitsPerSec': 364.5, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 124442, u'LogicalDiskSize': 1858940906, u'NSets': 1287650, u'TotalBloomFilterMemUsed': 1897664, u'BufferMemUsed': 1149952, u'RecentReadAmp': 0.0, u'NGetsPerSec': 148.03334045410156, u'NInserts': 153362, u'NCommitBatchesPerSec': 76.33333587646484, u'FragmentationWithOverhead': 0.5453575253486633, u'NCompactsPerSec': 0.6000000238418579, u'WALMemUsed': 147536, u'MemoryQuota': 108147512, u'NReadBytesPerSec': 2106931.25, u'TreeSnapshotMemoryUsed': 520944, u'BytesIncoming': 1573604203, u'NCompacts': 808, u'NumPersistentFlushes': 1818, u'NWriteIOsPerSec': 90.69999694824219, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 9426.775390625, u'ActiveDiskUsage': 1894174720, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 0.9666666388511658, u'NReadBytesSetPerSec': 84101.703125, u'BlockCacheHits': 447906, u'NWriteBytesPerSec': 2579205.75, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 3660, u'ReadIOAmp': 1.2330415754923414, u'MemtablesMemUsed': 1448088, u'NBloomFilterHits': 719, u'TotalIndexBlocksSize': 28745, u'CheckpointOverhead': 2752512, u'NItemsGet': 1828, u'NWriteIOs': 5613, u'NReadBytes': 6873339, u'LevelStats': [{u'TargetSize': 113246208, u'LogicalSize': 2317391, u'SizeRatio': 0.020463298868249963, u'NWriteBytes': 27607040, u'BloomFilterMemUsed': 3592, u'NumTables': 227, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 1859584}, {u'TargetSize': 113246208, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 113246208, u'LogicalSize': 286398, u'SizeRatio': 0.002528985341389974, u'NWriteBytes': 1622016, u'BloomFilterMemUsed': 864, u'NumTables': 27, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 221184}], u'NItemsGarbageCollected': 636889, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 12441, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 56640, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 9034, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 169292, u'CompactWriter': {u'NDataBlockWrites': 202, u'NWriteBytes': 1654784, u'NIndexBlockWrites': 202, u'NBytesIncoming': 625780, u'NSyncs': 0, u'NInserts': 4646, u'NWriteIOs': 202}, u'NonResidentBloomFilterSize': 4823, u'IteratorMemUsed': 0, u'NWriteBytes': 30954872, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 2254, u'NReadBytes': 786032, u'NDataBlockReads': 1314, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1314}, u'NBloomFilterMisses': 296, u'NItemsMerged': 0, u'LogicalDiskSize': 286398, u'TotalBloomFilterMemUsed': 4456, u'WriteAmp': 0.1258388542962164, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 4456, u'LogicalDataSize': 286398, u'NTableFiles': 590, u'NFileCountCompacts': 202, u'TableObjectMemUsed': 141600, u'TreeSnapshotMemoryUsed': 111952, u'TotalDiskUsage': 4833280, u'BytesIncoming': 245988190, u'NItemsDel': 0, u'PendingCheckpointBytes': 49152, u'NCompacts': 202, u'NItemsPut': 640569, u'NSyncs': 1953, u'CompactReader': {u'NReadIOs': 6780, u'NReadBytes': 6087307, u'NDataBlockReads': 3637, u'NBytesOutgoing': 0, u'NIndexBlockReads': 3637}, u'ActiveDiskUsage': 2080768, u'NTablesDeleted': 3637, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.2916256157635468, u'Writer': {u'NDataBlockWrites': 3458, u'NWriteBytes': 29300088, u'NIndexBlockWrites': 3458, u'NBytesIncoming': 5660752, u'NSyncs': 1953, u'NInserts': 36478, u'NWriteIOs': 5411}}}}} 2022-02-08 04:29:06,894 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:728] magma stats fragmentation result {'172.23.121.123': [0.39364928007125854, 0.4011293053627014, 0.38481301069259644, 0.38482776284217834, 0.3938903510570526, 0.35383838415145874, 0.3783632218837738, 0.3856278359889984], '172.23.121.124': [0.382883757352829, 0.3818452060222626, 0.364938348531723, 0.3831145167350769, 0.35684892535209656, 0.3861791491508484, 0.37695902585983276, 0.3724844455718994], '172.23.121.135': [0.4053623676300049, 0.4088839292526245, 0.3970124125480652, 0.4211713373661041, 0.40143653750419617, 0.41145333647727966, 0.4168134331703186, 0.4115375876426697], '172.23.121.136': [0.1151898130774498, 0.13816016912460327, 0.13930846750736237, 0.1382729560136795, 0.13057661056518555, 0.14427410066127777, 0.12969687581062317, 0.12074903398752213], '172.23.121.126': [0.36941400170326233, 0.3874751627445221, 0.3879754841327667, 0.3670428395271301, 0.37104353308677673, 0.3683944046497345, 0.3564731180667877, 0.3988252580165863], '172.23.121.115': [0.019265804439783096, 0.02079244889318943, 0.021087121218442917, 0.03523877635598183, 0.037108149379491806, 0.053921591490507126, 0.03213470056653023, 0.03010953776538372], '172.23.121.116': [0.40750613808631897, 0.4153549373149872, 0.4049360156059265, 0.40863293409347534, 0.40110304951667786, 0.3903709650039673, 0.4072185158729553, 0.38296666741371155], '172.23.121.127': [0.36464738845825195, 0.3705489933490753, 0.38137760758399963, 0.3875177204608917, 0.3712865710258484, 0.4159751236438751, 0.3723764419555664, 0.36919930577278137], '172.23.121.129': [0.41104328632354736, 0.3759550452232361, 0.3848194181919098, 0.3423527181148529, 0.36790239810943604, 0.3696596026420593, 0.3508308529853821, 0.3531632125377655], '172.23.120.170': [0.36598020792007446, 0.39233940839767456, 0.3582901954650879, 0.3469120264053345, 0.3900175094604492, 0.3882995545864105, 0.3835659921169281, 0.3701974153518677]} 2022-02-08 04:29:07,338 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.116 is 35.0 2022-02-08 04:29:07,816 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.123 is 35.0 2022-02-08 04:29:08,144 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.124 is 35.0 2022-02-08 04:29:08,411 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.126 is 35.0 2022-02-08 04:29:08,874 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.127 is 35.0 2022-02-08 04:29:09,230 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.129 is 35.0 2022-02-08 04:29:09,582 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.135 is 35.0 2022-02-08 04:29:09,910 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.115 is 35.0 2022-02-08 04:29:10,358 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.136 is 35.0 2022-02-08 04:29:10,744 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.120.170 is 35.0 2022-02-08 04:29:10,746 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:677] KV stats fragmentation values {'172.23.121.123': 35.0, '172.23.121.124': 35.0, '172.23.121.135': 35.0, '172.23.121.136': 35.0, '172.23.121.126': 35.0, '172.23.121.115': 35.0, '172.23.121.116': 35.0, '172.23.121.127': 35.0, '172.23.121.129': 35.0, '172.23.120.170': 35.0} ############################################################ # # Step 9: Crash Magma/memc with Loading of docs # ############################################################ ############################################################ # # RAM FootPrint: 4179.73828125 # ############################################################ 2022-02-08 04:29:11,046 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 04:29:11,046 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 92 seconds. Reason: Iteration:0 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 6417 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:30:43,111 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 04:30:43,122 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 04:30:48,141 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:30:48,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:30:48,474 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:30:48,565 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:30:48,566 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:30:48,581 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:30:48,594 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:30:48,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:30:48,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:30:48,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:30:48,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:30:48,684 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:30:48,686 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:30:48,734 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['118352 memcached\n'], []) 2022-02-08 04:30:48,734 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 04:30:48,762 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:30:48,786 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['118352 memcached\n'], []) 2022-02-08 04:30:48,786 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:30:48,789 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:30:48,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:30:49,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:30:49,226 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:30:49,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:30:49,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:30:49,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:30:49,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:30:49,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:30:49,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:30:49,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 5 queries submitted, 0 failed, 6438 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:30:49,321 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:30:49,321 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:30:49,377 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['28941 memcached\n'], []) 2022-02-08 04:30:49,377 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 04:30:49,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:30:49,420 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['28941 memcached\n'], []) 2022-02-08 04:30:49,420 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:30:49,421 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:30:49,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:30:49,724 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:30:49,795 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:30:49,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:30:49,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:30:49,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:30:49,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:30:49,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:30:49,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:30:49,890 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:30:49,904 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:30:49,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:30:49,960 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['17290 memcached\n'], []) 2022-02-08 04:30:49,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 04:30:49,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:30:50,009 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['17290 memcached\n'], []) 2022-02-08 04:30:50,009 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:30:50,010 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:30:50,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:30:50,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:30:50,369 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:30:50,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:30:50,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:30:50,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:30:50,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:30:50,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:30:50,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:30:50,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:30:50,493 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:30:50,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:30:50,549 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['52485 memcached\n'], []) 2022-02-08 04:30:50,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 04:30:50,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:30:50,598 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['52485 memcached\n'], []) 2022-02-08 04:30:50,598 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:30:50,601 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:30:50,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:30:50,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:30:50,954 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:30:50,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:30:50,969 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:30:50,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:30:51,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:30:51,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:30:51,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:30:51,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:30:51,062 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:30:51,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:30:51,138 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['46287 memcached\n'], []) 2022-02-08 04:30:51,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 04:30:51,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:30:51,188 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['46287 memcached\n'], []) 2022-02-08 04:30:51,190 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:30:51,190 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:30:51,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:30:51,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:30:51,536 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:30:51,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:30:51,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:30:51,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:30:51,576 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:30:51,594 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:30:51,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:30:51,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:30:51,632 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:30:51,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:30:51,674 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['81186 memcached\n'], []) 2022-02-08 04:30:51,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 04:30:51,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:30:51,721 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['81186 memcached\n'], []) 2022-02-08 04:30:51,723 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:30:51,724 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:30:51,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:30:52,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:30:52,065 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:30:52,066 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:30:52,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:30:52,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:30:52,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:30:52,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:30:52,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:30:52,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:30:52,164 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:30:52,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:30:52,223 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['15018 memcached\n'], []) 2022-02-08 04:30:52,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 04:30:52,249 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:30:52,269 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['15018 memcached\n'], []) 2022-02-08 04:30:52,269 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:30:52,270 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:30:52,349 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:30:52,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:30:52,642 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:30:52,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:30:52,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:30:52,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:30:52,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:30:52,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:30:52,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:30:52,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:30:52,744 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:30:52,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:30:52,888 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['23106 memcached\n'], []) 2022-02-08 04:30:52,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 04:30:52,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:30:52,944 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['23106 memcached\n'], []) 2022-02-08 04:30:52,944 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:30:52,946 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:30:53,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:30:53,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:30:53,296 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:30:53,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:30:53,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:30:53,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:30:53,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:30:53,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:30:53,364 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:30:53,377 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:30:53,391 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:30:53,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:30:53,434 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['31079 memcached\n'], []) 2022-02-08 04:30:53,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 04:30:53,464 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:30:53,493 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['31079 memcached\n'], []) 2022-02-08 04:30:53,493 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:30:53,494 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:30:53,581 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:30:53,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:30:53,861 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:30:53,862 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:30:53,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:30:53,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:30:53,905 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:30:53,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:30:53,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:30:53,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:30:53,963 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:30:53,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:30:54,006 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['98397 memcached\n'], []) 2022-02-08 04:30:54,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 04:30:54,035 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:30:54,055 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['98397 memcached\n'], []) 2022-02-08 04:30:54,055 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:30:54,058 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:30:54,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:30:54,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:30:54,428 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:30:54,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:30:54,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:30:54,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:30:54,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:30:54,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:30:54,499 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:30:54,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:30:54,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:30:54,542 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 04:30:54,566 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:30:54,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:30:54,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:30:55,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:30:55,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:30:55,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:30:55,753 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 04:30:55,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:30:55,767 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:30:55,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:30:55,795 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 04:30:55,796 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:30:55,798 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:30:55,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:30:56,089 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:30:56,148 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:30:56,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:30:56,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:30:56,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:30:56,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:30:56,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:30:56,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:30:56,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:30:56,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:30:56,252 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 04:30:56,278 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:30:56,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:30:56,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:30:56,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:30:56,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:30:56,367 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:30:56,381 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 04:30:56,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:30:56,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:30:56,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:30:56,420 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 04:30:56,420 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:30:56,423 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:30:56,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:30:56,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:30:56,779 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:30:56,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:30:56,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:30:56,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:30:56,816 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:30:56,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:30:56,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:30:56,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:30:56,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:30:56,888 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 04:30:56,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:30:56,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:30:57,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:30:57,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:30:57,900 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:30:58,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:30:58,138 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 04:30:58,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:30:58,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:30:58,168 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:30:58,184 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 04:30:58,184 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:30:58,186 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 04:30:58,280 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:30:58,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 04:30:58,565 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:30:58,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 04:30:58,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 04:30:58,594 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 04:30:58,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 04:30:58,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 04:30:58,637 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 04:30:58,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 04:30:58,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:30:58,677 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 04:30:58,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:30:58,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:30:58,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:30:58,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:30:58,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:30:58,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:30:58,798 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 04:30:58,798 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:30:58,811 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:30:58,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:30:58,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:30:58,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:30:58,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:30:58,931 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:30:58,969 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:30:58,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:30:59,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:30:59,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:30:59,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:30:59,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:30:59,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:30:59,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:30:59,196 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:30:59,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:30:59,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:30:59,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:30:59,255 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:30:59,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:30:59,285 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:30:59,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:30:59,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:30:59,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:30:59,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:30:59,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:30:59,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:30:59,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:30:59,405 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:30:59,420 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:30:59,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:30:59,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:30:59,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:30:59,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:30:59,494 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:30:59,512 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:30:59,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:30:59,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:30:59,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:30:59,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:30:59,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:30:59,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:30:59,632 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 04:30:59,634 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:30:59,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:30:59,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:31:00,032 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:31:00,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:31:00,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:31:00,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:31:00,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:31:00,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:31:00,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:31:00,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:31:00,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:31:00,138 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 04:31:00,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:31:00,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:00,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:31:00,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:00,727 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:00,844 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:31:00,858 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 04:31:00,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:31:00,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:00,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:31:00,904 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 04:31:00,904 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:31:00,910 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 04:31:00,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:31:01,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 04:31:01,282 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:31:01,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 04:31:01,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 04:31:01,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 04:31:01,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 04:31:01,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 04:31:01,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 04:31:01,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 04:31:01,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:31:01,394 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 04:31:01,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:31:01,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:01,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:31:01,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:01,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:01,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:31:01,510 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 04:31:01,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:31:01,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:01,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:31:01,553 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:01,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:31:01,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:01,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:01,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:01,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:01,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:01,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:01,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:01,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:01,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:01,931 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:31:01,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:01,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:01,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:01,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:02,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:02,028 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:02,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:02,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:02,073 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:02,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:31:02,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:02,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:02,147 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:02,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:02,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:02,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:02,216 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:02,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:02,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:02,262 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:31:02,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:02,295 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:02,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:02,326 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:02,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:02,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:02,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:02,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:02,411 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 04:31:02,413 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:31:02,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:31:02,698 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:31:02,782 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:31:02,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:31:02,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:31:02,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:31:02,821 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:31:02,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:31:02,849 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:31:02,862 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:31:02,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:31:02,895 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 04:31:02,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:31:02,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:03,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:31:03,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:03,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:04,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:31:04,187 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 04:31:04,188 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:31:04,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:04,214 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:31:04,229 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 04:31:04,229 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:31:04,232 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:31:04,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:31:04,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:31:04,615 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:31:04,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:31:04,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:31:04,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:31:04,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:31:04,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:31:04,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:31:04,711 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:31:04,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:31:04,750 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 04:31:04,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:31:04,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:05,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:31:05,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:05,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:05,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:31:05,934 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 04:31:05,936 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:31:05,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:05,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:31:05,984 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 04:31:05,984 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:31:05,987 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 04:31:06,099 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:31:06,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 04:31:06,372 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:31:06,372 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 04:31:06,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 04:31:06,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 04:31:06,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 04:31:06,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 04:31:06,441 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 04:31:06,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 04:31:06,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:31:06,483 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 04:31:06,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:31:06,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:06,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:31:06,553 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:06,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:06,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:31:06,592 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 04:31:06,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:31:06,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:06,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:31:06,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:06,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:31:06,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:06,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:06,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:06,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:06,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:06,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:06,943 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:06,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:07,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:07,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:31:07,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:07,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:07,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:07,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:07,109 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:07,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:07,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:07,158 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:07,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:07,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:31:07,209 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:07,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:07,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:07,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:07,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:07,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:07,303 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:07,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:07,334 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:07,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:31:07,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:07,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:07,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:07,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:07,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:07,441 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:07,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:07,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:07,486 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 04:31:07,489 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 04:31:07,579 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:31:07,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 04:31:07,934 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:31:07,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 04:31:07,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 04:31:07,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 04:31:07,982 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 04:31:08,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 04:31:08,015 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 04:31:08,030 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 04:31:08,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:31:08,069 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 04:31:08,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:31:08,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:08,132 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:31:08,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:08,165 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:08,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:31:08,196 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 04:31:08,196 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:31:08,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:08,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:31:08,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:08,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:31:08,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:08,352 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:08,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:08,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:08,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:08,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:08,548 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:08,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:08,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:08,640 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:31:08,658 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:08,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:08,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:08,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:08,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:08,747 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:08,765 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:08,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:08,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:08,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:31:08,834 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:08,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:08,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:08,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:08,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:08,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:08,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:08,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:08,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:08,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:31:09,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:09,038 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:09,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:09,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:09,091 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:09,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:09,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:09,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:09,167 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 04:31:09,170 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:31:09,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:31:09,522 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:31:09,581 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:31:09,581 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:31:09,594 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:31:09,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:31:09,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:31:09,635 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:31:09,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:31:09,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:31:09,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:31:09,687 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 04:31:09,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:31:09,724 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:10,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:31:10,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:10,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:10,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:31:10,888 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 04:31:10,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:31:10,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:10,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:31:10,931 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 04:31:10,931 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:31:10,934 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:31:11,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:31:11,229 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:31:11,286 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:31:11,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:31:11,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:31:11,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:31:11,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:31:11,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:31:11,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:31:11,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:31:11,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:31:11,391 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 04:31:11,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:31:11,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:11,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:31:11,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:12,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:12,275 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:31:12,290 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 04:31:12,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:31:12,303 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:12,318 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:31:12,331 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 04:31:12,331 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:31:12,334 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:31:12,446 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:31:12,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:31:12,796 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:31:12,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:31:12,811 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:31:12,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:31:12,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:31:12,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:31:12,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:31:12,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:31:12,936 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:31:12,951 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 04:31:12,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:31:12,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:13,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:31:13,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:13,706 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:13,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:31:13,880 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 04:31:13,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:31:13,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:13,920 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:31:13,934 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 04:31:13,934 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:31:13,938 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:31:14,038 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:31:14,260 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:31:14,335 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:31:14,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:31:14,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:31:14,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:31:14,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:31:14,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:31:14,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:31:14,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:31:14,431 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:31:14,446 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 04:31:14,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:31:14,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:14,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:31:15,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:15,533 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:15,778 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:31:15,792 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 04:31:15,793 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:31:15,806 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:15,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:31:15,832 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 04:31:15,832 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:31:15,835 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 04:31:15,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:31:16,137 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 04:31:16,198 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:31:16,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 04:31:16,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 04:31:16,229 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 04:31:16,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 04:31:16,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 04:31:16,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 04:31:16,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 04:31:16,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:31:16,319 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 04:31:16,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:31:16,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:16,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:31:16,390 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:16,403 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:31:16,415 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:31:16,428 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 04:31:16,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:31:16,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:31:16,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:31:16,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:16,507 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:31:16,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:16,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:16,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:16,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:16,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:16,739 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:16,775 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:16,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:31:16,844 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:16,859 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:31:16,877 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:16,892 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:16,910 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:16,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:16,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:16,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:16,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:17,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:31:17,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:17,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:31:17,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:17,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:17,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:17,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:17,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:17,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:17,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:17,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:31:17,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:17,216 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:31:17,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:17,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:17,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:17,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:17,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:17,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:17,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:17,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:31:17,374 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 04:31:17,375 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:31:17,377 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:31:17,454 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:31:17,670 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:31:17,730 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:31:17,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:31:17,744 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:31:17,757 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:31:17,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:31:17,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:31:17,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:31:17,821 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:31:17,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:17,976 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:17,976 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 04:31:19,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:20,119 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:20,121 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 04:31:22,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:22,444 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:22,444 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 04:31:24,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:24,585 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:24,585 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 04:31:26,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:26,747 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:26,747 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 04:31:28,747 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:28,913 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:28,913 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 04:31:30,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:31,124 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:31,125 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 04:31:33,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:33,267 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:31:33,269 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 15.8940000534. 2022-02-08 04:31:33,269 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:31:33,270 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:31:33,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:31:33,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:31:33,677 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:31:33,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:31:33,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:31:33,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:31:33,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:31:33,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:31:33,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:31:33,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:31:33,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:33,957 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:33,959 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:31:35,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:36,102 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:36,104 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:31:38,104 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:38,269 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:38,269 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:31:40,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:40,423 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:40,423 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:31:42,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:42,641 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:42,641 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:31:44,642 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:44,788 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:44,788 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:31:46,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:46,940 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:46,940 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:31:48,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:49,124 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:49,124 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 6448 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:31:51,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:51,382 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:51,382 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:31:53,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:53,644 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:53,645 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:31:55,645 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:55,983 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:55,983 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:31:57,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:31:58,210 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:31:58,210 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:32:00,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:00,683 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:32:00,683 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:32:02,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:03,135 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:32:03,137 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:32:05,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:05,526 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:32:05,526 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:32:07,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:07,732 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:32:07,732 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:32:09,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:10,164 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:32:10,165 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:32:12,165 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:12,420 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:32:12,420 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:32:14,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:14,627 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:32:14,627 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:32:16,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:16,900 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:32:16,900 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:32:18,900 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:19,138 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:32:19,138 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:32:21,140 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:21,338 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:32:21,339 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:32:23,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:23,483 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:32:23,483 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:32:25,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:25,628 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:32:25,628 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:32:27,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:27,773 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:32:27,773 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 54.504999876. 2022-02-08 04:32:27,773 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:32:27,776 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:32:28,252 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:32:28,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:32:28,615 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:32:28,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:32:28,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:32:28,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:32:28,670 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:32:28,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:32:28,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:32:28,727 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:32:28,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:28,976 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:32:28,976 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 04:32:30,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:31,111 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:32:31,112 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.33899998665. 2022-02-08 04:32:31,112 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:32:31,115 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:32:31,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:32:31,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:32:32,082 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:32:32,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:32:32,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:32:32,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:32:32,124 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:32:32,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:32:32,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:32:32,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:32:32,191 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:32,467 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:32:32,467 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.35500001907. 2022-02-08 04:32:32,469 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:32:32,470 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:32:32,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:32:32,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:32:32,809 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:32:32,811 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:32:32,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:32:32,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:32:32,846 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:32:32,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:32:32,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:32:32,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:32:32,897 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:33,279 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:32:33,279 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.8109998703. 2022-02-08 04:32:33,279 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:32:33,282 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:32:33,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:32:34,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:32:34,161 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:32:34,163 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:32:34,176 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:32:34,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:32:34,209 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:32:34,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:32:34,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:32:34,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:32:34,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:34,519 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:32:34,520 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.24099993706. 2022-02-08 04:32:34,520 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:32:34,523 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:32:35,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:32:35,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:32:35,573 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:32:35,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:32:35,594 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:32:35,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:32:35,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:32:35,645 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:32:35,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:32:35,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:32:35,701 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:36,480 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:32:36,480 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.96000003815. 2022-02-08 04:32:36,482 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:32:36,483 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:32:36,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:32:36,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:32:36,864 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:32:36,864 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:32:36,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:32:36,890 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:32:36,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:32:36,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:32:36,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:32:36,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:32:36,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:37,601 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:32:37,602 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.12099981308. 2022-02-08 04:32:37,604 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:32:37,605 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:32:38,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:32:38,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:32:38,707 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:32:38,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:32:38,721 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:32:38,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:32:38,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:32:38,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:32:38,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:32:38,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:32:38,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:40,954 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:32:40,954 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.35099983215. 2022-02-08 04:32:40,954 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:32:40,957 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:32:41,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:32:41,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:32:41,542 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:32:41,543 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:32:41,558 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:32:41,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:32:41,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:32:41,602 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:32:41,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:32:41,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:32:41,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:32:43,716 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:32:43,716 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.7610001564. ############################################################ # # RAM FootPrint: 4179.25 # ############################################################ 2022-02-08 04:32:43,950 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 04:32:43,950 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 92 seconds. Reason: Iteration:1 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 6457 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 6478 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:34:15,956 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 04:34:15,957 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 04:34:20,960 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:34:21,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:21,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:34:21,858 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:21,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:34:21,877 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:34:21,900 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:34:21,918 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:34:21,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:34:21,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:34:21,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:34:22,029 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:34:22,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:34:22,098 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['123117 memcached\n'], []) 2022-02-08 04:34:22,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 04:34:22,140 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:34:22,165 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['123117 memcached\n'], []) 2022-02-08 04:34:22,165 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:34:22,167 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:34:22,665 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:22,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:34:23,071 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:23,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:34:23,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:34:23,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:34:23,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:34:23,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:34:23,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:34:23,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:34:23,198 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:34:23,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:34:23,259 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['33869 memcached\n'], []) 2022-02-08 04:34:23,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 04:34:23,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:34:23,312 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['33869 memcached\n'], []) 2022-02-08 04:34:23,312 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:34:23,315 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:34:23,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:23,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:34:23,726 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:23,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:34:23,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:34:23,755 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:34:23,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:34:23,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:34:23,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:34:23,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:34:23,831 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:34:23,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:34:23,874 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['22406 memcached\n'], []) 2022-02-08 04:34:23,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 04:34:23,897 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:34:23,918 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['22406 memcached\n'], []) 2022-02-08 04:34:23,918 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:34:23,921 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:34:23,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:24,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:34:24,267 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:24,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:34:24,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:34:24,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:34:24,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:34:24,334 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:34:24,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:34:24,367 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:34:24,378 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:34:24,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:34:24,447 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['57241 memcached\n'], []) 2022-02-08 04:34:24,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 04:34:24,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:34:24,493 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['57241 memcached\n'], []) 2022-02-08 04:34:24,494 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:34:24,496 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:34:24,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:24,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:34:24,858 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:24,859 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:34:24,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:34:24,885 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:34:24,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:34:24,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:34:24,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:34:24,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:34:24,953 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:34:24,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:34:25,000 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['51050 memcached\n'], []) 2022-02-08 04:34:25,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 04:34:25,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:34:25,052 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['51050 memcached\n'], []) 2022-02-08 04:34:25,052 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:34:25,055 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:34:25,132 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:25,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:34:25,407 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:25,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:34:25,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:34:25,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:34:25,454 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:34:25,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:34:25,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:34:25,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:34:25,515 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:34:25,515 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:34:25,568 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['85971 memcached\n'], []) 2022-02-08 04:34:25,568 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 04:34:25,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:34:25,617 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['85971 memcached\n'], []) 2022-02-08 04:34:25,617 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:34:25,619 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:34:25,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:25,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:34:26,017 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:26,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:34:26,030 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:34:26,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:34:26,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:34:26,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:34:26,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:34:26,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:34:26,105 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:34:26,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:34:26,148 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['19799 memcached\n'], []) 2022-02-08 04:34:26,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 04:34:26,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:34:26,194 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['19799 memcached\n'], []) 2022-02-08 04:34:26,194 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:34:26,196 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:34:26,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:26,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:34:26,592 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:26,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:34:26,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:34:26,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:34:26,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:34:26,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:34:26,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:34:26,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:34:26,687 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:34:26,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:34:26,782 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['34726 memcached\n'], []) 2022-02-08 04:34:26,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 04:34:26,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:34:26,828 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['34726 memcached\n'], []) 2022-02-08 04:34:26,828 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:34:26,831 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:34:26,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:27,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:34:27,210 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:27,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:34:27,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:34:27,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:34:27,247 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:34:27,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:34:27,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:34:27,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:34:27,308 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:34:27,308 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:34:27,355 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['70749 memcached\n'], []) 2022-02-08 04:34:27,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 04:34:27,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:34:27,401 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['70749 memcached\n'], []) 2022-02-08 04:34:27,401 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:34:27,404 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:34:27,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:27,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:34:27,765 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:27,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:34:27,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:34:27,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:34:27,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:34:27,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:34:27,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:34:27,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:34:27,865 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:34:27,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:34:27,911 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['100405 memcached\n'], []) 2022-02-08 04:34:27,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 04:34:27,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:34:27,960 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['100405 memcached\n'], []) 2022-02-08 04:34:27,960 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:34:27,966 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:34:28,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:28,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:34:28,326 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:28,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:34:28,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:34:28,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:34:28,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:34:28,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:34:28,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:34:28,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:34:28,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:34:28,457 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 04:34:28,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:34:28,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:28,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:34:29,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:29,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:29,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:34:29,796 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 04:34:29,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:34:29,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:29,829 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:34:29,845 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 04:34:29,845 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:34:29,848 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:34:29,933 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:30,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:34:30,226 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:30,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:34:30,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:34:30,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:34:30,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:34:30,283 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:34:30,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:34:30,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:34:30,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:34:30,336 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 04:34:30,362 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:34:30,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:30,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:34:30,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:30,474 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:30,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:34:30,509 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 04:34:30,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:34:30,522 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:30,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:34:30,548 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 04:34:30,549 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:34:30,549 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:34:30,635 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:30,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:34:30,908 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:30,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:34:30,923 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:34:30,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:34:30,969 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:34:30,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:34:30,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:34:31,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:34:31,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:34:31,036 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 04:34:31,061 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:34:31,073 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:31,446 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:34:31,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:32,048 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:32,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:34:32,289 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 04:34:32,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:34:32,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:32,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:34:32,329 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 04:34:32,329 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:34:32,332 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 04:34:32,415 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:32,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 04:34:32,688 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:32,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 04:34:32,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 04:34:32,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 04:34:32,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 04:34:32,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 04:34:32,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 04:34:32,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 04:34:32,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:34:32,802 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 04:34:32,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:34:32,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:32,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:34:32,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:32,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:32,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:34:32,917 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 04:34:32,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:34:32,931 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:32,946 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:34:32,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:32,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:34:33,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:33,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:33,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:33,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:33,176 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:33,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:33,247 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:33,285 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:33,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:33,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:34:33,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:33,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:33,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:33,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:33,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:33,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:33,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:33,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:33,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:33,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:34:33,499 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:33,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:33,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:33,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:33,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:33,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:33,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:33,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:33,635 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:33,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:34:33,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:33,691 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:33,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:33,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:33,737 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:33,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:33,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:33,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:33,796 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 04:34:33,799 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:34:33,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:34,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:34:34,177 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:34,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:34:34,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:34:34,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:34:34,216 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:34:34,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:34:34,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:34:34,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:34:34,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:34:34,282 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 04:34:34,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:34:34,321 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:34,535 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:34:34,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:34,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:35,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:34:35,046 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 04:34:35,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:34:35,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:35,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:34:35,085 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 04:34:35,085 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:34:35,086 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 04:34:35,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:35,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 04:34:35,444 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:35,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 04:34:35,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 04:34:35,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 04:34:35,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 04:34:35,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 04:34:35,515 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 04:34:35,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 04:34:35,540 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:34:35,555 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 04:34:35,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:34:35,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:35,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:34:35,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:35,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:35,655 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:34:35,668 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 04:34:35,668 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:34:35,681 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:35,696 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:34:35,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:35,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:34:35,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:35,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:35,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:35,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:35,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:35,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:35,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:36,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:36,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:36,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:34:36,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:36,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:36,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:36,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:36,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:36,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:36,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:36,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:36,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:36,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:34:36,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:36,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:36,270 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:36,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:36,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:36,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:36,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:36,346 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:36,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:36,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:34:36,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:36,415 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:36,433 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:36,448 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:36,464 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:36,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:36,499 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:36,515 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:36,529 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 04:34:36,532 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:34:36,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:36,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:34:36,877 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:36,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:34:36,890 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:34:36,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:34:36,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:34:36,931 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:34:36,943 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:34:36,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:34:36,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:34:36,983 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 04:34:37,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:34:37,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:37,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:34:37,839 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:38,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:38,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:34:38,358 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 04:34:38,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:34:38,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:38,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:34:38,397 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 04:34:38,397 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:34:38,398 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:34:38,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:38,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:34:38,776 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:38,778 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:34:38,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:34:38,806 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:34:38,821 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:34:38,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:34:38,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:34:38,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:34:38,880 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:34:38,895 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 04:34:38,920 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:34:38,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:39,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:34:39,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:39,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:40,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:34:40,128 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 04:34:40,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:34:40,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:40,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:34:40,176 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 04:34:40,176 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:34:40,177 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 04:34:40,252 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:40,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 04:34:40,542 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:40,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 04:34:40,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 04:34:40,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 04:34:40,584 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 04:34:40,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 04:34:40,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 04:34:40,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 04:34:40,642 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:34:40,657 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 04:34:40,681 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:34:40,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:40,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:34:40,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:40,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:40,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:34:40,773 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 04:34:40,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:34:40,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:40,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:34:40,813 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:40,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:34:40,885 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:40,920 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:40,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:40,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:41,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:41,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:41,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:41,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:41,165 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:41,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:34:41,196 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:41,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:41,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:41,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:41,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:41,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:41,283 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:41,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:41,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:41,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:34:41,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:41,352 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:41,367 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:41,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:41,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:41,408 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:41,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:41,436 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:41,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:41,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:34:41,484 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:41,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:41,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:41,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:41,545 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:41,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:41,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:41,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:41,604 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 04:34:41,605 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 04:34:41,696 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:41,936 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 04:34:42,020 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:42,022 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 04:34:42,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 04:34:42,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 04:34:42,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 04:34:42,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 04:34:42,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 04:34:42,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 04:34:42,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:34:42,141 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 04:34:42,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:34:42,188 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:42,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:34:42,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:42,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:42,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:34:42,256 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 04:34:42,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:34:42,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:42,285 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:34:42,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:42,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:34:42,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:42,408 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:42,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:42,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:42,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:42,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:42,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:42,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:42,665 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:42,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:34:42,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:42,711 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:42,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:42,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:42,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:42,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:42,788 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:42,803 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:42,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:42,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:34:42,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:42,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:42,880 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:42,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:42,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:42,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:42,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:42,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:42,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:42,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:34:43,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:43,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:43,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:43,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:43,071 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:43,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:43,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:43,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:43,137 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 04:34:43,138 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:34:43,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:43,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:34:43,499 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:43,499 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:34:43,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:34:43,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:34:43,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:34:43,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:34:43,566 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:34:43,579 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:34:43,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:34:43,607 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 04:34:43,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:34:43,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:45,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:34:45,892 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:46,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:46,336 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:34:46,352 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 04:34:46,352 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:34:46,367 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:46,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:34:46,398 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 04:34:46,398 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:34:46,400 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:34:46,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:46,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:34:46,901 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:46,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:34:46,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:34:46,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:34:46,943 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:34:46,961 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:34:46,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:34:46,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:34:47,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:34:47,019 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 04:34:47,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:34:47,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:47,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:34:47,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:47,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:47,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:34:47,971 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 04:34:47,971 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:34:47,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:47,999 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:34:48,013 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 04:34:48,013 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:34:48,029 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:34:48,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:48,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:34:48,484 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:48,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:34:48,522 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:34:48,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:34:48,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:34:48,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:34:48,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:34:48,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:34:48,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:34:48,694 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 04:34:48,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:34:48,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:49,051 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 5 queries submitted, 0 failed, 6499 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:34:49,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:49,543 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:49,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:34:49,759 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 04:34:49,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:34:49,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:49,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:34:49,862 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 04:34:49,862 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:34:49,865 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:34:49,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:50,160 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:34:50,230 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:50,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:34:50,244 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:34:50,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:34:50,270 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:34:50,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:34:50,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:34:50,313 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:34:50,326 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:34:50,342 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 04:34:50,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:34:50,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:50,785 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:34:51,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:51,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:51,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:34:51,763 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 04:34:51,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:34:51,778 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:51,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:34:51,805 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 04:34:51,806 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:34:51,809 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 04:34:51,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:52,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 04:34:52,246 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:52,247 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 04:34:52,265 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 04:34:52,280 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 04:34:52,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 04:34:52,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 04:34:52,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 04:34:52,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 04:34:52,362 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:34:52,381 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 04:34:52,408 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:34:52,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:52,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:34:52,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:52,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:34:52,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:34:52,509 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 04:34:52,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:34:52,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:34:52,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:34:52,556 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:52,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:34:52,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:52,670 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:52,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:52,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:52,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:52,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:52,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:52,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:34:52,943 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:52,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:34:52,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:52,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:53,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:53,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:53,051 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:53,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:53,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:53,102 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:34:53,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:53,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:34:53,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:53,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:53,211 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:53,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:53,247 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:53,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:53,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:53,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:34:53,313 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:53,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:34:53,349 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:53,367 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:53,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:53,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:53,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:53,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:53,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:53,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:34:53,486 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 04:34:53,486 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:34:53,490 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:34:54,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:34:54,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:34:54,510 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:34:54,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:34:54,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:34:54,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:34:54,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:34:54,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:34:54,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:34:54,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:34:54,668 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:34:55,180 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:34:55,180 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 04:34:57,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:34:57,331 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:34:57,331 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 04:34:59,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:34:59,473 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:34:59,474 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 04:35:01,474 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:01,617 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:01,618 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 04:35:03,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:03,752 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:35:03,752 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 10.2650001049. 2022-02-08 04:35:03,753 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:35:03,755 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:35:03,839 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:35:04,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:35:04,176 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:35:04,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:35:04,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:35:04,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:35:04,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:35:04,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:35:04,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:35:04,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:35:04,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:04,436 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:04,436 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:06,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:06,696 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:06,696 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:08,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:08,844 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:08,844 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:10,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:10,997 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:10,997 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:12,999 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:13,150 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:13,150 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:15,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:15,308 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:15,308 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:17,308 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:17,453 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:17,454 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:19,454 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:19,605 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:19,605 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:21,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:22,244 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:22,244 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:24,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:24,589 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:24,589 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:26,591 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:27,098 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:27,098 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:29,099 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:29,598 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:29,598 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:31,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:32,006 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:32,007 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:34,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:34,265 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:34,265 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:36,265 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:36,470 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:36,471 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:38,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:38,766 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:38,766 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:40,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:40,980 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:40,980 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:42,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:43,213 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:35:43,213 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:35:45,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:45,433 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:35:45,434 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 41.6809999943. 2022-02-08 04:35:45,434 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:35:45,486 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:35:46,362 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:35:46,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:35:47,109 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:35:47,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:35:47,135 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:35:47,165 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:35:47,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:35:47,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:35:47,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:35:47,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:35:47,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:47,890 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:35:47,890 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.45500016212. 2022-02-08 04:35:47,891 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:35:47,892 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:35:48,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:35:48,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:35:49,062 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:35:49,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:35:49,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:35:49,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:35:49,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:35:49,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:35:49,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:35:49,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:35:49,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 6501 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:35:49,549 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:35:49,549 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.65999984741. 2022-02-08 04:35:49,551 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:35:49,552 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:35:50,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:35:50,543 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:35:50,697 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:35:50,698 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:35:50,727 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:35:50,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:35:50,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:35:50,793 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:35:50,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:35:50,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:35:50,849 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:51,127 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:35:51,128 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.57699990273. 2022-02-08 04:35:51,128 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:35:51,130 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:35:51,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:35:52,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:35:52,224 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:35:52,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:35:52,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:35:52,270 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:35:52,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:35:52,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:35:52,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:35:52,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:35:52,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:52,651 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:35:52,653 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.52500009537. 2022-02-08 04:35:52,653 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:35:52,655 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:35:53,260 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:35:53,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:35:53,594 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:35:53,594 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:35:53,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:35:53,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:35:53,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:35:53,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:35:53,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:35:53,691 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:35:53,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:53,989 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:35:53,990 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.33699989319. 2022-02-08 04:35:53,990 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:35:53,992 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:35:54,545 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:35:54,798 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:35:54,898 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:35:54,900 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:35:54,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:35:54,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:35:54,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:35:54,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:35:54,982 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:35:55,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:35:55,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:55,338 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:35:55,339 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.34999990463. 2022-02-08 04:35:55,339 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:35:55,342 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:35:55,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:35:56,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:35:56,282 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:35:56,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:35:56,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:35:56,313 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:35:56,326 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:35:56,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:35:56,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:35:56,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:35:56,392 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:56,630 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:35:56,630 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.29000020027. 2022-02-08 04:35:56,631 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:35:56,632 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:35:57,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:35:57,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:35:58,039 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:35:58,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:35:58,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:35:58,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:35:58,191 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:35:58,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:35:58,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:35:58,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:35:58,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:35:58,627 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:35:58,627 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.9960000515. ############################################################ # # RAM FootPrint: 4178.2578125 # ############################################################ 2022-02-08 04:35:58,957 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 04:35:58,957 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 62 seconds. Reason: Iteration:2 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 6525 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:37:01,032 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 04:37:01,049 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 04:37:06,071 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:37:06,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:06,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:37:06,497 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:06,499 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:37:06,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:37:06,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:37:06,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:37:06,591 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:37:06,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:37:06,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:37:06,658 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:37:06,658 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:37:06,724 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['123901 memcached\n'], []) 2022-02-08 04:37:06,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 04:37:06,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:37:06,788 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['123901 memcached\n'], []) 2022-02-08 04:37:06,788 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:37:06,792 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:37:07,438 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:07,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:37:07,772 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:07,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:37:07,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:37:07,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:37:07,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:37:07,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:37:07,846 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:37:07,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:37:07,891 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:37:07,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:37:07,953 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['34859 memcached\n'], []) 2022-02-08 04:37:07,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 04:37:07,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:37:07,997 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['34859 memcached\n'], []) 2022-02-08 04:37:07,999 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:37:08,000 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:37:08,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:08,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:37:08,368 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:08,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:37:08,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:37:08,395 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:37:08,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:37:08,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:37:08,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:37:08,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:37:08,477 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:37:08,477 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:37:08,528 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['23216 memcached\n'], []) 2022-02-08 04:37:08,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 04:37:08,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:37:08,578 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['23216 memcached\n'], []) 2022-02-08 04:37:08,578 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:37:08,581 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:37:08,655 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:08,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:37:08,928 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:08,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:37:08,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:37:08,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:37:08,969 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:37:08,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:37:08,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:37:09,012 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:37:09,025 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:37:09,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:37:09,075 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['57946 memcached\n'], []) 2022-02-08 04:37:09,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 04:37:09,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:37:09,118 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['57946 memcached\n'], []) 2022-02-08 04:37:09,118 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:37:09,121 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:37:09,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:09,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:37:09,456 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:09,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:37:09,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:37:09,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:37:09,494 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:37:09,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:37:09,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:37:09,540 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:37:09,552 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:37:09,553 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:37:09,608 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['51755 memcached\n'], []) 2022-02-08 04:37:09,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 04:37:09,645 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:37:09,667 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['51755 memcached\n'], []) 2022-02-08 04:37:09,668 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:37:09,671 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:37:09,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:09,961 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:37:10,020 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:10,020 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:37:10,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:37:10,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:37:10,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:37:10,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:37:10,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:37:10,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:37:10,128 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:37:10,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:37:10,170 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['86674 memcached\n'], []) 2022-02-08 04:37:10,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 04:37:10,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:37:10,220 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['86674 memcached\n'], []) 2022-02-08 04:37:10,220 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:37:10,221 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:37:10,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:10,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:37:10,584 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:10,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:37:10,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:37:10,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:37:10,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:37:10,635 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:37:10,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:37:10,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:37:10,673 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:37:10,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:37:10,717 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['20470 memcached\n'], []) 2022-02-08 04:37:10,719 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 04:37:10,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:37:10,763 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['20470 memcached\n'], []) 2022-02-08 04:37:10,763 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:37:10,765 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:37:10,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:11,091 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:37:11,200 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:11,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:37:11,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:37:11,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:37:11,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:37:11,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:37:11,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:37:11,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:37:11,299 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:37:11,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:37:11,375 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['35431 memcached\n'], []) 2022-02-08 04:37:11,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 04:37:11,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:37:11,423 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['35431 memcached\n'], []) 2022-02-08 04:37:11,423 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:37:11,424 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:37:11,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:11,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:37:11,801 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:11,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:37:11,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:37:11,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:37:11,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:37:11,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:37:11,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:37:11,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:37:11,900 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:37:11,900 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:37:11,946 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['71452 memcached\n'], []) 2022-02-08 04:37:11,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 04:37:11,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:37:12,000 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['71452 memcached\n'], []) 2022-02-08 04:37:12,000 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:37:12,003 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:37:12,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:12,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:37:12,371 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:12,372 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:37:12,385 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:37:12,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:37:12,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:37:12,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:37:12,441 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:37:12,454 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:37:12,467 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:37:12,469 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:37:12,533 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['101106 memcached\n'], []) 2022-02-08 04:37:12,533 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 04:37:12,563 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:37:12,595 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['101106 memcached\n'], []) 2022-02-08 04:37:12,595 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:37:12,598 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:37:12,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:12,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:37:12,969 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:12,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:37:12,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:37:13,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:37:13,022 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:37:13,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:37:13,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:37:13,089 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:37:13,102 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:37:13,117 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 04:37:13,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:37:13,160 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:14,280 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:37:14,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:14,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:15,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:37:15,102 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 04:37:15,102 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:37:15,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:15,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:37:15,171 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 04:37:15,173 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:37:15,176 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:37:15,265 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:15,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:37:15,545 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:15,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:37:15,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:37:15,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:37:15,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:37:15,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:37:15,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:37:15,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:37:15,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:37:15,657 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 04:37:15,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:37:15,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:15,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:37:15,767 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:15,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:15,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:37:15,844 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 04:37:15,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:37:15,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:15,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:37:15,888 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 04:37:15,888 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:37:15,890 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:37:15,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:16,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:37:16,240 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:16,242 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:37:16,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:37:16,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:37:16,280 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:37:16,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:37:16,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:37:16,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:37:16,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:37:16,352 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 04:37:16,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:37:16,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:16,762 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:37:17,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:17,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:17,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:37:17,621 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 04:37:17,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:37:17,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:17,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:37:17,667 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 04:37:17,667 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:37:17,668 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 04:37:17,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:17,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 04:37:18,055 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:18,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 04:37:18,071 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 04:37:18,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 04:37:18,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 04:37:18,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 04:37:18,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 04:37:18,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 04:37:18,163 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:37:18,186 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 04:37:18,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:37:18,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:18,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:37:18,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:18,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:18,283 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:37:18,298 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 04:37:18,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:37:18,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:18,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:37:18,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:18,372 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:37:18,415 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:18,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:18,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:18,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:18,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:18,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:18,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:18,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:18,727 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:18,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:37:18,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:18,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:18,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:18,806 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:18,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:18,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:18,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:18,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:18,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:18,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:37:18,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:18,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:18,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:18,971 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:18,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:19,002 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:19,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:19,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:19,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:19,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:37:19,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:19,107 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:19,124 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:19,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:19,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:19,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:19,191 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:19,209 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:19,224 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 04:37:19,226 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:37:19,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:19,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:37:19,605 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:19,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:37:19,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:37:19,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:37:19,645 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:37:19,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:37:19,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:37:19,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:37:19,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:37:19,719 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 04:37:19,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:37:19,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:19,969 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:37:20,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:20,372 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:20,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:37:20,526 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 04:37:20,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:37:20,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:20,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:37:20,568 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 04:37:20,569 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:37:20,571 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 04:37:20,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:20,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 04:37:20,956 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:20,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 04:37:20,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 04:37:20,982 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 04:37:20,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 04:37:21,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 04:37:21,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 04:37:21,030 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 04:37:21,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:37:21,058 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 04:37:21,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:37:21,102 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:21,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:37:21,132 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:21,147 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:21,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:37:21,174 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 04:37:21,176 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:37:21,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:21,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:37:21,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:21,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:37:21,249 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:21,262 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:21,275 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:21,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:21,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:21,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:21,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:21,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:21,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:21,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:37:21,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:21,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:21,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:21,431 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:21,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:21,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:21,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:21,497 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:21,522 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:21,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:37:21,558 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:21,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:21,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:21,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:21,622 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:21,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:21,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:21,668 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:21,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:21,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:37:21,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:21,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:21,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:21,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:21,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:21,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:21,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:21,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:21,846 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:21,862 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:37:21,880 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:21,897 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:21,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:21,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:21,946 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:21,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:21,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:21,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:22,016 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 04:37:22,017 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:37:22,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:22,285 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:37:22,342 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:22,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:37:22,357 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:37:22,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:37:22,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:37:22,395 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:37:22,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:37:22,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:37:22,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:37:22,457 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 04:37:22,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:37:22,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:24,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:37:24,885 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:25,137 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:25,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:37:25,372 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 04:37:25,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:37:25,385 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:25,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:37:25,438 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 04:37:25,438 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:37:25,440 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:37:25,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:25,737 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:37:25,819 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:25,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:37:25,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:37:25,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:37:25,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:37:25,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:37:25,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:37:25,926 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:37:25,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:37:25,957 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 04:37:25,982 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:37:25,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:26,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:37:27,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:27,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:27,739 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:37:27,756 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 04:37:27,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:37:27,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:27,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:37:27,848 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 04:37:27,848 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:37:27,852 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 04:37:27,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:28,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 04:37:28,253 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:28,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 04:37:28,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 04:37:28,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 04:37:28,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 04:37:28,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 04:37:28,318 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 04:37:28,329 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 04:37:28,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:37:28,357 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 04:37:28,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:37:28,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:28,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:37:28,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:28,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:28,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:37:28,467 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 04:37:28,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:37:28,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:28,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:37:28,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:28,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:37:28,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:28,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:28,561 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:28,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:28,584 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:28,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:28,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:28,619 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:28,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:28,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:37:28,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:28,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:28,696 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:28,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:28,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:28,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:28,757 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:28,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:28,788 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:28,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:37:28,826 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:28,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:28,857 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:28,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:28,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:28,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:28,920 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:28,936 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:28,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:28,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:37:28,982 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:28,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:29,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:29,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:29,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:29,061 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:29,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:29,091 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:29,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:29,124 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:37:29,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:29,160 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:29,176 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:29,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:29,214 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:29,232 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:29,249 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:29,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:29,280 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 04:37:29,283 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 04:37:29,362 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:29,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 04:37:29,713 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:29,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 04:37:29,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 04:37:29,744 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 04:37:29,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 04:37:29,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 04:37:29,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 04:37:29,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 04:37:29,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:37:29,862 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 04:37:29,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:37:29,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:29,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:37:29,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:29,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:29,989 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:37:30,005 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 04:37:30,005 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:37:30,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:30,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:37:30,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:30,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:37:30,086 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:30,104 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:30,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:30,140 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:30,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:30,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:30,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:30,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:30,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:30,244 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:37:30,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:30,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:30,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:30,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:30,334 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:30,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:30,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:30,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:30,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:30,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:37:30,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:30,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:30,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:30,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:30,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:30,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:30,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:30,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:30,584 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:30,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:37:30,619 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:30,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:30,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:30,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:30,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:30,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:30,734 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:30,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:30,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:30,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:37:30,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:30,829 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:30,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:30,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:30,890 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:30,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:30,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:30,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:30,967 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 04:37:30,970 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:37:31,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:31,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:37:31,352 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:31,354 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:37:31,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:37:31,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:37:31,395 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:37:31,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:37:31,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:37:31,436 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:37:31,448 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:37:31,461 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 04:37:31,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:37:31,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:32,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:37:32,469 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:32,698 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:32,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:37:32,924 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 04:37:32,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:37:32,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:32,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:37:32,977 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 04:37:32,977 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:37:32,980 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:37:33,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:33,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:37:33,361 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:33,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:37:33,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:37:33,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:37:33,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:37:33,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:37:33,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:37:33,438 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:37:33,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:37:33,467 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 04:37:33,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:37:33,515 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:33,762 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:37:34,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:34,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:34,395 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:37:34,411 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 04:37:34,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:37:34,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:34,469 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:37:34,483 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 04:37:34,483 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:37:34,486 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:37:34,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:34,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:37:34,917 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:34,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:37:34,933 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:37:34,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:37:34,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:37:34,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:37:34,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:37:35,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:37:35,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:37:35,049 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 04:37:35,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:37:35,091 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:35,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:37:35,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:35,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:36,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:37:36,101 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 04:37:36,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:37:36,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:36,135 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:37:36,150 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 04:37:36,150 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:37:36,151 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:37:36,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:36,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:37:36,545 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:36,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:37:36,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:37:36,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:37:36,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:37:36,602 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:37:36,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:37:36,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:37:36,640 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:37:36,653 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 04:37:36,681 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:37:36,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:37,130 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:37:37,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:37,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:38,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:37:38,069 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 04:37:38,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:37:38,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:38,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:37:38,109 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 04:37:38,109 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:37:38,111 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 04:37:38,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:38,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 04:37:38,493 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:38,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 04:37:38,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 04:37:38,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 04:37:38,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 04:37:38,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 04:37:38,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 04:37:38,584 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 04:37:38,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:37:38,615 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 04:37:38,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:37:38,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:38,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:37:38,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:38,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:37:38,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:37:38,736 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 04:37:38,737 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:37:38,752 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:37:38,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:37:38,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:38,798 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:37:38,813 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:38,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:38,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:38,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:38,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:38,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:38,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:38,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:37:38,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:38,948 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:37:38,969 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:38,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:39,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:39,030 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:39,048 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:39,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:39,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:39,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:37:39,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:39,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:37:39,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:39,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:39,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:39,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:39,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:39,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:39,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:39,270 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:37:39,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:39,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:37:39,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:39,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:39,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:39,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:39,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:39,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:39,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:39,433 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:37:39,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:39,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:37:39,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:39,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:39,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:39,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:39,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:39,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:39,591 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:39,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:37:39,625 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 04:37:39,625 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:37:39,627 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:37:39,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:40,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:37:40,630 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:40,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:37:40,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:37:40,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:37:40,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:37:40,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:37:40,719 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:37:40,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:37:40,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:37:41,147 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:37:41,147 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 04:37:43,147 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:37:43,349 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:37:43,349 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 04:37:45,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:37:45,523 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:37:45,523 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.89800000191. 2022-02-08 04:37:45,523 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:37:45,526 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:37:45,635 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:37:45,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:37:45,959 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:37:45,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:37:45,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:37:45,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:37:46,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:37:46,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:37:46,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:37:46,061 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:37:46,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:37:46,236 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:37:46,236 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:37:48,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:37:48,392 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:37:48,394 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 6540 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:37:50,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:37:50,552 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:37:50,552 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:37:52,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:37:52,726 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:37:52,726 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:37:54,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:37:55,121 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:37:55,121 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:37:57,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:37:57,421 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:37:57,421 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:37:59,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:37:59,686 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:37:59,686 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:01,686 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:01,891 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:01,891 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:03,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:04,081 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:04,081 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:06,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:06,328 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:06,329 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:08,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:08,591 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:08,591 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:10,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:10,793 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:10,793 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:12,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:12,987 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:12,989 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:14,989 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:15,256 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:15,257 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:17,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:17,420 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:17,420 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:19,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:19,615 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:19,615 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:21,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:21,825 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:21,826 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:23,826 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:24,220 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:24,220 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:26,221 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:26,411 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:26,411 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:28,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:28,634 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:28,634 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:30,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:30,783 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:30,783 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:32,785 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:32,931 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:32,933 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:34,933 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:35,078 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:35,078 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:37,079 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:37,229 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:37,229 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:39,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:39,380 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:39,381 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:41,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:41,535 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:41,535 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:43,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:43,720 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:43,720 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:38:45,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:45,865 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:38:45,891 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 60.367000103. 2022-02-08 04:38:45,891 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:38:45,914 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:38:46,637 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:38:47,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:38:47,595 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:38:47,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:38:47,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:38:47,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:38:48,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:38:48,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:38:48,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:38:48,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:38:48,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:49,181 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:49,181 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 5 queries submitted, 0 failed, 6545 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:38:51,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:51,621 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:38:51,622 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 04:38:53,624 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:53,770 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:38:53,772 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.88000011444. 2022-02-08 04:38:53,772 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:38:53,773 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:38:54,655 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:38:55,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:38:55,167 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:38:55,168 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:38:55,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:38:55,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:38:55,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:38:55,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:38:55,249 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:38:55,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:38:55,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:38:56,644 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:38:56,645 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.87399983406. 2022-02-08 04:38:56,645 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:38:56,648 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:38:57,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:38:58,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:38:58,174 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:38:58,176 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:38:58,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:38:58,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:38:58,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:38:58,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:38:58,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:38:58,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:38:58,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:39:02,119 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:39:02,121 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.47500014305. 2022-02-08 04:39:02,121 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:39:02,122 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:39:03,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:39:03,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:39:03,408 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:39:03,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:39:03,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:39:03,448 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:39:03,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:39:03,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:39:03,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:39:03,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:39:03,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:39:05,308 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:39:05,309 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.18799996376. 2022-02-08 04:39:05,309 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:39:05,311 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:39:06,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:39:06,540 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:39:06,664 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:39:06,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:39:06,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:39:06,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:39:06,706 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:39:06,724 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:39:06,739 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:39:06,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:39:06,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:39:07,266 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:39:07,267 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.95899987221. 2022-02-08 04:39:07,267 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:39:07,269 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:39:07,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:39:08,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:39:08,191 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:39:08,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:39:08,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:39:08,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:39:08,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:39:08,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:39:08,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:39:08,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:39:08,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:39:09,546 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:39:09,546 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.27900004387. 2022-02-08 04:39:09,548 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:39:09,549 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:39:10,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:39:10,775 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:39:10,917 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:39:10,918 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:39:10,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:39:10,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:39:10,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:39:10,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:39:10,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:39:11,012 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:39:11,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:39:11,538 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:39:11,539 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.99199986458. 2022-02-08 04:39:11,539 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:39:11,542 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:39:12,094 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:39:12,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:39:12,433 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:39:12,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:39:12,448 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:39:12,484 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:39:12,497 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:39:12,515 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:39:12,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:39:12,558 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:39:12,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:39:13,880 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:39:13,880 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.34000015259. ############################################################ # # RAM FootPrint: 4174.26953125 # ############################################################ 2022-02-08 04:39:14,109 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 04:39:14,109 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 76 seconds. Reason: Iteration:3 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 6570 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:40:30,164 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 04:40:30,174 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 04:40:35,187 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:40:35,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:40:35,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:40:35,637 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:40:35,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:40:35,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:40:35,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:40:35,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:40:35,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:40:35,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:40:35,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:40:35,763 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:40:35,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:40:35,822 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['124541 memcached\n'], []) 2022-02-08 04:40:35,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 04:40:35,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:40:35,871 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['124541 memcached\n'], []) 2022-02-08 04:40:35,871 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:40:35,872 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:40:36,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:40:36,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:40:36,957 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:40:36,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:40:36,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:40:37,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:40:37,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:40:37,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:40:37,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:40:37,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:40:37,081 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:40:37,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:40:37,135 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['35709 memcached\n'], []) 2022-02-08 04:40:37,135 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 04:40:37,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:40:37,177 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['35709 memcached\n'], []) 2022-02-08 04:40:37,178 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:40:37,180 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:40:37,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:40:37,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:40:37,551 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:40:37,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:40:37,568 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:40:37,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:40:37,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:40:37,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:40:37,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:40:37,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:40:37,670 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:40:37,670 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:40:37,710 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['23942 memcached\n'], []) 2022-02-08 04:40:37,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 04:40:37,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:40:37,762 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['23942 memcached\n'], []) 2022-02-08 04:40:37,763 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:40:37,765 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:40:37,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:40:38,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:40:38,131 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:40:38,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:40:38,145 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:40:38,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:40:38,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:40:38,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:40:38,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:40:38,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:40:38,230 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:40:38,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:40:38,276 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['58536 memcached\n'], []) 2022-02-08 04:40:38,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 04:40:38,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:40:38,322 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['58536 memcached\n'], []) 2022-02-08 04:40:38,322 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:40:38,325 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:40:38,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:40:38,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:40:38,690 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:40:38,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:40:38,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:40:38,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:40:38,734 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:40:38,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:40:38,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:40:38,790 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:40:38,805 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:40:38,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:40:38,855 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['52386 memcached\n'], []) 2022-02-08 04:40:38,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 04:40:38,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:40:38,903 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['52386 memcached\n'], []) 2022-02-08 04:40:38,903 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:40:38,907 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:40:39,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:40:39,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:40:39,269 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:40:39,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:40:39,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:40:39,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:40:39,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:40:39,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:40:39,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:40:39,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:40:39,364 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:40:39,364 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:40:39,444 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['87306 memcached\n'], []) 2022-02-08 04:40:39,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 04:40:39,469 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:40:39,489 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['87306 memcached\n'], []) 2022-02-08 04:40:39,490 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:40:39,490 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:40:39,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:40:39,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:40:39,862 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:40:39,862 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:40:39,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:40:39,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:40:39,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:40:39,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:40:39,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:40:39,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:40:40,000 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:40:40,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:40:40,065 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['21090 memcached\n'], []) 2022-02-08 04:40:40,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 04:40:40,089 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:40:40,111 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['21090 memcached\n'], []) 2022-02-08 04:40:40,111 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:40:40,118 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:40:40,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:40:40,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:40:40,479 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:40:40,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:40:40,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:40:40,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:40:40,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:40:40,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:40:40,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:40:40,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:40:40,578 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:40:40,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:40:40,644 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['36021 memcached\n'], []) 2022-02-08 04:40:40,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 04:40:40,678 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:40:40,707 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['36021 memcached\n'], []) 2022-02-08 04:40:40,709 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:40:40,710 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:40:40,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:40:41,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:40:41,068 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:40:41,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:40:41,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:40:41,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:40:41,109 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:40:41,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:40:41,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:40:41,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:40:41,164 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:40:41,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:40:41,223 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['72042 memcached\n'], []) 2022-02-08 04:40:41,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 04:40:41,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:40:41,283 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['72042 memcached\n'], []) 2022-02-08 04:40:41,283 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:40:41,286 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:40:41,362 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:40:41,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:40:41,687 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:40:41,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:40:41,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:40:41,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:40:41,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:40:41,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:40:41,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:40:41,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:40:41,786 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:40:41,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:40:41,832 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['101725 memcached\n'], []) 2022-02-08 04:40:41,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 04:40:41,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:40:41,878 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['101725 memcached\n'], []) 2022-02-08 04:40:41,880 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:40:41,881 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:40:42,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:40:42,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:40:42,339 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:40:42,341 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:40:42,354 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:40:42,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:40:42,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:40:42,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:40:42,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:40:42,420 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:40:42,433 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:40:42,447 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 04:40:42,477 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:40:42,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:40:45,091 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:40:45,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:40:45,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:40:45,918 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:40:45,933 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 04:40:45,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:40:45,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:40:45,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:40:46,003 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 04:40:46,003 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:40:46,007 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:40:46,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:40:46,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:40:46,404 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:40:46,405 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:40:46,420 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:40:46,433 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:40:46,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:40:46,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:40:46,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:40:46,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:40:46,505 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:40:46,519 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 04:40:46,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:40:46,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:40:46,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:40:46,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:40:46,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:40:46,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:40:46,877 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 04:40:46,877 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:40:46,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:40:46,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:40:46,930 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 04:40:46,930 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:40:46,931 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:40:47,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:40:47,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:40:47,293 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:40:47,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:40:47,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:40:47,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:40:47,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:40:47,346 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:40:47,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:40:47,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:40:47,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:40:47,403 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 04:40:47,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:40:47,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 6597 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:40:50,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:40:50,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:40:50,637 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:40:50,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:40:50,872 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 04:40:50,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:40:50,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:40:50,905 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:40:50,917 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 04:40:50,918 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:40:50,921 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 04:40:50,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:40:51,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 04:40:51,283 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:40:51,283 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 04:40:51,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 04:40:51,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 04:40:51,323 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 04:40:51,341 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 04:40:51,354 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 04:40:51,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 04:40:51,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:40:51,398 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 04:40:51,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:40:51,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:40:51,454 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:40:51,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:40:51,484 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:40:51,499 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:40:51,513 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 04:40:51,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:40:51,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:40:51,545 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:40:51,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:51,576 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:40:51,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:51,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:51,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:51,635 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:51,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:51,665 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:51,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:51,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:51,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:51,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:40:51,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:51,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:51,783 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:51,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:51,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:51,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:51,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:51,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:51,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:51,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:40:51,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:51,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:51,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:51,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:51,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:52,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:52,028 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:52,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:52,061 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:52,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:40:52,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:52,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:52,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:52,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:52,160 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:52,176 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:52,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:52,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:52,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:52,252 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:40:52,270 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:52,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:52,323 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:52,341 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:52,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:52,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:52,395 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:52,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:52,431 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 04:40:52,434 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:40:52,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:40:52,793 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:40:52,917 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:40:52,918 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:40:52,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:40:52,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:40:52,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:40:53,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:40:53,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:40:53,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:40:53,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:40:53,086 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 04:40:53,124 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:40:53,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:40:55,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:40:55,505 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:40:55,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:40:55,798 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:40:55,812 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 04:40:55,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:40:55,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:40:55,882 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:40:55,898 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 04:40:55,898 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:40:55,901 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 04:40:56,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:40:56,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 04:40:56,295 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:40:56,295 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 04:40:56,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 04:40:56,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 04:40:56,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 04:40:56,357 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 04:40:56,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 04:40:56,385 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 04:40:56,403 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:40:56,420 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 04:40:56,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:40:56,464 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:40:56,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:40:56,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:40:56,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:40:56,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:40:56,539 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 04:40:56,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:40:56,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:40:56,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:40:56,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:56,599 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:40:56,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:56,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:56,645 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:56,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:56,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:56,693 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:56,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:56,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:40:56,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:56,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:40:56,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:56,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:56,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:56,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:56,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:56,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:56,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:56,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:40:56,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:56,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:40:56,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:56,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:57,020 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:57,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:57,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:57,079 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:57,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:57,114 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:40:57,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:57,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:40:57,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:57,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:57,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:57,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:57,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:57,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:57,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:57,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:40:57,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:57,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:40:57,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:57,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:57,377 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:57,395 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:57,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:57,431 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:57,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:57,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:40:57,486 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 04:40:57,487 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:40:57,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:40:57,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:40:57,815 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:40:57,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:40:57,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:40:57,839 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:40:57,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:40:57,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:40:57,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:40:57,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:40:57,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:40:57,924 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 04:40:57,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:40:57,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:40:59,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:40:59,948 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:00,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:00,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:41:00,480 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 04:41:00,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:41:00,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:41:00,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:41:00,525 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 04:41:00,525 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:41:00,526 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:41:00,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:41:00,829 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:41:00,895 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:41:00,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:41:00,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:41:00,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:41:00,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:41:00,964 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:41:00,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:41:00,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:41:01,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:41:01,023 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 04:41:01,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:41:01,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:41:04,163 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:41:04,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:04,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:04,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:41:04,996 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 04:41:04,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:41:05,012 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:41:05,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:41:05,049 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 04:41:05,051 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:41:05,058 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 04:41:05,137 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:41:05,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 04:41:05,411 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:41:05,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 04:41:05,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 04:41:05,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 04:41:05,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 04:41:05,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 04:41:05,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 04:41:05,497 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 04:41:05,512 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:41:05,526 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 04:41:05,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:41:05,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:41:05,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:41:05,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:05,604 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:05,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:41:05,628 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 04:41:05,630 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:41:05,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:41:05,655 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:41:05,670 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:05,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:41:05,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:05,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:05,727 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:05,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:05,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:05,767 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:05,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:05,793 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:05,806 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:05,821 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:41:05,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:05,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:05,867 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:05,882 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:05,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:05,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:05,933 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:05,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:05,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:05,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:41:05,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:06,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:06,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:06,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:06,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:06,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:06,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:06,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:06,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:06,130 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:41:06,145 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:06,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:06,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:06,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:06,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:06,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:06,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:06,265 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:06,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:06,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:41:06,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:06,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:06,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:06,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:06,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:06,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:06,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:06,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:06,447 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 04:41:06,450 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 04:41:06,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:41:06,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 04:41:06,864 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:41:06,864 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 04:41:06,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 04:41:06,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 04:41:06,905 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 04:41:06,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 04:41:06,936 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 04:41:06,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 04:41:06,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:41:06,983 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 04:41:07,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:41:07,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:41:07,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:41:07,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:07,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:07,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:41:07,101 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 04:41:07,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:41:07,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:41:07,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:41:07,147 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:07,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:41:07,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:07,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:07,214 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:07,232 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:07,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:07,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:07,285 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:07,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:07,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:07,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:41:07,354 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:07,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:07,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:07,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:07,420 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:07,436 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:07,454 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:07,474 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:07,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:07,512 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:41:07,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:07,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:07,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:07,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:07,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:07,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:07,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:07,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:07,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:07,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:41:07,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:07,719 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:07,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:07,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:07,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:07,790 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:07,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:07,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:07,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:07,867 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:41:07,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:07,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:07,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:07,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:07,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:08,002 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:08,017 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:08,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:08,052 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 04:41:08,055 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:41:08,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:41:08,392 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:41:08,457 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:41:08,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:41:08,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:41:08,487 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:41:08,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:41:08,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:41:08,533 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:41:08,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:41:08,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:41:08,588 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 04:41:08,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:41:08,630 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:41:12,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:41:12,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:12,757 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:12,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:41:13,006 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 04:41:13,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:41:13,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:41:13,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:41:13,095 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 04:41:13,095 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:41:13,098 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:41:13,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:41:13,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:41:13,536 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:41:13,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:41:13,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:41:13,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:41:13,581 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:41:13,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:41:13,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:41:13,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:41:13,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:41:13,671 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 04:41:13,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:41:13,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:41:16,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:41:16,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:16,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:17,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:41:17,039 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 04:41:17,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:41:17,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:41:17,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:41:17,118 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 04:41:17,118 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:41:17,119 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:41:17,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:41:18,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:41:18,157 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:41:18,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:41:18,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:41:18,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:41:18,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:41:18,247 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:41:18,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:41:18,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:41:18,318 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:41:18,334 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 04:41:18,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:41:18,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:41:20,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:41:20,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:21,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:21,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:41:21,321 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 04:41:21,321 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:41:21,336 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:41:21,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:41:21,387 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 04:41:21,388 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:41:21,388 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:41:21,477 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:41:21,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:41:21,887 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:41:21,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:41:21,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:41:21,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:41:21,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:41:21,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:41:22,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:41:22,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:41:22,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:41:22,066 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 04:41:22,091 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:41:22,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:41:22,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:41:23,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:23,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:23,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:41:23,944 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 04:41:23,946 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:41:23,961 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:41:23,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:41:24,003 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 04:41:24,003 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:41:24,005 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 04:41:24,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:41:24,318 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 04:41:24,388 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:41:24,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 04:41:24,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 04:41:24,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 04:41:24,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 04:41:24,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 04:41:24,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 04:41:24,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 04:41:24,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:41:24,532 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 04:41:24,561 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:41:24,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:41:24,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:41:24,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:24,630 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:41:24,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:41:24,661 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 04:41:24,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:41:24,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:41:24,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:41:24,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:24,739 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:41:24,757 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:24,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:24,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:24,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:24,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:24,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:24,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:24,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:41:24,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:24,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:41:24,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:24,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:24,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:25,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:25,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:25,048 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:25,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:25,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:41:25,102 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:25,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:41:25,140 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:25,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:25,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:25,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:25,209 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:25,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:25,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:25,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:41:25,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:25,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:41:25,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:25,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:25,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:25,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:25,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:25,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:25,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:25,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:41:25,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:25,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:41:25,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:25,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:25,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:25,545 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:25,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:25,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:25,604 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:25,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:41:25,641 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 04:41:25,641 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:41:25,644 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:41:25,724 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:41:25,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:41:26,029 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:41:26,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:41:26,048 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:41:26,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:41:26,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:41:26,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:41:26,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:41:26,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:41:26,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:41:26,395 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:41:26,397 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.753999948502. 2022-02-08 04:41:26,397 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:41:26,398 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:41:26,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:41:26,790 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:41:26,901 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:41:26,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:41:26,923 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:41:26,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:41:26,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:41:26,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:41:26,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:41:27,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:41:27,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:41:27,247 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:41:27,247 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:41:29,249 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:41:29,474 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:41:29,474 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:41:31,474 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:41:31,920 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:41:31,921 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:41:33,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:41:34,226 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:41:34,226 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:41:36,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:41:36,427 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:41:36,427 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:41:38,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:41:38,726 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:41:38,727 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:41:40,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:41:40,984 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:41:40,984 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:41:42,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:41:43,328 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:41:43,329 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:41:45,329 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:41:45,546 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:41:45,548 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:41:47,548 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:41:47,980 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:41:47,980 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 6602 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:41:49,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:41:50,178 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:41:50,180 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:41:52,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:41:52,490 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:41:52,490 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:41:54,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:41:54,686 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:41:54,687 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:41:56,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:41:56,993 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:41:56,993 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:41:58,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:41:59,209 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:41:59,209 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:42:01,209 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:42:01,405 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:42:01,407 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:42:03,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:42:03,592 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:42:03,592 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:42:05,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:42:05,756 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:42:05,757 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:42:07,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:42:07,960 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:42:07,961 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:42:09,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:42:10,112 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:42:10,112 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:42:12,114 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:42:12,249 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:42:12,250 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 45.8529999256. 2022-02-08 04:42:12,250 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:42:12,252 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:42:13,066 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:42:13,512 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:42:13,654 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:42:13,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:42:13,670 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:42:13,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:42:13,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:42:13,721 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:42:13,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:42:13,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:42:13,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:42:14,240 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:42:14,240 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.99000000954. 2022-02-08 04:42:14,240 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:42:14,243 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:42:14,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:42:15,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:42:15,230 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:42:15,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:42:15,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:42:15,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:42:15,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:42:15,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:42:15,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:42:15,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:42:15,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:42:15,888 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:42:15,888 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.64700007439. 2022-02-08 04:42:15,888 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:42:15,890 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:42:16,512 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:42:16,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:42:16,936 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:42:16,936 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:42:16,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:42:16,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:42:16,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:42:17,015 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:42:17,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:42:17,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:42:17,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:42:17,538 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:42:17,538 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.65000009537. 2022-02-08 04:42:17,538 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:42:17,539 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:42:18,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:42:18,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:42:18,608 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:42:18,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:42:18,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:42:18,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:42:18,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:42:18,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:42:18,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:42:18,737 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:42:18,752 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:42:19,155 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:42:19,157 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.61899995804. 2022-02-08 04:42:19,157 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:42:19,158 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:42:19,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:42:20,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:42:20,141 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:42:20,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:42:20,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:42:20,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:42:20,186 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:42:20,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:42:20,214 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:42:20,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:42:20,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:42:20,624 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:42:20,625 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.46700000763. 2022-02-08 04:42:20,625 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:42:20,625 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:42:21,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:42:21,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:42:21,565 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:42:21,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:42:21,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:42:21,604 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:42:21,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:42:21,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:42:21,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:42:21,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:42:21,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:42:22,424 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:42:22,424 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.79999995232. 2022-02-08 04:42:22,424 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:42:22,427 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:42:23,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:42:23,278 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:42:23,367 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:42:23,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:42:23,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:42:23,395 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:42:23,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:42:23,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:42:23,441 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:42:23,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:42:23,487 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:42:23,851 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:42:23,851 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.42600011826. 2022-02-08 04:42:23,851 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:42:23,852 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:42:24,512 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:42:24,767 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:42:24,867 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:42:24,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:42:24,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:42:24,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:42:24,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:42:24,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:42:24,964 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:42:24,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:42:25,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:42:25,926 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:42:25,927 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.0759999752. ############################################################ # # RAM FootPrint: 4195.578125 # ############################################################ 2022-02-08 04:42:26,200 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 04:42:26,201 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 115 seconds. Reason: Iteration:4 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 6615 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 6641 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:44:21,269 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 04:44:21,279 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 04:44:26,290 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:44:26,441 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:26,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:44:26,917 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:26,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:44:26,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:44:26,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:44:26,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:44:26,989 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:44:27,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:44:27,038 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:44:27,061 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:44:27,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:44:27,130 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['125264 memcached\n'], []) 2022-02-08 04:44:27,130 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 04:44:27,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:44:27,190 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['125264 memcached\n'], []) 2022-02-08 04:44:27,190 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:44:27,193 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:44:27,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:28,028 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:44:28,125 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:28,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:44:28,140 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:44:28,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:44:28,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:44:28,216 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:44:28,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:44:28,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:44:28,266 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:44:28,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:44:28,321 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['36689 memcached\n'], []) 2022-02-08 04:44:28,321 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 04:44:28,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:44:28,367 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['36689 memcached\n'], []) 2022-02-08 04:44:28,367 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:44:28,368 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:44:28,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:28,719 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:44:28,782 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:28,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:44:28,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:44:28,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:44:28,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:44:28,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:44:28,859 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:44:28,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:44:28,888 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:44:28,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:44:28,950 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['24725 memcached\n'], []) 2022-02-08 04:44:28,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 04:44:28,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:44:29,002 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['24725 memcached\n'], []) 2022-02-08 04:44:29,003 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:44:29,006 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:44:29,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:29,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:44:29,358 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:29,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:44:29,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:44:29,385 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:44:29,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:44:29,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:44:29,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:44:29,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:44:29,453 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:44:29,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:44:29,513 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['59235 memcached\n'], []) 2022-02-08 04:44:29,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 04:44:29,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:44:29,565 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['59235 memcached\n'], []) 2022-02-08 04:44:29,565 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:44:29,568 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:44:29,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:29,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:44:29,944 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:29,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:44:29,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:44:29,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:44:29,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:44:30,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:44:30,038 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:44:30,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:44:30,065 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:44:30,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:44:30,134 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['53043 memcached\n'], []) 2022-02-08 04:44:30,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 04:44:30,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:44:30,177 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['53043 memcached\n'], []) 2022-02-08 04:44:30,177 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:44:30,177 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:44:30,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:30,477 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:44:30,535 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:30,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:44:30,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:44:30,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:44:30,576 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:44:30,594 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:44:30,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:44:30,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:44:30,641 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:44:30,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:44:30,694 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['87962 memcached\n'], []) 2022-02-08 04:44:30,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 04:44:30,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:44:30,742 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['87962 memcached\n'], []) 2022-02-08 04:44:30,742 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:44:30,743 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:44:30,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:31,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:44:31,118 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:31,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:44:31,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:44:31,147 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:44:31,160 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:44:31,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:44:31,191 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:44:31,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:44:31,217 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:44:31,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:44:31,266 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['21796 memcached\n'], []) 2022-02-08 04:44:31,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 04:44:31,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:44:31,315 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['21796 memcached\n'], []) 2022-02-08 04:44:31,315 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:44:31,321 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:44:31,392 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:31,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:44:31,667 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:31,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:44:31,681 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:44:31,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:44:31,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:44:31,724 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:44:31,737 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:44:31,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:44:31,766 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:44:31,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:44:31,828 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['36724 memcached\n'], []) 2022-02-08 04:44:31,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 04:44:31,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:44:31,874 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['36724 memcached\n'], []) 2022-02-08 04:44:31,874 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:44:31,875 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:44:31,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:32,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:44:32,253 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:32,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:44:32,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:44:32,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:44:32,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:44:32,321 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:44:32,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:44:32,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:44:32,358 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:44:32,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:44:32,403 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['72745 memcached\n'], []) 2022-02-08 04:44:32,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 04:44:32,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:44:32,446 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['72745 memcached\n'], []) 2022-02-08 04:44:32,446 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:44:32,447 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:44:32,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:32,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:44:32,815 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:32,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:44:32,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:44:32,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:44:32,862 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:44:32,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:44:32,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:44:32,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:44:32,954 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:44:32,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:44:33,012 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['102400 memcached\n'], []) 2022-02-08 04:44:33,012 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 04:44:33,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:44:33,059 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['102400 memcached\n'], []) 2022-02-08 04:44:33,061 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:44:33,063 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:44:33,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:33,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:44:33,446 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:33,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:44:33,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:44:33,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:44:33,497 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:44:33,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:44:33,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:44:33,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:44:33,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:44:33,573 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 04:44:33,599 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:44:33,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:34,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:44:34,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:34,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:34,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:44:35,002 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 04:44:35,002 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:44:35,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:35,038 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:44:35,052 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 04:44:35,053 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:44:35,055 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:44:35,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:35,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:44:35,440 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:35,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:44:35,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:44:35,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:44:35,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:44:35,505 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:44:35,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:44:35,535 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:44:35,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:44:35,569 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 04:44:35,604 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:44:35,624 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:35,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:44:35,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:35,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:35,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:44:35,792 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 04:44:35,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:44:35,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:35,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:44:35,841 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 04:44:35,841 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:44:35,844 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:44:35,931 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:36,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:44:36,219 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:36,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:44:36,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:44:36,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:44:36,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:44:36,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:44:36,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:44:36,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:44:36,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:44:36,336 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 04:44:36,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:44:36,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:36,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:44:37,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:37,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:37,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:44:37,681 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 04:44:37,681 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:44:37,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:37,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:44:37,756 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 04:44:37,756 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:44:37,759 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 04:44:37,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:38,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 04:44:38,240 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:38,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 04:44:38,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 04:44:38,270 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 04:44:38,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 04:44:38,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 04:44:38,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 04:44:38,334 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 04:44:38,349 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:44:38,367 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 04:44:38,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:44:38,408 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:38,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:44:38,438 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:38,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:38,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:44:38,483 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 04:44:38,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:44:38,497 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:38,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:44:38,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:38,543 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:44:38,561 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:38,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:38,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:38,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:38,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:38,635 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:38,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:38,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:38,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:38,701 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:44:38,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:38,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:38,757 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:38,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:38,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:38,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:38,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:38,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:38,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:38,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:44:38,905 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:38,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:38,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:38,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:38,974 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:38,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:39,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:39,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:39,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:39,076 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:44:39,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:39,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:39,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:39,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:39,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:39,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:39,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:39,201 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:39,216 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:39,232 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:44:39,247 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:39,265 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:39,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:39,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:39,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:39,326 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:39,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:39,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:39,374 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 04:44:39,375 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:44:39,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:39,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:44:39,720 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:39,721 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:44:39,737 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:44:39,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:44:39,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:44:39,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:44:39,803 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:44:39,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:44:39,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:44:39,851 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 04:44:39,877 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:44:39,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:40,160 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:44:40,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:40,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:40,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:44:40,868 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 04:44:40,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:44:40,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:41,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:44:41,173 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 04:44:41,173 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:44:41,176 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 04:44:41,260 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:41,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 04:44:41,581 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:41,581 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 04:44:41,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 04:44:41,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 04:44:41,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 04:44:41,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 04:44:41,650 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 04:44:41,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 04:44:41,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:44:41,700 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 04:44:41,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:44:41,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:41,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:44:41,783 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:41,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:41,813 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:44:41,828 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 04:44:41,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:44:41,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:41,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:44:41,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:41,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:44:41,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:41,926 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:41,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:41,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:41,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:41,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:42,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:42,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:42,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:42,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:44:42,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:42,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:42,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:42,137 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:42,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:42,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:42,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:42,209 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:42,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:42,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:44:42,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:42,275 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:42,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:42,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:42,323 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:42,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:42,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:42,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:42,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:42,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:44:42,436 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:42,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:42,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:42,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:42,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:42,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:42,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:42,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:42,568 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:42,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:44:42,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:42,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:42,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:42,658 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:42,676 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:42,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:42,711 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:42,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:42,750 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 04:44:42,753 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:44:42,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:43,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:44:43,108 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:43,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:44:43,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:44:43,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:44:43,145 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:44:43,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:44:43,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:44:43,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:44:43,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:44:43,219 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 04:44:43,249 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:44:43,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:44,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:44:45,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:45,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:45,602 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:44:45,621 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 04:44:45,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:44:45,635 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:45,650 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:44:45,663 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 04:44:45,664 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:44:45,665 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:44:45,737 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:45,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:44:46,039 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:46,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:44:46,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:44:46,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:44:46,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:44:46,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:44:46,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:44:46,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:44:46,158 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:44:46,176 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 04:44:46,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:44:46,214 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:47,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:44:47,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:48,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:48,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:44:48,250 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 04:44:48,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:44:48,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:48,334 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:44:48,349 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 04:44:48,351 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:44:48,352 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 04:44:48,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:48,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 04:44:48,703 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:48,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 04:44:48,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 04:44:48,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 04:44:48,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 04:44:48,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 04:44:48,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 04:44:48,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 04:44:48,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:44:48,813 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 04:44:48,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:44:48,857 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:48,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:44:48,885 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:48,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:48,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:44:48,924 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 04:44:48,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:44:48,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:48,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:44:48,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:48,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:44:48,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:49,012 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:49,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:49,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:49,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:49,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:49,086 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:49,102 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:49,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:49,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:44:49,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:49,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:49,196 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:49,211 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:49,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:49,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:49,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:49,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:49,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:49,303 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:44:49,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 5 queries submitted, 0 failed, 6666 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:44:49,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:49,346 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:49,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:49,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:49,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:49,405 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:49,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:49,436 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:49,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:44:49,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:49,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:49,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:49,512 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:49,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:49,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:49,558 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:49,576 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:49,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:49,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:44:49,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:49,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:49,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:49,678 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:49,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:49,711 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:49,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:49,744 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:49,760 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 04:44:49,763 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 04:44:49,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:50,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 04:44:50,183 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:50,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 04:44:50,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 04:44:50,211 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 04:44:50,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 04:44:50,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 04:44:50,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 04:44:50,270 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 04:44:50,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:44:50,302 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 04:44:50,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:44:50,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:50,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:44:50,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:50,385 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:50,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:44:50,414 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 04:44:50,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:44:50,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:50,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:44:50,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:50,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:44:50,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:50,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:50,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:50,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:50,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:50,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:50,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:50,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:44:50,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:50,665 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:44:50,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:50,701 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:50,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:50,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:50,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:50,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:50,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:50,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:44:50,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:50,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:44:50,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:50,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:50,882 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:50,900 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:50,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:50,931 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:50,948 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:50,964 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:44:50,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:50,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:44:51,015 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:51,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:51,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:51,066 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:51,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:51,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:51,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:51,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:44:51,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:51,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:44:51,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:51,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:51,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:51,242 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:51,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:51,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:51,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:51,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:44:51,341 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 04:44:51,344 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:44:51,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:51,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:44:51,799 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:51,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:44:51,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:44:51,829 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:44:51,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:44:51,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:44:51,880 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:44:51,910 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:44:51,946 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:44:51,970 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 04:44:51,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:44:52,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:52,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:44:52,826 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:53,086 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:53,313 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:44:53,328 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 04:44:53,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:44:53,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:53,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:44:53,374 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 04:44:53,374 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:44:53,375 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:44:53,484 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:53,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:44:53,776 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:53,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:44:53,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:44:53,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:44:53,816 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:44:53,834 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:44:53,846 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:44:53,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:44:53,877 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:44:53,897 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 04:44:53,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:44:53,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:54,255 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:44:54,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:54,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:54,969 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:44:54,983 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 04:44:54,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:44:54,999 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:55,038 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:44:55,053 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 04:44:55,053 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:44:55,056 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:44:55,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:55,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:44:56,098 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:56,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:44:56,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:44:56,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:44:56,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:44:56,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:44:56,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:44:56,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:44:56,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:44:56,260 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 04:44:56,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:44:56,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:56,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:44:56,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:57,099 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:44:57,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:44:57,279 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 04:44:57,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:44:57,295 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:44:57,334 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:44:57,348 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 04:44:57,349 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:44:57,351 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:44:57,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:44:57,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:44:57,802 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:44:57,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:44:57,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:44:57,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:44:57,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:44:57,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:44:57,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:44:57,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:44:57,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:44:57,967 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 04:44:57,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:44:58,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:45:00,701 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:45:01,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:45:01,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:45:01,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:45:01,811 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 04:45:01,811 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:45:01,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:45:01,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:45:01,869 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 04:45:01,869 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:45:01,871 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 04:45:01,948 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:45:02,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 04:45:02,247 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:45:02,247 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 04:45:02,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 04:45:02,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 04:45:02,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 04:45:02,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 04:45:02,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 04:45:02,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 04:45:02,357 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:45:02,372 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 04:45:02,403 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:45:02,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:45:02,436 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:45:02,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:45:02,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:45:02,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:45:02,493 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 04:45:02,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:45:02,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:45:02,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:45:02,540 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:45:02,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:45:02,576 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:45:02,594 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:45:02,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:45:02,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:45:02,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:45:02,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:45:02,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:45:02,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:45:02,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:45:02,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:45:02,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:45:02,767 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:45:02,785 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:45:02,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:45:02,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:45:02,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:45:02,854 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:45:02,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:45:02,892 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:45:02,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:45:02,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:45:02,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:45:02,964 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:45:02,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:45:03,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:45:03,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:45:03,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:45:03,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:45:03,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:45:03,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:45:03,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:45:03,137 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:45:03,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:45:03,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:45:03,188 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:45:03,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:45:03,221 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:45:03,244 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:45:03,260 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:45:03,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:45:03,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:45:03,318 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:45:03,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:45:03,352 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:45:03,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:45:03,390 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:45:03,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:45:03,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:45:03,446 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 04:45:03,447 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:45:03,447 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:45:03,566 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:45:03,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:45:03,976 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:45:03,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:45:04,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:45:04,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:45:04,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:45:04,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:45:04,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:45:04,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:45:04,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:04,450 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:45:04,451 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.00499987602. 2022-02-08 04:45:04,451 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:45:04,453 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:45:04,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:45:04,803 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:45:04,880 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:45:04,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:45:04,900 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:45:04,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:45:04,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:45:04,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:45:04,971 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:45:04,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:45:05,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:05,315 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:05,315 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:07,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:07,480 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:07,482 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:09,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:09,644 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:09,644 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:11,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:11,805 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:11,805 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:13,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:13,957 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:13,957 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:15,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:16,171 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:16,173 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:18,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:18,477 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:18,479 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:20,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:20,674 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:20,674 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:22,676 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:22,888 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:22,888 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:24,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:25,095 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:25,095 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:27,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:27,453 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:27,453 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:29,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:29,911 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:29,911 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:31,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:32,276 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:32,276 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:34,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:34,571 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:34,571 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:36,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:36,864 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:36,865 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:38,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:39,165 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:39,165 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:41,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:41,433 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:41,434 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:43,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:43,704 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:43,704 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:45,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:46,092 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:46,092 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:48,094 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:48,328 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:48,328 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 6674 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:45:50,329 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:50,542 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:50,542 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:45:52,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:52,721 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:45:52,721 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 48.2699999809. 2022-02-08 04:45:52,721 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:45:52,723 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:45:53,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:45:53,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:45:54,387 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:45:54,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:45:54,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:45:54,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:45:54,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:45:54,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:45:54,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:45:54,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:45:54,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:55,651 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:45:55,651 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 04:45:57,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:45:57,812 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:45:57,812 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.09100008011. 2022-02-08 04:45:57,812 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:45:57,815 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:45:58,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:45:59,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:45:59,417 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:45:59,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:45:59,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:45:59,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:45:59,469 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:45:59,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:45:59,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:45:59,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:45:59,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:46:00,282 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:46:00,282 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.47000002861. 2022-02-08 04:46:00,282 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:46:00,285 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:46:01,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:46:01,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:46:02,062 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:46:02,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:46:02,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:46:02,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:46:02,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:46:02,135 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:46:02,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:46:02,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:46:02,191 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:46:02,796 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:46:02,798 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.5150001049. 2022-02-08 04:46:02,798 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:46:02,799 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:46:03,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:46:04,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:46:04,450 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:46:04,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:46:04,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:46:04,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:46:04,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:46:04,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:46:04,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:46:04,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:46:04,581 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:46:06,243 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:46:06,243 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.44599986076. 2022-02-08 04:46:06,243 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:46:06,246 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:46:06,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:46:07,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:46:07,207 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:46:07,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:46:07,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:46:07,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:46:07,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:46:07,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:46:07,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:46:07,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:46:07,336 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:46:07,690 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:46:07,690 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.4470000267. 2022-02-08 04:46:07,690 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:46:07,693 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:46:08,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:46:09,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:46:09,200 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:46:09,201 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:46:09,216 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:46:09,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:46:09,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:46:09,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:46:09,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:46:09,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:46:09,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:46:10,302 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:46:10,302 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.61199998856. 2022-02-08 04:46:10,302 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:46:10,303 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:46:10,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:46:11,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:46:11,239 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:46:11,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:46:11,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:46:11,275 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:46:11,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:46:11,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:46:11,313 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:46:11,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:46:11,352 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:46:11,681 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:46:11,683 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.38000011444. 2022-02-08 04:46:11,683 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:46:11,686 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:46:12,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:46:12,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:46:13,012 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:46:13,012 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:46:13,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:46:13,051 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:46:13,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:46:13,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:46:13,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:46:13,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:46:13,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:46:13,632 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:46:13,632 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.94999980927. ############################################################ # # RAM FootPrint: 4246.12890625 # ############################################################ 2022-02-08 04:46:13,903 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 04:46:13,903 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 74 seconds. Reason: Iteration:5 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 6696 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:47:27,911 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 04:47:27,911 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 04:47:32,917 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:47:33,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:33,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:47:33,407 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:33,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:47:33,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:47:33,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:47:33,448 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:47:33,464 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:47:33,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:47:33,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:47:33,503 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:47:33,505 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:47:33,569 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['125986 memcached\n'], []) 2022-02-08 04:47:33,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 04:47:33,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:47:33,618 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['125986 memcached\n'], []) 2022-02-08 04:47:33,618 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:47:33,619 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:47:34,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:34,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:47:34,450 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:34,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:47:34,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:47:34,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:47:34,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:47:34,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:47:34,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:47:34,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:47:34,578 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:47:34,579 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:47:34,634 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['37645 memcached\n'], []) 2022-02-08 04:47:34,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 04:47:34,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:47:34,684 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['37645 memcached\n'], []) 2022-02-08 04:47:34,684 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:47:34,687 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:47:34,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:34,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:47:35,059 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:35,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:47:35,076 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:47:35,094 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:47:35,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:47:35,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:47:35,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:47:35,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:47:35,171 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:47:35,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:47:35,214 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['25537 memcached\n'], []) 2022-02-08 04:47:35,214 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 04:47:35,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:47:35,267 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['25537 memcached\n'], []) 2022-02-08 04:47:35,267 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:47:35,270 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:47:35,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:35,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:47:35,641 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:35,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:47:35,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:47:35,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:47:35,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:47:35,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:47:35,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:47:35,727 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:47:35,742 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:47:35,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:47:35,855 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['59952 memcached\n'], []) 2022-02-08 04:47:35,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 04:47:35,882 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:47:35,904 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['59952 memcached\n'], []) 2022-02-08 04:47:35,904 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:47:35,905 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:47:36,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:36,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:47:36,285 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:36,285 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:47:36,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:47:36,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:47:36,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:47:36,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:47:36,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:47:36,362 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:47:36,377 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:47:36,377 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:47:36,427 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['53763 memcached\n'], []) 2022-02-08 04:47:36,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 04:47:36,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:47:36,471 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['53763 memcached\n'], []) 2022-02-08 04:47:36,471 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:47:36,473 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:47:36,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:36,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:47:36,828 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:36,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:47:36,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:47:36,854 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:47:36,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:47:36,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:47:36,892 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:47:36,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:47:36,917 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:47:36,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:47:36,980 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['88680 memcached\n'], []) 2022-02-08 04:47:36,982 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 04:47:37,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:47:37,026 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['88680 memcached\n'], []) 2022-02-08 04:47:37,028 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:47:37,029 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:47:37,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:37,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:47:37,384 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:37,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:47:37,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:47:37,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:47:37,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:47:37,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:47:37,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:47:37,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:47:37,480 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:47:37,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:47:37,529 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['22517 memcached\n'], []) 2022-02-08 04:47:37,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 04:47:37,553 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:47:37,575 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['22517 memcached\n'], []) 2022-02-08 04:47:37,575 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:47:37,578 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:47:37,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:37,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:47:37,941 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:37,943 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:47:37,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:47:37,971 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:47:37,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:47:38,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:47:38,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:47:38,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:47:38,049 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:47:38,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:47:38,115 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['37442 memcached\n'], []) 2022-02-08 04:47:38,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 04:47:38,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:47:38,167 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['37442 memcached\n'], []) 2022-02-08 04:47:38,167 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:47:38,170 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:47:38,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:38,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:47:38,538 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:38,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:47:38,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:47:38,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:47:38,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:47:38,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:47:38,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:47:38,624 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:47:38,638 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:47:38,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:47:38,686 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['73462 memcached\n'], []) 2022-02-08 04:47:38,686 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 04:47:38,721 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:47:38,743 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['73462 memcached\n'], []) 2022-02-08 04:47:38,743 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:47:38,746 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:47:38,829 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:39,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:47:39,117 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:39,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:47:39,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:47:39,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:47:39,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:47:39,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:47:39,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:47:39,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:47:39,216 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:47:39,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:47:39,263 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['103116 memcached\n'], []) 2022-02-08 04:47:39,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 04:47:39,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:47:39,309 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['103116 memcached\n'], []) 2022-02-08 04:47:39,309 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:47:39,312 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:47:39,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:39,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:47:39,680 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:39,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:47:39,696 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:47:39,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:47:39,724 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:47:39,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:47:39,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:47:39,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:47:39,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:47:39,798 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 04:47:39,826 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:47:39,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:47:42,920 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:47:43,303 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:47:43,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:47:43,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:47:43,786 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 04:47:43,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:47:43,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:47:43,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:47:43,869 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 04:47:43,871 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:47:43,872 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:47:43,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:44,201 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:47:44,279 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:44,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:47:44,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:47:44,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:47:44,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:47:44,341 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:47:44,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:47:44,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:47:44,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:47:44,401 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 04:47:44,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:47:44,446 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:47:44,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:47:44,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:47:44,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:47:44,619 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:47:44,634 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 04:47:44,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:47:44,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:47:44,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:47:44,680 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 04:47:44,680 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:47:44,681 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:47:44,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:44,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:47:45,039 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:45,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:47:45,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:47:45,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:47:45,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:47:45,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:47:45,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:47:45,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:47:45,165 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:47:45,181 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 04:47:45,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:47:45,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:47:49,091 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 5 queries submitted, 0 failed, 6717 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:47:49,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:47:49,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:47:49,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:47:50,009 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 04:47:50,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:47:50,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:47:50,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:47:50,089 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 04:47:50,091 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:47:50,092 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 04:47:50,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:50,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 04:47:50,506 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:50,507 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 04:47:50,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 04:47:50,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 04:47:50,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 04:47:50,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 04:47:50,586 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 04:47:50,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 04:47:50,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:47:50,632 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 04:47:50,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:47:50,676 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:47:50,696 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:47:50,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:47:50,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:47:50,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:47:50,766 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 04:47:50,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:47:50,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:47:50,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:47:50,816 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:50,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:47:50,854 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:50,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:50,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:50,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:50,926 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:50,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:50,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:50,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:50,999 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:51,017 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:47:51,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:51,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:51,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:51,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:51,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:51,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:51,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:51,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:51,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:51,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:47:51,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:51,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:51,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:51,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:51,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:51,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:51,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:51,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:51,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:51,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:47:51,395 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:51,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:51,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:51,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:51,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:51,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:51,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:51,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:51,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:51,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:47:51,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:51,612 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:51,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:51,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:51,670 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:51,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:51,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:51,727 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:51,746 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 04:47:51,749 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:47:51,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:52,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:47:52,255 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:52,255 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:47:52,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:47:52,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:47:52,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:47:52,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:47:52,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:47:52,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:47:52,352 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:47:52,368 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 04:47:52,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:47:52,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:47:52,642 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:47:52,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:47:53,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:47:53,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:47:53,266 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 04:47:53,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:47:53,336 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:47:53,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:47:53,440 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 04:47:53,440 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:47:53,443 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 04:47:53,520 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:53,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 04:47:53,821 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:53,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 04:47:53,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 04:47:53,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 04:47:53,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 04:47:53,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 04:47:53,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 04:47:53,920 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 04:47:53,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:47:53,954 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 04:47:53,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:47:54,002 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:47:54,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:47:54,035 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:47:54,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:47:54,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:47:54,082 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 04:47:54,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:47:54,099 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:47:54,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:47:54,135 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:54,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:47:54,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:54,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:54,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:54,221 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:54,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:54,260 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:54,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:54,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:47:54,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:54,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:47:54,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:54,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:54,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:54,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:54,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:54,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:54,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:54,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:47:54,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:54,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:47:54,520 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:54,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:54,553 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:54,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:54,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:54,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:54,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:54,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:47:54,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:54,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:47:54,676 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:54,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:54,706 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:54,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:54,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:54,757 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:54,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:54,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:47:54,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:54,826 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:47:54,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:54,864 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:54,882 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:54,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:54,920 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:54,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:54,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:54,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:47:54,996 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 04:47:55,000 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:47:55,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:55,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:47:55,388 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:55,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:47:55,403 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:47:55,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:47:55,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:47:55,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:47:55,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:47:55,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:47:55,484 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:47:55,500 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 04:47:55,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:47:55,540 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:47:56,811 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:47:57,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:47:57,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:47:57,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:47:57,773 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 04:47:57,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:47:57,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:47:57,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:47:57,818 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 04:47:57,818 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:47:57,819 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:47:57,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:47:58,114 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:47:58,180 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:47:58,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:47:58,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:47:58,211 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:47:58,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:47:58,244 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:47:58,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:47:58,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:47:58,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:47:58,313 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 04:47:58,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:47:58,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:47:59,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:47:59,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:48:00,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:48:00,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:48:00,450 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 04:48:00,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:48:00,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:48:00,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:48:00,562 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 04:48:00,563 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:48:00,565 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 04:48:00,647 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:48:00,882 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 04:48:00,950 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:48:00,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 04:48:00,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 04:48:00,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 04:48:00,999 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 04:48:01,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 04:48:01,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 04:48:01,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 04:48:01,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:48:01,099 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 04:48:01,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:48:01,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:48:01,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:48:01,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:48:01,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:48:01,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:48:01,229 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 04:48:01,229 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:48:01,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:48:01,260 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:48:01,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:01,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:48:01,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:01,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:01,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:01,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:01,395 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:01,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:01,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:01,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:01,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:01,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:48:01,489 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:01,505 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:01,522 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:01,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:01,556 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:01,573 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:01,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:01,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:01,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:01,647 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:48:01,668 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:01,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:01,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:01,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:01,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:01,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:01,790 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:01,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:01,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:01,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:48:01,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:01,890 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:01,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:01,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:01,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:01,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:01,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:02,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:02,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:02,040 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:48:02,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:02,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:02,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:02,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:02,137 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:02,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:02,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:02,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:02,217 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 04:48:02,220 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 04:48:02,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:48:02,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 04:48:02,599 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:48:02,599 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 04:48:02,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 04:48:02,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 04:48:02,640 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 04:48:02,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 04:48:02,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 04:48:02,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 04:48:02,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:48:02,721 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 04:48:02,744 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:48:02,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:48:02,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:48:02,793 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:48:02,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:48:02,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:48:02,832 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 04:48:02,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:48:02,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:48:02,859 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:48:02,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:02,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:48:02,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:02,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:02,943 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:02,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:02,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:02,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:03,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:03,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:03,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:03,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:48:03,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:03,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:03,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:03,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:03,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:03,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:03,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:03,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:03,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:03,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:48:03,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:03,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:03,265 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:03,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:03,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:03,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:03,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:03,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:03,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:03,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:48:03,392 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:03,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:03,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:03,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:03,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:03,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:03,487 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:03,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:03,520 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:03,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:48:03,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:03,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:03,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:03,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:03,622 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:03,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:03,658 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:03,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:03,693 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 04:48:03,696 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:48:03,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:48:04,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:48:04,138 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:48:04,140 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:48:04,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:48:04,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:48:04,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:48:04,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:48:04,252 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:48:04,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:48:04,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:48:04,296 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 04:48:04,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:48:04,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:48:05,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:48:06,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:48:06,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:48:06,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:48:06,983 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 04:48:06,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:48:07,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:48:07,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:48:07,069 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 04:48:07,069 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:48:07,072 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:48:07,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:48:07,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:48:07,506 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:48:07,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:48:07,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:48:07,540 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:48:07,556 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:48:07,581 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:48:07,599 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:48:07,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:48:07,637 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:48:07,654 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 04:48:07,678 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:48:07,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:48:08,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:48:08,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:48:08,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:48:08,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:48:08,719 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 04:48:08,719 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:48:08,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:48:08,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:48:08,788 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 04:48:08,789 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:48:08,790 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:48:09,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:48:09,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:48:10,111 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:48:10,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:48:10,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:48:10,265 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:48:10,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:48:10,469 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:48:10,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:48:10,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:48:10,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:48:10,612 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 04:48:10,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:48:10,676 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:48:13,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:48:13,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:48:13,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:48:14,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:48:14,180 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 04:48:14,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:48:14,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:48:14,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:48:14,242 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 04:48:14,242 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:48:14,243 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:48:14,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:48:14,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:48:14,645 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:48:14,647 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:48:14,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:48:14,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:48:14,701 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:48:14,724 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:48:14,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:48:14,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:48:14,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:48:14,802 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 04:48:14,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:48:14,854 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:48:15,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:48:16,066 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:48:16,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:48:16,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:48:16,687 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 04:48:16,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:48:16,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:48:16,737 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:48:16,760 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 04:48:16,762 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:48:16,763 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 04:48:16,862 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:48:17,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 04:48:17,193 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:48:17,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 04:48:17,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 04:48:17,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 04:48:17,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 04:48:17,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 04:48:17,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 04:48:17,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 04:48:17,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:48:17,323 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 04:48:17,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:48:17,364 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:48:17,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:48:17,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:48:17,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:48:17,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:48:17,446 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 04:48:17,446 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:48:17,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:48:17,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:48:17,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:17,517 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:48:17,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:17,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:17,581 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:17,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:17,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:17,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:17,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:17,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:48:17,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:17,721 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:48:17,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:17,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:17,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:17,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:17,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:17,846 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:17,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:17,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:48:17,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:17,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:48:17,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:17,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:17,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:18,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:18,022 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:18,040 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:18,061 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:18,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:48:18,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:18,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:48:18,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:18,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:18,168 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:18,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:18,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:18,221 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:18,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:18,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:48:18,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:18,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:48:18,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:18,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:18,352 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:18,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:18,390 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:18,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:18,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:18,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:48:18,467 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 04:48:18,467 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:48:18,470 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:48:18,561 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:48:18,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:48:18,888 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:48:18,890 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:48:18,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:48:18,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:48:18,948 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:48:18,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:48:18,982 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:48:18,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:48:19,015 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:19,522 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:48:19,523 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.05500006676. 2022-02-08 04:48:19,523 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:48:19,525 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:48:19,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:48:19,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:48:20,108 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:48:20,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:48:20,165 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:48:20,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:48:20,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:48:20,244 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:48:20,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:48:20,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:48:20,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:20,509 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:48:20,510 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:48:22,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:22,686 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:48:22,686 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:48:24,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:24,891 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:48:24,891 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:48:26,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:27,140 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:48:27,140 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:48:29,140 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:29,321 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:48:29,322 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:48:31,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:31,594 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:48:31,595 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:48:33,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:33,926 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:48:33,927 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:48:35,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:36,236 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:48:36,237 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:48:38,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:38,575 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:48:38,576 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:48:40,576 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:40,757 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:48:40,759 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:48:42,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:43,105 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:48:43,105 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:48:45,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:45,371 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:48:45,371 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:48:47,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:47,732 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:48:47,733 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 6719 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:48:49,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:49,957 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:48:49,959 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:48:51,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:52,263 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:48:52,263 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:48:54,265 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:54,493 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:48:54,494 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:48:56,494 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:56,667 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:48:56,667 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 37.1440000534. 2022-02-08 04:48:56,667 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:48:56,670 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:48:57,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:48:58,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:48:58,144 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:48:58,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:48:58,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:48:58,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:48:58,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:48:58,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:48:58,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:48:58,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:48:58,318 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:48:58,717 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:48:58,717 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 04:49:00,719 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:49:00,874 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:49:00,874 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.20600008965. 2022-02-08 04:49:00,875 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:49:00,877 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:49:01,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:49:01,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:49:01,887 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:49:01,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:49:01,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:49:01,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:49:01,969 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:49:01,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:49:02,002 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:49:02,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:49:02,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:49:02,644 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:49:02,644 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.76900005341. 2022-02-08 04:49:02,644 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:49:02,645 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:49:03,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:49:03,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:49:03,844 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:49:03,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:49:03,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:49:03,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:49:03,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:49:03,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:49:03,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:49:04,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:49:04,035 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:49:04,500 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:49:04,500 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.85599994659. 2022-02-08 04:49:04,500 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:49:04,502 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:49:05,441 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:49:05,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:49:05,805 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:49:05,806 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:49:05,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:49:05,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:49:05,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:49:05,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:49:05,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:49:05,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:49:05,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:49:06,367 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:49:06,367 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.867000103. 2022-02-08 04:49:06,367 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:49:06,368 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:49:06,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:49:07,244 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:49:07,331 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:49:07,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:49:07,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:49:07,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:49:07,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:49:07,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:49:07,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:49:07,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:49:07,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:49:07,990 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:49:07,990 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.62399983406. 2022-02-08 04:49:07,990 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:49:07,993 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:49:08,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:49:08,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:49:08,983 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:49:08,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:49:09,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:49:09,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:49:09,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:49:09,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:49:09,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:49:09,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:49:09,124 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:49:09,803 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:49:09,805 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.8140001297. 2022-02-08 04:49:09,805 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:49:09,806 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:49:10,604 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:49:10,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:49:11,102 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:49:11,104 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:49:11,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:49:11,135 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:49:11,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:49:11,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:49:11,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:49:11,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:49:11,214 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:49:11,700 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:49:11,701 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.89699983597. 2022-02-08 04:49:11,701 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:49:11,703 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:49:12,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:49:12,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:49:12,717 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:49:12,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:49:12,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:49:12,747 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:49:12,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:49:12,778 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:49:12,790 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:49:12,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:49:12,834 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:49:13,328 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:49:13,328 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.62700009346. ############################################################ # # RAM FootPrint: 4276.3515625 # ############################################################ 2022-02-08 04:49:13,611 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 04:49:13,611 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 103 seconds. Reason: Iteration:6 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 6741 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 6767 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:50:56,676 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 04:50:56,677 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 04:51:01,688 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:51:01,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:02,012 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:51:02,082 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:02,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:51:02,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:51:02,109 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:51:02,124 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:51:02,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:51:02,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:51:02,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:51:02,183 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:51:02,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:51:02,253 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['126638 memcached\n'], []) 2022-02-08 04:51:02,255 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 04:51:02,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:51:02,312 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['126638 memcached\n'], []) 2022-02-08 04:51:02,313 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:51:02,315 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:51:02,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:03,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:51:03,358 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:03,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:51:03,372 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:51:03,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:51:03,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:51:03,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:51:03,436 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:51:03,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:51:03,482 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:51:03,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:51:03,536 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['38550 memcached\n'], []) 2022-02-08 04:51:03,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 04:51:03,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:51:03,586 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['38550 memcached\n'], []) 2022-02-08 04:51:03,586 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:51:03,595 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:51:03,670 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:03,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:51:03,948 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:03,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:51:03,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:51:03,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:51:03,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:51:04,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:51:04,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:51:04,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:51:04,055 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:51:04,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:51:04,098 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['26296 memcached\n'], []) 2022-02-08 04:51:04,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 04:51:04,124 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:51:04,148 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['26296 memcached\n'], []) 2022-02-08 04:51:04,148 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:51:04,151 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:51:04,242 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:04,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:51:04,539 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:04,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:51:04,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:51:04,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:51:04,581 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:51:04,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:51:04,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:51:04,622 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:51:04,648 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:51:04,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:51:04,696 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['60588 memcached\n'], []) 2022-02-08 04:51:04,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 04:51:04,721 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:51:04,743 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['60588 memcached\n'], []) 2022-02-08 04:51:04,744 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:51:04,746 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:51:04,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:05,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:51:05,099 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:05,099 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:51:05,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:51:05,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:51:05,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:51:05,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:51:05,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:51:05,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:51:05,193 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:51:05,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:51:05,247 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['54413 memcached\n'], []) 2022-02-08 04:51:05,247 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 04:51:05,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:51:05,299 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['54413 memcached\n'], []) 2022-02-08 04:51:05,299 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:51:05,302 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:51:05,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:05,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:51:05,684 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:05,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:51:05,698 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:51:05,711 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:51:05,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:51:05,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:51:05,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:51:05,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:51:05,786 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:51:05,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:51:05,829 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['89333 memcached\n'], []) 2022-02-08 04:51:05,829 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 04:51:05,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:51:05,875 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['89333 memcached\n'], []) 2022-02-08 04:51:05,875 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:51:05,878 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:51:05,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:06,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:51:06,303 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:06,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:51:06,318 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:51:06,329 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:51:06,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:51:06,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:51:06,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:51:06,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:51:06,401 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:51:06,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:51:06,460 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['23170 memcached\n'], []) 2022-02-08 04:51:06,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 04:51:06,484 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:51:06,506 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['23170 memcached\n'], []) 2022-02-08 04:51:06,506 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:51:06,507 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:51:06,624 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:06,826 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:51:06,884 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:06,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:51:06,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:51:06,910 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:51:06,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:51:06,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:51:06,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:51:06,961 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:51:06,977 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:51:06,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:51:07,042 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['38081 memcached\n'], []) 2022-02-08 04:51:07,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 04:51:07,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:51:07,086 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['38081 memcached\n'], []) 2022-02-08 04:51:07,086 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:51:07,088 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:51:07,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:07,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:51:07,476 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:07,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:51:07,489 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:51:07,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:51:07,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:51:07,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:51:07,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:51:07,556 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:51:07,569 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:51:07,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:51:07,631 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['74114 memcached\n'], []) 2022-02-08 04:51:07,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 04:51:07,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:51:07,696 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['74114 memcached\n'], []) 2022-02-08 04:51:07,696 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:51:07,698 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:51:07,793 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:08,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:51:08,073 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:08,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:51:08,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:51:08,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:51:08,114 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:51:08,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:51:08,145 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:51:08,160 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:51:08,174 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:51:08,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:51:08,220 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['103753 memcached\n'], []) 2022-02-08 04:51:08,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 04:51:08,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:51:08,266 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['103753 memcached\n'], []) 2022-02-08 04:51:08,266 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:51:08,267 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:51:08,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:08,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:51:08,638 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:08,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:51:08,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:51:08,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:51:08,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:51:08,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:51:08,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:51:08,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:51:08,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:51:08,756 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 04:51:08,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:51:08,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:09,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:51:09,573 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:09,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:10,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:51:10,040 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 04:51:10,040 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:51:10,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:10,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:51:10,115 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 04:51:10,115 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:51:10,117 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:51:10,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:10,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:51:10,500 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:10,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:51:10,515 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:51:10,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:51:10,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:51:10,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:51:10,568 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:51:10,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:51:10,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:51:10,609 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 04:51:10,640 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:51:10,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:10,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:51:11,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:11,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:11,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:51:11,115 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 04:51:11,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:51:11,130 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:11,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:51:11,171 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 04:51:11,171 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:51:11,174 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:51:11,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:11,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:51:11,558 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:11,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:51:11,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:51:11,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:51:11,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:51:11,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:51:11,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:51:11,637 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:51:11,650 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:51:11,680 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 04:51:11,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:51:11,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:12,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:51:12,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:12,785 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:13,012 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:51:13,028 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 04:51:13,028 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:51:13,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:13,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:51:13,111 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 04:51:13,112 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:51:13,148 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 04:51:13,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:13,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 04:51:13,515 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:13,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 04:51:13,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 04:51:13,543 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 04:51:13,558 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 04:51:13,576 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 04:51:13,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 04:51:13,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 04:51:13,622 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:51:13,638 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 04:51:13,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:51:13,681 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:13,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:51:13,711 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:13,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:13,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:51:13,756 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 04:51:13,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:51:13,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:13,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:51:13,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:13,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:51:13,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:13,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:13,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:13,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:13,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:13,920 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:13,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:13,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:13,971 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:13,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:51:14,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:14,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:14,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:14,061 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:14,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:14,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:14,114 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:14,130 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:14,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:14,160 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:51:14,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:14,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:14,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:14,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:14,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:14,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:14,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:14,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:14,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:14,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:51:14,341 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:14,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:14,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:14,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:14,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:14,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:14,436 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:14,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:14,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:14,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:51:14,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:14,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:14,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:14,556 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:14,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:14,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:14,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:14,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:14,644 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 04:51:14,645 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:51:14,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:14,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:51:15,019 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:15,020 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:51:15,035 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:51:15,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:51:15,061 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:51:15,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:51:15,089 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:51:15,102 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:51:15,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:51:15,131 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 04:51:15,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:51:15,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:15,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:51:16,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:16,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:16,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:51:16,424 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 04:51:16,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:51:16,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:16,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:51:16,503 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 04:51:16,503 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:51:16,506 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 04:51:16,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:16,798 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 04:51:16,869 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:16,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 04:51:16,885 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 04:51:16,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 04:51:16,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 04:51:16,931 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 04:51:16,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 04:51:16,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 04:51:16,971 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:51:16,987 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 04:51:17,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:51:17,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:17,051 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:51:17,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:17,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:17,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:51:17,105 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 04:51:17,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:51:17,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:17,135 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:51:17,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:17,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:51:17,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:17,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:17,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:17,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:17,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:17,270 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:17,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:17,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:17,323 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:17,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:51:17,357 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:17,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:17,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:17,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:17,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:17,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:17,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:17,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:17,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:17,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:51:17,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:17,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:17,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:17,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:17,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:17,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:17,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:17,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:17,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:17,698 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:51:17,719 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:17,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:17,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:17,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:17,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:17,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:17,821 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:17,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:17,854 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:17,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:51:17,892 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:17,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:17,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:17,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:17,964 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:17,982 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:17,999 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:18,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:18,030 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 04:51:18,033 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:51:18,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:18,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:51:18,387 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:18,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:51:18,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:51:18,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:51:18,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:51:18,438 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:51:18,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:51:18,464 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:51:18,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:51:18,493 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 04:51:18,522 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:51:18,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:20,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:51:21,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:21,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:21,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:51:21,766 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 04:51:21,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:51:21,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:21,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:51:21,825 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 04:51:21,825 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:51:21,828 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:51:21,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:22,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:51:22,209 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:22,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:51:22,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:51:22,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:51:22,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:51:22,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:51:22,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:51:22,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:51:22,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:51:22,335 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 04:51:22,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:51:22,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:24,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:51:24,862 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:25,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:25,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:51:25,331 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 04:51:25,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:51:25,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:25,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:51:25,417 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 04:51:25,417 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:51:25,420 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 04:51:25,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:25,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 04:51:25,770 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:25,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 04:51:25,783 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 04:51:25,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 04:51:25,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 04:51:25,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 04:51:25,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 04:51:25,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 04:51:25,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:51:25,875 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 04:51:25,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:51:25,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:25,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:51:25,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:25,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:25,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:51:25,980 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 04:51:25,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:51:25,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:26,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:51:26,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:26,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:51:26,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:26,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:26,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:26,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:26,107 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:26,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:26,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:26,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:26,168 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:26,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:51:26,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:26,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:26,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:26,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:26,275 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:26,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:26,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:26,323 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:26,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:26,357 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:51:26,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:26,390 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:26,408 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:26,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:26,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:26,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:26,489 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:26,505 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:26,522 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:26,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:51:26,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:26,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:26,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:26,599 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:26,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:26,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:26,647 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:26,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:26,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:26,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:51:26,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:26,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:26,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:26,762 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:26,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:26,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:26,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:26,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:26,839 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 04:51:26,842 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 04:51:26,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:27,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 04:51:27,279 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:27,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 04:51:27,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 04:51:27,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 04:51:27,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 04:51:27,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 04:51:27,354 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 04:51:27,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 04:51:27,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:51:27,403 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 04:51:27,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:51:27,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:27,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:51:27,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:27,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:27,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:51:27,523 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 04:51:27,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:51:27,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:27,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:51:27,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:27,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:51:27,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:27,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:27,635 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:27,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:27,670 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:27,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:27,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:27,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:27,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:27,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:51:27,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:27,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:27,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:27,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:27,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:27,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:27,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:27,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:27,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:27,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:51:27,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:27,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:27,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:27,999 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:28,015 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:28,030 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:28,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:28,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:28,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:28,094 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:51:28,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:28,124 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:28,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:28,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:28,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:28,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:28,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:28,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:28,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:28,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:51:28,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:28,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:28,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:28,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:28,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:28,352 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:28,367 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:28,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:28,397 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 04:51:28,398 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:51:28,497 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:28,762 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:51:28,829 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:28,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:51:28,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:51:28,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:51:28,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:51:28,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:51:28,910 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:51:28,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:51:28,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:51:28,960 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 04:51:28,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:51:29,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:31,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:51:31,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:31,885 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:32,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:51:32,163 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 04:51:32,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:51:32,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:32,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:51:32,226 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 04:51:32,226 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:51:32,229 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:51:32,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:32,788 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:51:33,055 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:33,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:51:33,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:51:33,280 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:51:33,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:51:33,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:51:33,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:51:33,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:51:33,696 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:51:33,713 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 04:51:33,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:51:33,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:34,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:51:34,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:34,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:34,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:51:34,880 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 04:51:34,880 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:51:34,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:35,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:51:35,092 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 04:51:35,094 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:51:35,095 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:51:35,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:36,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:51:36,937 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:36,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:51:37,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:51:37,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:51:37,186 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:51:37,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:51:37,270 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:51:37,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:51:37,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:51:37,344 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 04:51:37,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:51:37,385 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:37,775 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:51:38,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:38,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:38,612 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:51:38,676 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 04:51:38,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:51:38,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:38,785 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:51:38,848 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 04:51:38,848 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:51:38,849 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:51:38,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:39,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:51:39,263 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:39,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:51:39,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:51:39,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:51:39,313 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:51:39,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:51:39,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:51:39,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:51:39,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:51:39,391 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 04:51:39,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:51:39,438 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:43,372 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:51:43,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:44,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:44,336 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:51:44,351 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 04:51:44,352 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:51:44,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:44,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:51:44,427 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 04:51:44,428 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:51:44,430 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 04:51:44,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:44,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 04:51:44,825 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:44,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 04:51:44,844 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 04:51:44,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 04:51:44,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 04:51:44,892 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 04:51:44,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 04:51:44,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 04:51:44,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:51:44,953 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 04:51:44,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:51:44,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:45,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:51:45,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:45,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:51:45,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:51:45,085 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 04:51:45,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:51:45,099 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:51:45,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:51:45,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:45,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:51:45,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:45,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:45,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:45,229 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:45,249 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:45,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:45,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:45,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:51:45,323 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:45,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:51:45,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:45,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:45,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:45,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:45,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:45,448 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:45,469 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:45,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:51:45,505 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:45,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:51:45,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:45,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:45,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:45,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:45,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:45,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:45,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:45,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:51:45,678 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:45,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:51:45,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:45,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:45,752 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:45,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:45,785 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:45,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:45,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:45,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:51:45,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:45,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:51:45,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:45,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:45,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:45,946 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:45,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:45,982 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:46,002 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:46,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:51:46,048 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 04:51:46,048 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:51:46,049 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:51:46,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:46,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:51:46,548 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:46,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:51:46,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:51:46,579 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:51:46,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:51:46,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:51:46,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:51:46,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:51:46,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:51:47,224 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:51:47,224 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.17699980736. 2022-02-08 04:51:47,224 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:51:47,226 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:51:47,321 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:51:47,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:51:47,654 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:51:47,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:51:47,678 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:51:47,691 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:51:47,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:51:47,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:51:47,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:51:47,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:51:47,793 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:51:47,950 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:51:47,951 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 6782 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:51:49,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:51:50,102 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:51:50,104 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:51:52,104 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:51:52,249 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:51:52,250 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:51:54,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:51:54,397 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:51:54,397 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:51:56,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:51:56,549 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:51:56,549 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:51:58,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:51:58,907 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:51:58,907 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:52:00,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:01,365 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:52:01,367 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:52:03,367 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:03,632 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:52:03,632 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:52:05,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:05,839 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:52:05,841 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:52:07,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:08,105 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:52:08,105 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:52:10,107 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:10,427 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:52:10,427 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:52:12,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:12,776 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:52:12,776 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:52:14,778 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:15,049 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:52:15,049 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:52:17,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:17,619 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:52:17,621 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:52:19,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:20,121 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:52:20,121 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:52:22,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:22,448 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:52:22,448 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:52:24,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:24,828 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:52:24,828 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:52:26,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:27,220 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:52:27,221 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:52:29,221 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:29,565 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:52:29,565 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:52:31,566 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:31,970 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:52:31,970 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:52:33,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:34,387 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:52:34,387 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:52:36,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:36,563 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:52:36,563 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:52:38,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:38,816 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:52:38,818 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 51.5930001736. 2022-02-08 04:52:38,818 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:52:38,819 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:52:40,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:52:40,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:52:41,038 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:52:41,038 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:52:41,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:52:41,094 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:52:41,140 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:52:41,163 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:52:41,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:52:41,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:52:41,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:42,382 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:52:42,382 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.5649998188. 2022-02-08 04:52:42,384 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:52:42,384 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:52:43,489 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:52:44,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:52:44,420 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:52:44,420 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:52:44,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:52:44,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:52:44,487 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:52:44,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:52:44,535 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:52:44,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:52:44,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:45,530 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:52:45,535 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.15200018883. 2022-02-08 04:52:45,535 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:52:45,536 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:52:46,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:52:47,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:52:47,309 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:52:47,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:52:47,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:52:47,349 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:52:47,367 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:52:47,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:52:47,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:52:47,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:52:47,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 6789 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:52:49,618 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:52:49,618 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.08299994469. 2022-02-08 04:52:49,618 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:52:49,619 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:52:50,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:52:51,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:52:51,207 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:52:51,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:52:51,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:52:51,244 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:52:51,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:52:51,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:52:51,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:52:51,323 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:52:51,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:52,204 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:52:52,206 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.58800005913. 2022-02-08 04:52:52,206 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:52:52,207 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:52:53,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:52:53,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:52:53,538 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:52:53,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:52:53,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:52:53,566 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:52:53,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:52:53,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:52:53,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:52:53,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:52:53,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:54,142 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:52:54,144 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.93799996376. 2022-02-08 04:52:54,144 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:52:54,145 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:52:54,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:52:54,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:52:54,954 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:52:54,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:52:54,969 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:52:54,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:52:55,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:52:55,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:52:55,028 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:52:55,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:52:55,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:55,483 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:52:55,484 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.34099984169. 2022-02-08 04:52:55,484 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:52:55,486 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:52:55,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:52:56,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:52:56,296 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:52:56,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:52:56,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:52:56,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:52:56,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:52:56,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:52:56,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:52:56,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:52:56,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:56,823 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:52:56,823 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.33899998665. 2022-02-08 04:52:56,825 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:52:56,826 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:52:57,464 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:52:57,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:52:57,855 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:52:57,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:52:57,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:52:57,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:52:57,900 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:52:57,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:52:57,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:52:57,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:52:57,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:52:58,617 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:52:58,617 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.79200005531. ############################################################ # # RAM FootPrint: 4306.109375 # ############################################################ 2022-02-08 04:52:58,931 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 04:52:58,933 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 86 seconds. Reason: Iteration:7 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 6810 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:54:24,937 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 04:54:24,937 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 04:54:29,999 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:54:30,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:30,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:54:30,427 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:30,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:54:30,446 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:54:30,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:54:30,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:54:30,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:54:30,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:54:30,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:54:30,540 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:54:30,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:54:30,598 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['127291 memcached\n'], []) 2022-02-08 04:54:30,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 04:54:30,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:54:30,655 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['127291 memcached\n'], []) 2022-02-08 04:54:30,655 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:54:30,658 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:54:31,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:31,385 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:54:31,476 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:31,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:54:31,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:54:31,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:54:31,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:54:31,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:54:31,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:54:31,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:54:31,604 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:54:31,604 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:54:31,663 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['39442 memcached\n'], []) 2022-02-08 04:54:31,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 04:54:31,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:54:31,707 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['39442 memcached\n'], []) 2022-02-08 04:54:31,707 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:54:31,710 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:54:31,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:32,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:54:32,085 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:32,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:54:32,102 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:54:32,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:54:32,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:54:32,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:54:32,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:54:32,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:54:32,196 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:54:32,196 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:54:32,263 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['27067 memcached\n'], []) 2022-02-08 04:54:32,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 04:54:32,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:54:32,315 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['27067 memcached\n'], []) 2022-02-08 04:54:32,315 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:54:32,316 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:54:32,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:32,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:54:32,680 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:32,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:54:32,693 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:54:32,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:54:32,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:54:32,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:54:32,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:54:32,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:54:32,769 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:54:32,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:54:32,811 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['61259 memcached\n'], []) 2022-02-08 04:54:32,811 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 04:54:32,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:54:32,861 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['61259 memcached\n'], []) 2022-02-08 04:54:32,861 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:54:32,862 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:54:32,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:33,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:54:33,226 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:33,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:54:33,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:54:33,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:54:33,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:54:33,285 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:54:33,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:54:33,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:54:33,322 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:54:33,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:54:33,372 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['55072 memcached\n'], []) 2022-02-08 04:54:33,372 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 04:54:33,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:54:33,418 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['55072 memcached\n'], []) 2022-02-08 04:54:33,418 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:54:33,421 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:54:33,505 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:33,719 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:54:33,773 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:33,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:54:33,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:54:33,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:54:33,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:54:33,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:54:33,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:54:33,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:54:33,875 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:54:33,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:54:33,937 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['89988 memcached\n'], []) 2022-02-08 04:54:33,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 04:54:33,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:54:33,982 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['89988 memcached\n'], []) 2022-02-08 04:54:33,982 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:54:33,983 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:54:34,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:34,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:54:34,387 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:34,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:54:34,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:54:34,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:54:34,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:54:34,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:54:34,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:54:34,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:54:34,480 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:54:34,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:54:34,546 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['23835 memcached\n'], []) 2022-02-08 04:54:34,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 04:54:34,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:54:34,598 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['23835 memcached\n'], []) 2022-02-08 04:54:34,598 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:54:34,601 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:54:34,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:34,900 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:54:34,966 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:34,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:54:34,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:54:34,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:54:35,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:54:35,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:54:35,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:54:35,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:54:35,066 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:54:35,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:54:35,128 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['38747 memcached\n'], []) 2022-02-08 04:54:35,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 04:54:35,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:54:35,174 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['38747 memcached\n'], []) 2022-02-08 04:54:35,174 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:54:35,177 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:54:35,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:35,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:54:35,539 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:35,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:54:35,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:54:35,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:54:35,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:54:35,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:54:35,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:54:35,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:54:35,645 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:54:35,647 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:54:35,694 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['74773 memcached\n'], []) 2022-02-08 04:54:35,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 04:54:35,719 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:54:35,739 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['74773 memcached\n'], []) 2022-02-08 04:54:35,739 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:54:35,740 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:54:35,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:36,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:54:36,114 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:36,114 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:54:36,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:54:36,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:54:36,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:54:36,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:54:36,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:54:36,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:54:36,210 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:54:36,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:54:36,263 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['104462 memcached\n'], []) 2022-02-08 04:54:36,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 04:54:36,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:54:36,328 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['104462 memcached\n'], []) 2022-02-08 04:54:36,328 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:54:36,329 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:54:36,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:36,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:54:36,698 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:36,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:54:36,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:54:36,727 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:54:36,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:54:36,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:54:36,775 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:54:36,788 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:54:36,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:54:36,822 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 04:54:36,849 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:54:36,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:37,341 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:54:37,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:37,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:38,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:54:38,198 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 04:54:38,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:54:38,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:38,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:54:38,249 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 04:54:38,250 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:54:38,252 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:54:38,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:38,553 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:54:38,630 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:38,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:54:38,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:54:38,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:54:38,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:54:38,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:54:38,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:54:38,721 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:54:38,734 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:54:38,750 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 04:54:38,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:54:38,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:38,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:54:38,923 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:38,961 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:38,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:54:39,010 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 04:54:39,012 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:54:39,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:39,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:54:39,055 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 04:54:39,055 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:54:39,058 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:54:39,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:39,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:54:39,430 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:39,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:54:39,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:54:39,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:54:39,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:54:39,492 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:54:39,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:54:39,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:54:39,533 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:54:39,548 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 04:54:39,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:54:39,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:43,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:54:43,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:43,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:44,089 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:54:44,105 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 04:54:44,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:54:44,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:44,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:54:44,184 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 04:54:44,184 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:54:44,187 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 04:54:44,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:44,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 04:54:44,573 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:44,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 04:54:44,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 04:54:44,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 04:54:44,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 04:54:44,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 04:54:44,650 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 04:54:44,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 04:54:44,678 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:54:44,694 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 04:54:44,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:54:44,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:44,747 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:54:44,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:44,783 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:44,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:54:44,815 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 04:54:44,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:54:44,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:44,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:54:44,854 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:44,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:54:44,890 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:44,905 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:44,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:44,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:44,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:44,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:44,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:45,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:45,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:45,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:54:45,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:45,071 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:45,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:45,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:45,114 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:45,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:45,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:45,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:45,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:45,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:54:45,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:45,229 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:45,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:45,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:45,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:45,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:45,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:45,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:45,341 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:45,357 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:54:45,372 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:45,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:45,405 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:45,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:45,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:45,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:45,477 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:45,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:45,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:45,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:54:45,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:45,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:45,573 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:45,591 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:45,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:45,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:45,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:45,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:45,678 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 04:54:45,680 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:54:45,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:46,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:54:46,069 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:46,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:54:46,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:54:46,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:54:46,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:54:46,130 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:54:46,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:54:46,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:54:46,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:54:46,187 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 04:54:46,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:54:46,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:48,163 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:54:48,599 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:48,867 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:49,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:54:49,046 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 04:54:49,048 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:54:49,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:49,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:54:49,132 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 04:54:49,132 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:54:49,135 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 04:54:49,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 6832 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:54:49,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 04:54:49,523 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:49,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 04:54:49,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 04:54:49,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 04:54:49,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 04:54:49,591 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 04:54:49,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 04:54:49,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 04:54:49,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:54:49,657 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 04:54:49,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:54:49,706 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:49,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:54:49,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:49,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:49,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:54:49,786 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 04:54:49,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:54:49,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:49,821 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:54:49,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:49,857 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:54:49,877 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:49,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:49,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:49,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:49,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:49,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:49,999 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:50,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:50,038 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:50,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:54:50,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:50,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:50,109 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:50,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:50,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:50,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:50,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:50,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:50,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:50,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:54:50,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:50,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:50,295 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:50,313 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:50,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:50,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:50,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:50,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:50,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:50,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:54:50,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:50,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:50,484 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:50,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:50,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:50,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:50,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:50,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:50,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:50,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:54:50,624 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:50,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:50,658 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:50,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:50,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:50,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:50,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:50,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:50,766 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 04:54:50,769 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:54:50,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:51,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:54:51,091 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:51,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:54:51,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:54:51,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:54:51,130 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:54:51,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:54:51,158 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:54:51,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:54:51,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:54:51,203 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 04:54:51,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:54:51,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:53,028 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:54:53,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:53,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:53,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:54:54,009 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 04:54:54,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:54:54,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:54,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:54:54,055 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 04:54:54,056 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:54:54,059 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:54:54,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:54,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:54:54,421 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:54,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:54:54,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:54:54,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:54:54,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:54:54,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:54:54,505 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:54:54,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:54:54,535 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:54:54,552 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 04:54:54,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:54:54,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:56,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:54:57,303 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:57,545 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:57,765 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:54:57,783 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 04:54:57,785 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:54:57,803 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:57,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:54:57,872 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 04:54:57,874 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:54:57,875 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 04:54:57,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:58,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 04:54:58,313 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:58,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 04:54:58,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 04:54:58,354 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 04:54:58,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 04:54:58,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 04:54:58,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 04:54:58,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 04:54:58,441 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:54:58,457 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 04:54:58,484 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:54:58,512 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:58,533 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:54:58,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:58,566 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:54:58,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:54:58,599 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 04:54:58,599 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:54:58,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:54:58,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:54:58,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:58,668 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:54:58,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:58,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:58,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:58,744 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:58,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:58,785 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:58,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:58,826 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:54:58,846 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:58,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:54:58,885 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:58,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:58,920 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:58,936 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:58,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:58,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:58,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:59,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:54:59,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:59,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:54:59,066 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:59,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:59,102 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:59,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:59,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:59,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:59,176 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:59,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:54:59,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:59,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:54:59,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:59,260 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:59,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:59,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:59,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:59,326 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:59,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:59,364 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:54:59,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:59,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:54:59,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:59,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:59,469 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:59,489 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:59,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:59,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:59,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:59,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:54:59,584 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 04:54:59,585 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 04:54:59,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:54:59,905 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 04:54:59,964 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:54:59,964 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 04:54:59,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 04:54:59,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 04:55:00,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 04:55:00,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 04:55:00,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 04:55:00,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 04:55:00,089 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:55:00,105 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 04:55:00,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:55:00,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:55:00,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:55:00,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:55:00,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:55:00,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:55:00,223 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 04:55:00,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:55:00,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:55:00,255 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:55:00,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:00,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:55:00,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:00,326 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:00,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:00,364 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:00,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:00,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:00,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:00,446 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:00,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:00,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:55:00,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:00,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:00,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:00,612 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:00,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:00,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:00,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:00,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:00,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:00,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:55:00,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:00,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:00,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:00,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:00,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:00,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:00,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:00,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:00,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:00,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:55:00,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:00,918 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:00,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:00,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:00,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:00,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:01,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:01,028 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:01,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:01,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:55:01,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:01,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:01,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:01,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:01,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:01,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:01,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:01,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:01,232 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 04:55:01,233 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:55:01,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:55:01,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:55:01,953 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:55:01,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:55:01,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:55:01,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:55:01,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:55:02,017 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:55:02,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:55:02,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:55:02,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:55:02,076 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 04:55:02,104 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:55:02,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:55:02,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:55:03,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:55:03,645 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:55:03,923 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:55:03,957 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 04:55:03,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:55:04,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:55:04,053 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:55:04,068 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 04:55:04,069 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:55:04,071 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:55:04,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:55:04,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:55:04,509 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:55:04,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:55:04,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:55:04,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:55:04,556 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:55:04,576 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:55:04,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:55:04,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:55:04,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:55:04,638 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 04:55:04,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:55:04,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:55:05,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:55:05,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:55:05,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:55:05,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:55:05,977 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 04:55:05,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:55:06,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:55:06,040 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:55:06,058 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 04:55:06,059 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:55:06,061 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:55:06,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:55:07,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:55:07,148 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:55:07,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:55:07,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:55:07,191 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:55:07,255 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:55:07,275 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:55:07,303 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:55:07,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:55:07,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:55:07,377 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 04:55:07,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:55:07,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:55:07,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:55:08,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:55:08,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:55:08,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:55:08,496 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 04:55:08,497 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:55:08,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:55:08,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:55:08,719 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 04:55:08,719 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:55:08,720 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:55:08,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:55:09,035 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:55:09,132 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:55:09,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:55:09,158 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:55:09,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:55:09,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:55:09,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:55:09,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:55:09,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:55:09,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:55:09,309 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 04:55:09,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:55:09,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:55:10,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:55:10,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:55:11,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:55:11,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:55:11,542 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 04:55:11,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:55:11,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:55:11,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:55:11,592 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 04:55:11,592 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:55:11,594 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 04:55:11,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:55:11,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 04:55:11,976 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:55:11,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 04:55:11,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 04:55:12,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 04:55:12,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 04:55:12,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 04:55:12,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 04:55:12,073 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 04:55:12,089 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:55:12,108 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 04:55:12,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:55:12,160 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:55:12,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:55:12,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:55:12,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:55:12,249 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:55:12,263 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 04:55:12,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:55:12,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:55:12,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:55:12,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:12,326 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:55:12,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:12,362 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:12,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:12,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:12,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:12,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:12,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:12,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:55:12,497 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:12,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:55:12,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:12,548 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:12,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:12,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:12,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:12,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:12,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:12,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:55:12,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:12,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:55:12,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:12,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:12,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:12,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:12,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:12,816 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:12,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:12,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:55:12,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:12,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:55:12,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:12,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:12,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:12,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:12,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:13,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:13,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:13,030 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:55:13,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:13,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:55:13,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:13,104 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:13,124 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:13,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:13,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:13,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:13,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:13,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:55:13,236 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 04:55:13,236 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:55:13,239 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:55:13,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:55:13,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:55:13,638 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:55:13,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:55:13,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:55:13,670 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:55:13,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:55:13,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:55:13,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:55:13,765 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:55:13,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:14,576 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:55:14,578 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.33999991417. 2022-02-08 04:55:14,578 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:55:14,579 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:55:14,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:55:14,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:55:14,976 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:55:14,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:55:14,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:55:15,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:55:15,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:55:15,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:55:15,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:55:15,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:55:15,094 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:15,345 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:15,346 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:17,346 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:17,828 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:17,828 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:19,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:20,079 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:20,079 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:22,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:22,244 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:22,244 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:24,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:24,460 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:24,460 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:26,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:26,658 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:26,660 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:28,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:28,918 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:28,920 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:30,920 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:31,259 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:31,260 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:33,260 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:33,937 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:33,937 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:35,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:36,326 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:36,326 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:38,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:38,811 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:38,812 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:40,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:41,313 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:41,315 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:43,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:43,625 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:43,625 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:45,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:45,966 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:45,966 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:47,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:48,255 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:48,256 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 6834 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:55:50,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:50,536 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:50,536 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:52,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:52,759 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:52,759 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:54,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:55,046 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:55,046 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:57,048 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:57,266 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:55:57,266 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:55:59,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:55:59,450 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:55:59,450 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 44.871999979. 2022-02-08 04:55:59,450 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:55:59,467 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:56:00,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:56:00,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:56:01,040 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:56:01,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:56:01,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:56:01,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:56:01,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:56:01,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:56:01,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:56:01,278 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:56:01,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:56:01,801 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:56:01,801 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.35100007057. 2022-02-08 04:56:01,802 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:56:01,803 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:56:02,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:56:03,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:56:03,260 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:56:03,260 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:56:03,278 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:56:03,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:56:03,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:56:03,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:56:03,362 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:56:03,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:56:03,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:56:04,125 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:56:04,125 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.32399988174. 2022-02-08 04:56:04,125 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:56:04,128 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:56:05,512 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:56:05,974 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:56:06,144 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:56:06,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:56:06,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:56:06,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:56:06,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:56:06,283 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:56:06,362 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:56:06,433 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:56:06,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:56:07,835 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:56:07,835 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.71000003815. 2022-02-08 04:56:07,835 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:56:07,838 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:56:08,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:56:09,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:56:09,165 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:56:09,165 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:56:09,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:56:09,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:56:09,252 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:56:09,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:56:09,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:56:09,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:56:09,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:56:10,417 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:56:10,418 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.58299994469. 2022-02-08 04:56:10,418 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:56:10,421 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:56:11,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:56:11,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:56:11,898 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:56:11,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:56:11,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:56:11,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:56:11,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:56:11,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:56:11,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:56:12,015 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:56:12,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:56:13,296 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:56:13,298 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.87900018692. 2022-02-08 04:56:13,298 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:56:13,299 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:56:14,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:56:14,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:56:14,709 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:56:14,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:56:14,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:56:14,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:56:14,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:56:14,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:56:14,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:56:14,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:56:14,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:56:16,164 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:56:16,164 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.86699986458. 2022-02-08 04:56:16,164 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:56:16,167 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:56:16,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:56:16,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:56:17,016 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:56:17,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:56:17,030 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:56:17,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:56:17,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:56:17,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:56:17,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:56:17,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:56:17,135 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:56:17,509 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:56:17,509 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.34400010109. 2022-02-08 04:56:17,509 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:56:17,510 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:56:18,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:56:18,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:56:18,395 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:56:18,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:56:18,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:56:18,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:56:18,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:56:18,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:56:18,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:56:18,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:56:18,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:56:19,173 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:56:19,173 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.66400003433. ############################################################ # # RAM FootPrint: 4329.41015625 # ############################################################ 2022-02-08 04:56:19,523 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 04:56:19,523 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 117 seconds. Reason: Iteration:8 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 6855 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 6879 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:58:16,529 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 04:58:16,529 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 04:58:21,532 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:58:21,619 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:21,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:58:21,937 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:21,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:58:21,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:58:21,971 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:58:21,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:58:22,028 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:58:22,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:58:22,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:58:22,072 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:58:22,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:58:22,131 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['127995 memcached\n'], []) 2022-02-08 04:58:22,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 04:58:22,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 04:58:22,180 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['127995 memcached\n'], []) 2022-02-08 04:58:22,180 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:58:22,183 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:58:22,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:22,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:58:23,009 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:23,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:58:23,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:58:23,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:58:23,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:58:23,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:58:23,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:58:23,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:58:23,125 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:58:23,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:58:23,183 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['40361 memcached\n'], []) 2022-02-08 04:58:23,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 04:58:23,209 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 04:58:23,230 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['40361 memcached\n'], []) 2022-02-08 04:58:23,230 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:58:23,233 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:58:23,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:23,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:58:23,598 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:23,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:58:23,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:58:23,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:58:23,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:58:23,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:58:23,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:58:23,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:58:23,704 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:58:23,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:58:23,747 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['27861 memcached\n'], []) 2022-02-08 04:58:23,747 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 04:58:23,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 04:58:23,798 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['27861 memcached\n'], []) 2022-02-08 04:58:23,799 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:58:23,801 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:58:23,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:24,124 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:58:24,180 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:24,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:58:24,196 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:58:24,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:58:24,221 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:58:24,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:58:24,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:58:24,265 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:58:24,280 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:58:24,280 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:58:24,345 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['61959 memcached\n'], []) 2022-02-08 04:58:24,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 04:58:24,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 04:58:24,391 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['61959 memcached\n'], []) 2022-02-08 04:58:24,391 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:58:24,392 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:58:24,469 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:24,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:58:24,729 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:24,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:58:24,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:58:24,757 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:58:24,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:58:24,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:58:24,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:58:24,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:58:24,825 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:58:24,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:58:24,881 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['55774 memcached\n'], []) 2022-02-08 04:58:24,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 04:58:24,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 04:58:24,950 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['55774 memcached\n'], []) 2022-02-08 04:58:24,951 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:58:24,953 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:58:25,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:25,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:58:25,296 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:25,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:58:25,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:58:25,321 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:58:25,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:58:25,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:58:25,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:58:25,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:58:25,388 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:58:25,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:58:25,446 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['90689 memcached\n'], []) 2022-02-08 04:58:25,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 04:58:25,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 04:58:25,496 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['90689 memcached\n'], []) 2022-02-08 04:58:25,496 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:58:25,500 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:58:25,584 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:25,798 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:58:25,865 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:25,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:58:25,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:58:25,890 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:58:25,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:58:25,918 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:58:25,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:58:25,943 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:58:25,954 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:58:25,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:58:26,017 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['24545 memcached\n'], []) 2022-02-08 04:58:26,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 04:58:26,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 04:58:26,068 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['24545 memcached\n'], []) 2022-02-08 04:58:26,069 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:58:26,071 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:58:26,163 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:26,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:58:26,447 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:26,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:58:26,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:58:26,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:58:26,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:58:26,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:58:26,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:58:26,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:58:26,540 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:58:26,540 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:58:26,605 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['39449 memcached\n'], []) 2022-02-08 04:58:26,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 04:58:26,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 04:58:26,647 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['39449 memcached\n'], []) 2022-02-08 04:58:26,647 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:58:26,648 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:58:26,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:26,948 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:58:27,005 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:27,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:58:27,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:58:27,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:58:27,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:58:27,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:58:27,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:58:27,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:58:27,108 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:58:27,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:58:27,155 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['75471 memcached\n'], []) 2022-02-08 04:58:27,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 04:58:27,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 04:58:27,200 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['75471 memcached\n'], []) 2022-02-08 04:58:27,201 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:58:27,206 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:58:27,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:27,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:58:27,582 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:27,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:58:27,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:58:27,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:58:27,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:58:27,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:58:27,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:58:27,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:58:27,680 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 04:58:27,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:58:27,724 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['105134 memcached\n'], []) 2022-02-08 04:58:27,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 04:58:27,752 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 04:58:27,773 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['105134 memcached\n'], []) 2022-02-08 04:58:27,773 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:58:27,776 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 04:58:27,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:28,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 04:58:28,186 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:28,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 04:58:28,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 04:58:28,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 04:58:28,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 04:58:28,249 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 04:58:28,265 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 04:58:28,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 04:58:28,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:58:28,309 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 04:58:28,336 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:58:28,352 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:31,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:58:32,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:32,392 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:32,622 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:58:32,637 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 04:58:32,637 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:58:32,650 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:32,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:58:32,716 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 04:58:32,716 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 04:58:32,719 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 04:58:33,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:33,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 04:58:33,578 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:33,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 04:58:33,594 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 04:58:33,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 04:58:33,622 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 04:58:33,640 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 04:58:33,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 04:58:33,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 04:58:33,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:58:33,698 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 04:58:33,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:58:33,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:33,813 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:58:33,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:33,920 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:33,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:58:33,973 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 04:58:33,974 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:58:33,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:34,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:58:34,023 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 04:58:34,023 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 04:58:34,025 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:58:34,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:34,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:58:34,371 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:34,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:58:34,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:58:34,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:58:34,408 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:58:34,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:58:34,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:58:34,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:58:34,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:58:34,489 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 04:58:34,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:58:34,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:34,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:58:35,367 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:35,640 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:35,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:58:35,884 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 04:58:35,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:58:35,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:35,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:58:35,967 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 04:58:35,967 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:58:35,970 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 04:58:36,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:36,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 04:58:36,362 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:36,364 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 04:58:36,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 04:58:36,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 04:58:36,408 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 04:58:36,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 04:58:36,441 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 04:58:36,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 04:58:36,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:58:36,490 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 04:58:36,520 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:58:36,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:36,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:58:36,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:36,584 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:36,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:58:36,612 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 04:58:36,612 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:58:36,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:36,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:58:36,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:36,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:58:36,691 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:36,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:36,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:36,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:36,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:36,788 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:36,806 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:36,826 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:36,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:36,862 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:58:36,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:36,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:36,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:36,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:36,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:36,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:36,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:37,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:37,020 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:37,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:58:37,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:37,073 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:37,089 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:37,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:37,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:37,135 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:37,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:37,165 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:37,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:37,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:58:37,214 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:37,229 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:37,242 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:37,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:37,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:37,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:37,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:37,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:37,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:37,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:58:37,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:37,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:37,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:37,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:37,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:37,494 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:37,512 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:37,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:37,545 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 04:58:37,546 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:58:37,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:37,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:58:37,931 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:37,931 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:58:37,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:58:37,969 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:58:37,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:58:38,005 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:58:38,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:58:38,035 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:58:38,051 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:58:38,068 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 04:58:38,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:58:38,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:39,897 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:58:40,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:40,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:41,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:58:41,039 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 04:58:41,040 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:58:41,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:41,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:58:41,138 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 04:58:41,138 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:58:41,140 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 04:58:41,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:41,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 04:58:41,549 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:41,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 04:58:41,563 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 04:58:41,576 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 04:58:41,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 04:58:41,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 04:58:41,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 04:58:41,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 04:58:41,642 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:58:41,657 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 04:58:41,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:58:41,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:41,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:58:41,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:41,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:41,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:58:41,802 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 04:58:41,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:58:41,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:41,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:58:41,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:41,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:58:41,882 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:41,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:41,918 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:41,936 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:41,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:41,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:41,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:42,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:42,030 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:42,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:58:42,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:42,076 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:42,094 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:42,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:42,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:42,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:42,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:42,186 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:42,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:42,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:58:42,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:42,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:42,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:42,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:42,308 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:42,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:42,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:42,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:42,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:42,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:58:42,408 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:42,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:42,441 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:42,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:42,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:42,499 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:42,517 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:42,533 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:42,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:42,568 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:58:42,586 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:42,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:42,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:42,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:42,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:42,678 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:42,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:42,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:42,736 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 04:58:42,737 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:58:42,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:43,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:58:43,062 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:43,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:58:43,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:58:43,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:58:43,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:58:43,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:58:43,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:58:43,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:58:43,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:58:43,171 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 04:58:43,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:58:43,214 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:45,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:58:45,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:46,124 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:46,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:58:46,390 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 04:58:46,390 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:58:46,405 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:46,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:58:46,463 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 04:58:46,463 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:58:46,464 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:58:46,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:46,783 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:58:46,868 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:46,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:58:46,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:58:46,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:58:46,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:58:46,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:58:46,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:58:46,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:58:46,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:58:47,012 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 04:58:47,038 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:58:47,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:47,982 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:58:48,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:48,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:48,897 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:58:48,917 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 04:58:48,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:58:48,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:48,989 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:58:49,005 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 04:58:49,006 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:58:49,006 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 04:58:49,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 6898 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:58:49,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 04:58:49,430 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:49,431 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 04:58:49,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 04:58:49,464 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 04:58:49,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 04:58:49,505 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 04:58:49,522 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 04:58:49,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 04:58:49,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:58:49,575 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 04:58:49,599 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:58:49,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:49,640 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:58:49,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:49,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:49,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:58:49,716 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 04:58:49,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:58:49,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:49,757 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:58:49,778 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:49,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:58:49,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:49,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:49,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:49,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:49,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:49,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:49,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:50,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:50,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:50,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:58:50,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:50,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:50,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:50,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:50,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:50,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:50,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:50,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:50,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:50,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:58:50,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:50,346 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:50,367 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:50,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:50,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:50,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:50,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:50,474 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:50,494 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:50,517 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:58:50,540 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:50,563 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:50,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:50,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:50,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:50,650 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:50,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:50,693 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:50,711 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:50,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:58:50,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:50,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:50,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:50,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:50,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:50,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:50,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:50,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:50,941 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 04:58:50,944 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 04:58:51,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:51,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 04:58:51,349 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:51,349 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 04:58:51,364 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 04:58:51,377 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 04:58:51,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 04:58:51,408 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 04:58:51,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 04:58:51,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 04:58:51,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:58:51,467 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 04:58:51,492 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:58:51,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:51,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:58:51,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:51,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:51,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:58:51,578 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 04:58:51,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:58:51,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:51,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:58:51,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:51,645 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:58:51,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:51,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:51,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:51,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:51,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:51,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:51,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:51,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:58:51,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:51,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:58:51,854 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:51,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:51,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:51,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:51,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:51,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:51,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:51,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:58:52,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:52,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:58:52,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:52,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:52,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:52,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:52,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:52,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:52,132 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:52,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:58:52,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:52,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:58:52,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:52,216 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:52,232 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:52,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:52,262 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:52,278 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:52,295 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:52,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:58:52,329 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:52,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:58:52,367 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:52,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:52,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:52,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:52,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:52,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:52,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:52,487 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:58:52,505 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 04:58:52,506 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:58:52,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:53,012 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:58:53,151 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:53,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:58:53,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:58:53,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:58:53,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:58:53,357 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:58:53,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:58:53,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:58:53,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:58:53,411 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 04:58:53,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:58:53,448 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:54,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:58:55,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:55,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:55,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:58:55,868 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 04:58:55,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:58:55,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:55,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:58:55,943 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 04:58:55,944 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:58:55,946 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:58:56,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:56,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:58:56,348 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:56,349 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:58:56,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:58:56,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:58:56,390 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:58:56,405 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:58:56,420 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:58:56,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:58:56,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:58:56,466 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 04:58:56,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:58:56,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:57,844 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:58:58,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:58,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:58:58,540 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:58:58,558 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 04:58:58,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:58:58,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:58:58,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:58:58,621 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 04:58:58,621 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:58:58,624 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:58:59,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:58:59,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:58:59,779 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:58:59,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:58:59,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:58:59,811 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:58:59,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:58:59,877 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:58:59,946 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:58:59,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:58:59,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:59:00,032 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 04:59:00,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:59:00,099 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:59:00,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:59:00,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:59:01,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:59:01,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:59:01,301 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 04:59:01,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:59:01,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:59:01,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:59:01,390 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 04:59:01,391 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:59:01,392 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:59:01,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:59:01,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:59:01,770 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:59:01,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:59:01,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:59:01,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:59:01,816 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:59:01,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:59:01,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:59:01,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:59:01,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:59:01,908 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 04:59:01,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:59:01,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:59:03,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:59:04,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:59:04,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:59:05,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:59:05,200 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 04:59:05,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:59:05,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:59:05,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:59:05,328 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 04:59:05,328 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:59:05,332 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 04:59:05,415 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:59:05,658 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 04:59:05,723 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:59:05,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 04:59:05,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 04:59:05,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 04:59:05,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 04:59:05,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 04:59:05,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 04:59:05,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 04:59:05,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 04:59:05,861 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 04:59:05,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 04:59:05,905 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:59:05,923 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 04:59:05,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:59:05,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 04:59:05,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 04:59:05,990 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 04:59:05,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 04:59:06,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 04:59:06,022 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 04:59:06,038 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:59:06,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 04:59:06,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:59:06,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:59:06,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:59:06,140 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:59:06,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:59:06,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:59:06,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:59:06,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 04:59:06,244 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:59:06,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 04:59:06,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:59:06,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:59:06,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:59:06,336 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:59:06,352 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:59:06,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:59:06,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:59:06,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 04:59:06,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:59:06,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 04:59:06,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:59:06,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:59:06,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:59:06,515 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:59:06,533 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:59:06,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:59:06,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:59:06,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 04:59:06,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:59:06,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 04:59:06,645 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:59:06,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:59:06,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:59:06,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:59:06,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:59:06,734 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:59:06,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:59:06,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 04:59:06,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:59:06,811 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 04:59:06,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:59:06,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:59:06,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:59:06,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:59:06,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:59:06,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:59:06,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:59:06,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 04:59:06,993 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 04:59:06,993 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:59:06,994 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 04:59:07,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:59:07,318 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 04:59:07,427 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:59:07,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 04:59:07,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 04:59:07,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 04:59:07,477 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 04:59:07,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 04:59:07,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 04:59:07,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 04:59:07,540 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:08,039 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 04:59:08,040 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.04799985886. 2022-02-08 04:59:08,040 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:59:08,042 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 04:59:08,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:59:08,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 04:59:08,753 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:59:08,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 04:59:08,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 04:59:08,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 04:59:08,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 04:59:08,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 04:59:08,885 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 04:59:08,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 04:59:08,926 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:09,233 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:59:09,233 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:59:11,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:11,453 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:59:11,453 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:59:13,454 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:13,663 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:59:13,663 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:59:15,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:15,884 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:59:15,884 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:59:17,890 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:18,059 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:59:18,059 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:59:20,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:20,240 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:59:20,240 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:59:22,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:22,621 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:59:22,621 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:59:24,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:24,825 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:59:24,825 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:59:26,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:27,105 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:59:27,105 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:59:29,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:29,387 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:59:29,387 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:59:31,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:31,588 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:59:31,588 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:59:33,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:33,890 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:59:33,891 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:59:35,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:36,198 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:59:36,200 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:59:38,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:38,414 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:59:38,414 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:59:40,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:40,709 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:59:40,709 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:59:42,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:42,934 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:59:42,934 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:59:44,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:45,119 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 04:59:45,119 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 04:59:47,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:47,299 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 04:59:47,299 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 39.2590000629. 2022-02-08 04:59:47,301 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:59:47,302 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 04:59:48,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:59:48,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 04:59:49,032 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:59:49,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 04:59:49,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 04:59:49,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 04:59:49,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 04:59:49,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 04:59:49,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 04:59:49,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 04:59:49,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 6902 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 04:59:49,953 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 04:59:49,953 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.65299987793. 2022-02-08 04:59:49,953 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:59:49,957 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 04:59:50,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:59:50,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 04:59:51,042 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:59:51,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 04:59:51,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 04:59:51,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 04:59:51,107 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 04:59:51,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 04:59:51,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 04:59:51,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 04:59:51,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:51,874 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 04:59:51,875 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.92100000381. 2022-02-08 04:59:51,875 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:59:51,877 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 04:59:52,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:59:53,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 04:59:53,585 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:59:53,586 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 04:59:53,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 04:59:53,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 04:59:53,658 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 04:59:53,678 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 04:59:53,696 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 04:59:53,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 04:59:53,765 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:54,446 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 04:59:54,446 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.57100009918. 2022-02-08 04:59:54,447 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:59:54,447 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 04:59:55,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:59:55,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 04:59:55,578 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:59:55,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 04:59:55,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 04:59:55,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 04:59:55,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 04:59:55,665 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 04:59:55,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 04:59:55,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 04:59:55,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:56,256 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 04:59:56,256 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.80999994278. 2022-02-08 04:59:56,256 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:59:56,259 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 04:59:56,890 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:59:57,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 04:59:57,232 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:59:57,232 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 04:59:57,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 04:59:57,260 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 04:59:57,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 04:59:57,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 04:59:57,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 04:59:57,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 04:59:57,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:57,785 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 04:59:57,786 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.52900004387. 2022-02-08 04:59:57,786 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:59:57,788 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 04:59:58,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 04:59:58,647 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 04:59:58,739 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 04:59:58,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 04:59:58,757 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 04:59:58,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 04:59:58,821 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 04:59:58,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 04:59:58,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 04:59:58,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 04:59:58,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 04:59:59,443 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 04:59:59,443 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.65700006485. 2022-02-08 04:59:59,444 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 04:59:59,444 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:00:00,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:00:01,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:00:01,647 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:00:01,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:00:01,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:00:01,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:00:01,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:00:01,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:00:01,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:00:01,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:00:01,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:00:02,436 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:00:02,436 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.99300003052. 2022-02-08 05:00:02,436 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:00:02,437 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:00:03,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:00:03,744 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:00:03,836 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:00:03,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:00:03,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:00:03,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:00:03,943 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:00:03,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:00:03,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:00:04,002 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:00:04,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:00:04,723 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:00:04,723 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.28799986839. ############################################################ # # RAM FootPrint: 4372.8828125 # ############################################################ 2022-02-08 05:00:04,763 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 05:00:04,763 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 116 seconds. Reason: Iteration:9 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 6924 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 6947 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:02:00,773 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 05:02:00,773 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 05:02:05,776 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:02:05,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:06,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:02:06,197 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:06,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:02:06,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:02:06,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:02:06,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:02:06,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:02:06,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:02:06,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:02:06,313 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:02:06,313 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:02:06,368 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['128686 memcached\n'], []) 2022-02-08 05:02:06,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 05:02:06,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:02:06,421 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['128686 memcached\n'], []) 2022-02-08 05:02:06,421 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:02:06,424 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:02:06,813 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:07,073 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:02:07,194 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:07,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:02:07,209 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:02:07,229 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:02:07,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:02:07,262 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:02:07,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:02:07,308 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:02:07,323 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:02:07,323 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:02:07,388 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['41308 memcached\n'], []) 2022-02-08 05:02:07,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 05:02:07,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:02:07,434 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['41308 memcached\n'], []) 2022-02-08 05:02:07,434 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:02:07,437 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:02:07,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:07,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:02:07,806 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:07,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:02:07,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:02:07,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:02:07,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:02:07,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:02:07,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:02:07,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:02:07,911 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:02:07,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:02:07,956 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['28673 memcached\n'], []) 2022-02-08 05:02:07,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 05:02:07,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:02:08,006 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['28673 memcached\n'], []) 2022-02-08 05:02:08,006 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:02:08,009 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:02:08,079 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:08,308 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:02:08,367 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:08,367 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:02:08,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:02:08,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:02:08,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:02:08,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:02:08,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:02:08,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:02:08,460 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:02:08,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:02:08,500 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['62634 memcached\n'], []) 2022-02-08 05:02:08,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 05:02:08,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:02:08,548 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['62634 memcached\n'], []) 2022-02-08 05:02:08,548 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:02:08,551 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:02:08,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:08,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:02:08,930 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:08,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:02:08,943 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:02:08,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:02:08,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:02:08,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:02:08,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:02:09,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:02:09,019 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:02:09,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:02:09,061 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['56494 memcached\n'], []) 2022-02-08 05:02:09,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 05:02:09,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:02:09,107 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['56494 memcached\n'], []) 2022-02-08 05:02:09,108 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:02:09,109 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:02:09,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:09,448 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:02:09,513 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:09,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:02:09,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:02:09,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:02:09,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:02:09,576 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:02:09,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:02:09,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:02:09,614 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:02:09,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:02:09,657 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['91408 memcached\n'], []) 2022-02-08 05:02:09,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 05:02:09,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:02:09,709 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['91408 memcached\n'], []) 2022-02-08 05:02:09,709 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:02:09,710 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:02:09,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:09,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:02:10,062 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:10,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:02:10,089 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:02:10,102 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:02:10,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:02:10,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:02:10,147 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:02:10,158 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:02:10,171 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:02:10,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:02:10,217 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['25233 memcached\n'], []) 2022-02-08 05:02:10,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 05:02:10,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:02:10,262 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['25233 memcached\n'], []) 2022-02-08 05:02:10,262 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:02:10,265 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:02:10,354 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:10,581 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:02:10,638 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:10,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:02:10,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:02:10,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:02:10,676 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:02:10,693 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:02:10,706 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:02:10,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:02:10,733 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:02:10,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:02:10,825 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['40168 memcached\n'], []) 2022-02-08 05:02:10,826 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 05:02:10,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:02:10,874 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['40168 memcached\n'], []) 2022-02-08 05:02:10,874 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:02:10,875 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:02:10,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:11,191 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:02:11,250 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:11,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:02:11,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:02:11,275 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:02:11,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:02:11,303 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:02:11,323 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:02:11,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:02:11,368 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:02:11,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:02:11,427 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['76197 memcached\n'], []) 2022-02-08 05:02:11,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 05:02:11,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:02:11,474 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['76197 memcached\n'], []) 2022-02-08 05:02:11,474 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:02:11,477 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:02:11,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:11,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:02:11,848 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:11,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:02:11,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:02:11,877 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:02:11,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:02:11,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:02:11,926 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:02:11,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:02:11,953 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:02:11,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:02:11,999 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['105853 memcached\n'], []) 2022-02-08 05:02:12,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 05:02:12,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:02:12,046 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['105853 memcached\n'], []) 2022-02-08 05:02:12,046 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:02:12,049 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:02:12,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:12,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:02:12,418 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:12,420 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:02:12,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:02:12,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:02:12,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:02:12,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:02:12,492 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:02:12,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:02:12,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:02:12,533 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 05:02:12,558 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:02:12,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:12,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:02:13,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:13,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:13,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:02:13,868 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 05:02:13,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:02:13,882 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:13,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:02:13,926 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 05:02:13,927 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:02:13,928 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:02:14,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:14,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:02:14,296 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:14,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:02:14,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:02:14,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:02:14,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:02:14,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:02:14,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:02:14,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:02:14,395 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:02:14,410 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 05:02:14,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:02:14,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:15,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:02:15,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:15,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:15,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:02:15,236 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 05:02:15,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:02:15,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:15,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:02:15,285 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 05:02:15,285 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:02:15,286 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:02:15,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:15,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:02:15,638 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:15,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:02:15,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:02:15,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:02:15,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:02:15,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:02:15,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:02:15,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:02:15,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:02:15,750 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 05:02:15,785 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:02:15,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:17,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:02:18,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:18,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:18,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:02:18,565 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 05:02:18,566 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:02:18,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:18,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:02:18,648 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 05:02:18,648 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:02:18,651 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 05:02:18,739 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:18,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 05:02:19,043 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:19,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 05:02:19,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 05:02:19,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 05:02:19,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 05:02:19,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 05:02:19,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 05:02:19,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 05:02:19,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:02:19,148 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 05:02:19,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:02:19,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:19,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:02:19,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:19,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:19,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:02:19,272 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 05:02:19,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:02:19,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:19,308 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:02:19,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:19,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:02:19,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:19,377 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:19,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:19,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:19,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:19,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:19,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:19,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:19,497 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:19,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:02:19,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:19,548 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:19,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:19,581 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:19,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:19,612 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:19,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:19,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:19,658 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:19,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:02:19,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:19,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:19,721 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:19,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:19,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:19,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:19,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:19,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:19,811 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:19,826 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:02:19,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:19,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:19,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:19,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:19,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:19,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:19,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:19,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:19,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:19,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:02:20,005 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:20,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:20,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:20,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:20,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:20,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:20,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:20,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:20,148 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 05:02:20,151 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:02:20,242 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:20,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:02:20,516 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:20,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:02:20,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:02:20,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:02:20,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:02:20,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:02:20,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:02:20,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:02:20,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:02:20,628 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 05:02:20,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:02:20,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:20,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:02:21,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:21,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:21,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:02:21,592 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 05:02:21,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:02:21,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:21,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:02:21,736 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 05:02:21,736 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:02:21,737 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 05:02:21,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:22,066 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 05:02:22,130 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:22,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 05:02:22,147 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 05:02:22,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 05:02:22,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 05:02:22,196 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 05:02:22,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 05:02:22,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 05:02:22,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:02:22,262 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 05:02:22,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:02:22,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:22,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:02:22,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:22,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:22,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:02:22,377 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 05:02:22,377 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:02:22,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:22,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:02:22,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:22,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:02:22,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:22,477 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:22,497 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:22,517 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:22,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:22,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:22,591 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:22,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:22,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:22,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:02:22,665 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:22,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:22,706 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:22,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:22,739 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:22,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:22,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:22,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:22,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:22,826 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:02:22,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:22,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:22,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:22,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:22,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:22,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:22,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:22,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:22,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:22,999 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:02:23,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:23,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:23,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:23,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:23,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:23,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:23,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:23,147 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:23,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:23,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:02:23,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:23,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:23,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:23,260 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:23,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:23,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:23,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:23,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:23,352 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 05:02:23,355 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:02:23,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:23,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:02:23,690 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:23,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:02:23,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:02:23,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:02:23,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:02:23,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:02:23,757 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:02:23,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:02:23,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:02:23,796 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 05:02:23,826 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:02:23,839 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:25,989 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:02:26,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:26,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:26,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:02:27,003 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 05:02:27,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:02:27,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:27,038 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:02:27,055 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 05:02:27,055 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:02:27,056 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:02:27,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:27,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:02:27,434 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:27,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:02:27,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:02:27,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:02:27,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:02:27,497 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:02:27,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:02:27,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:02:27,545 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:02:27,563 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 05:02:27,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:02:27,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:30,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:02:31,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:31,594 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:31,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:02:31,838 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 05:02:31,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:02:31,854 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:31,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:02:31,924 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 05:02:31,926 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:02:31,927 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 05:02:32,022 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:32,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 05:02:32,292 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:32,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 05:02:32,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 05:02:32,318 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 05:02:32,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 05:02:32,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 05:02:32,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 05:02:32,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 05:02:32,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:02:32,403 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 05:02:32,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:02:32,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:32,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:02:32,474 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:32,487 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:32,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:02:32,513 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 05:02:32,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:02:32,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:32,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:02:32,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:32,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:02:32,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:32,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:32,624 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:32,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:32,658 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:32,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:32,696 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:32,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:32,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:32,747 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:02:32,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:32,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:32,798 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:32,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:32,834 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:32,854 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:32,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:32,890 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:32,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:32,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:02:32,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:32,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:32,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:32,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:33,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:33,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:33,048 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:33,066 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:33,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:33,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:02:33,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:33,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:33,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:33,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:33,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:33,196 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:33,211 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:33,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:33,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:33,262 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:02:33,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:33,295 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:33,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:33,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:33,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:33,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:33,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:33,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:33,420 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 05:02:33,421 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 05:02:33,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:33,734 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 05:02:33,801 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:33,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 05:02:33,816 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 05:02:33,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 05:02:33,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 05:02:33,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 05:02:33,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 05:02:33,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 05:02:33,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:02:33,920 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 05:02:33,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:02:33,961 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:33,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:02:33,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:34,005 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:34,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:02:34,035 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 05:02:34,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:02:34,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:34,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:02:34,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:34,102 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:02:34,124 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:34,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:34,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:34,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:34,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:34,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:34,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:34,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:34,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:34,341 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:02:34,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:34,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:34,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:34,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:34,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:34,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:34,454 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:34,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:34,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:34,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:02:34,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:34,535 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:34,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:34,579 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:34,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:34,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:34,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:34,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:34,655 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:34,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:02:34,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:34,698 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:34,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:34,727 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:34,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:34,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:34,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:34,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:34,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:34,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:02:34,854 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:34,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:34,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:34,905 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:34,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:34,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:34,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:34,971 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:34,987 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 05:02:34,990 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:02:35,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:35,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:02:35,486 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:35,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:02:35,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:02:35,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:02:35,548 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:02:35,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:02:35,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:02:35,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:02:35,619 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:02:35,641 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 05:02:35,665 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:02:35,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:38,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:02:38,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:39,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:39,584 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:02:39,647 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 05:02:39,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:02:39,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:39,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:02:39,828 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 05:02:39,829 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:02:39,832 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:02:40,186 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:40,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:02:41,424 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:41,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:02:41,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:02:41,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:02:41,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:02:41,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:02:41,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:02:42,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:02:42,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:02:42,200 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 05:02:42,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:02:42,270 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:42,803 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:02:43,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:43,676 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:44,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:02:44,085 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 05:02:44,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:02:44,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:44,232 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:02:44,256 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 05:02:44,257 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:02:44,260 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:02:44,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:45,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:02:45,440 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:45,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:02:45,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:02:45,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:02:45,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:02:45,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:02:45,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:02:45,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:02:45,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:02:45,782 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 05:02:45,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:02:45,844 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:46,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:02:46,961 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:47,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:47,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:02:47,398 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 05:02:47,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:02:47,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:47,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:02:47,522 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 05:02:47,522 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:02:47,523 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:02:47,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:47,867 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:02:47,954 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:47,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:02:47,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:02:47,999 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:02:48,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:02:48,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:02:48,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:02:48,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:02:48,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:02:48,140 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 05:02:48,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:02:48,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 6967 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:02:50,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:02:51,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:51,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:52,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:02:52,407 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 05:02:52,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:02:52,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:52,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:02:52,861 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 05:02:52,862 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:02:52,865 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 05:02:52,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:53,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 05:02:53,282 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:53,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 05:02:53,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 05:02:53,313 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 05:02:53,326 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 05:02:53,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 05:02:53,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 05:02:53,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 05:02:53,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:02:53,410 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 05:02:53,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:02:53,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:53,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:02:53,484 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:53,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:02:53,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:02:53,532 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 05:02:53,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:02:53,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:02:53,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:02:53,579 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:53,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:02:53,622 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:53,645 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:53,668 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:53,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:53,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:53,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:53,755 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:53,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:02:53,803 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:53,821 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:02:53,839 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:53,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:53,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:53,892 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:53,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:53,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:53,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:53,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:02:53,989 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:54,005 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:02:54,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:54,040 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:54,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:54,076 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:54,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:54,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:54,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:54,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:02:54,209 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:54,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:02:54,265 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:54,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:54,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:54,318 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:54,341 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:54,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:54,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:54,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:02:54,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:54,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:02:54,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:54,492 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:54,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:54,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:54,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:54,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:54,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:54,602 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:02:54,619 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 05:02:54,621 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:02:54,622 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:02:54,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:54,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:02:55,063 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:55,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:02:55,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:02:55,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:02:55,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:02:55,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:02:55,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:02:55,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:02:55,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:02:55,753 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:02:55,755 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.13500022888. 2022-02-08 05:02:55,755 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:02:55,756 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:02:55,859 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:02:56,130 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:02:56,217 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:02:56,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:02:56,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:02:56,252 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:02:56,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:02:56,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:02:56,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:02:56,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:02:56,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:02:56,529 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:02:56,529 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:02:58,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:02:58,805 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:02:58,805 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:03:00,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:01,006 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:03:01,007 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:03:03,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:03,197 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:03:03,198 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:03:05,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:05,522 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:03:05,522 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:03:07,522 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:07,811 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:03:07,811 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:03:09,811 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:10,384 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:03:10,384 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:03:12,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:12,803 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:03:12,803 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:03:14,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:15,003 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:03:15,003 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:03:17,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:17,375 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:03:17,377 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:03:19,377 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:19,645 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:03:19,645 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:03:21,647 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:21,933 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:03:21,933 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:03:23,933 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:24,276 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:03:24,276 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:03:26,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:26,671 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:03:26,671 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:03:28,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:28,936 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:03:28,936 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:03:30,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:31,154 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:03:31,154 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:03:33,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:33,372 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:03:33,372 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:03:35,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:35,556 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:03:35,558 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 39.8029999733. 2022-02-08 05:03:35,558 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:03:35,559 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:03:36,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:03:37,089 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:03:37,276 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:03:37,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:03:37,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:03:37,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:03:37,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:03:37,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:03:37,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:03:37,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:03:37,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:38,010 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:03:38,010 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.45299983025. 2022-02-08 05:03:38,010 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:03:38,013 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:03:38,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:03:39,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:03:39,648 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:03:39,650 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:03:39,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:03:39,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:03:39,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:03:39,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:03:39,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:03:39,844 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:03:39,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:41,079 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:03:41,082 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.07100009918. 2022-02-08 05:03:41,082 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:03:41,084 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:03:41,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:03:42,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:03:42,441 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:03:42,441 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:03:42,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:03:42,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:03:42,515 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:03:42,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:03:42,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:03:42,599 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:03:42,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:43,188 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:03:43,190 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.10800004005. 2022-02-08 05:03:43,190 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:03:43,191 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:03:44,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:03:44,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:03:45,082 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:03:45,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:03:45,104 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:03:45,132 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:03:45,158 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:03:45,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:03:45,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:03:45,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:03:45,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:46,417 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:03:46,417 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.22699999809. 2022-02-08 05:03:46,417 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:03:46,420 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:03:47,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:03:47,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:03:47,447 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:03:47,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:03:47,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:03:47,484 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:03:47,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:03:47,517 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:03:47,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:03:47,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:03:47,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:48,134 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:03:48,134 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.71700000763. 2022-02-08 05:03:48,134 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:03:48,138 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:03:48,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:03:49,278 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 5 queries submitted, 0 failed, 6970 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:03:49,401 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:03:49,403 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:03:49,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:03:49,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:03:49,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:03:49,494 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:03:49,535 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:03:49,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:03:49,584 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:50,101 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:03:50,101 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.96700000763. 2022-02-08 05:03:50,102 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:03:50,104 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:03:50,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:03:51,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:03:51,115 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:03:51,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:03:51,130 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:03:51,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:03:51,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:03:51,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:03:51,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:03:51,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:03:51,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:51,710 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:03:51,710 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.60800004005. 2022-02-08 05:03:51,711 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:03:51,713 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:03:52,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:03:52,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:03:52,717 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:03:52,719 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:03:52,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:03:52,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:03:52,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:03:52,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:03:52,999 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:03:53,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:03:53,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:03:53,773 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:03:53,775 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.0640001297. ############################################################ # # RAM FootPrint: 4381.00390625 # ############################################################ 2022-02-08 05:03:53,819 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 05:03:53,819 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 96 seconds. Reason: Iteration:10 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 6992 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:05:29,825 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 05:05:29,825 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 05:05:34,828 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:05:35,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:05:35,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:05:35,588 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:05:35,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:05:35,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:05:35,642 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:05:35,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:05:35,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:05:35,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:05:35,747 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:05:35,763 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:05:35,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:05:35,825 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['129410 memcached\n'], []) 2022-02-08 05:05:35,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 05:05:35,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:05:35,878 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['129410 memcached\n'], []) 2022-02-08 05:05:35,878 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:05:35,881 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:05:36,436 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:05:36,803 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:05:36,914 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:05:36,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:05:36,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:05:36,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:05:36,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:05:37,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:05:37,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:05:37,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:05:37,068 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:05:37,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:05:37,118 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['42240 memcached\n'], []) 2022-02-08 05:05:37,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 05:05:37,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:05:37,164 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['42240 memcached\n'], []) 2022-02-08 05:05:37,164 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:05:37,167 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:05:37,242 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:05:37,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:05:37,519 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:05:37,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:05:37,535 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:05:37,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:05:37,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:05:37,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:05:37,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:05:37,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:05:37,631 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:05:37,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:05:37,674 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['29468 memcached\n'], []) 2022-02-08 05:05:37,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 05:05:37,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:05:37,746 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['29468 memcached\n'], []) 2022-02-08 05:05:37,746 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:05:37,749 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:05:37,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:05:38,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:05:38,121 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:05:38,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:05:38,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:05:38,147 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:05:38,160 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:05:38,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:05:38,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:05:38,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:05:38,220 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:05:38,221 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:05:38,270 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['63359 memcached\n'], []) 2022-02-08 05:05:38,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 05:05:38,295 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:05:38,315 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['63359 memcached\n'], []) 2022-02-08 05:05:38,315 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:05:38,318 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:05:38,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:05:38,604 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:05:38,658 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:05:38,658 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:05:38,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:05:38,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:05:38,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:05:38,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:05:38,721 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:05:38,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:05:38,750 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:05:38,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:05:38,801 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['57189 memcached\n'], []) 2022-02-08 05:05:38,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 05:05:38,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:05:38,845 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['57189 memcached\n'], []) 2022-02-08 05:05:38,845 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:05:38,848 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:05:38,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:05:39,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:05:39,201 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:05:39,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:05:39,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:05:39,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:05:39,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:05:39,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:05:39,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:05:39,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:05:39,306 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:05:39,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:05:39,355 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['92133 memcached\n'], []) 2022-02-08 05:05:39,357 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 05:05:39,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:05:39,407 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['92133 memcached\n'], []) 2022-02-08 05:05:39,407 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:05:39,408 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:05:39,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:05:39,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:05:39,801 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:05:39,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:05:39,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:05:39,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:05:39,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:05:39,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:05:39,867 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:05:39,880 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:05:39,891 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:05:39,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:05:39,940 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['25966 memcached\n'], []) 2022-02-08 05:05:39,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 05:05:39,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:05:39,984 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['25966 memcached\n'], []) 2022-02-08 05:05:39,984 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:05:39,986 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:05:40,071 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:05:40,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:05:40,365 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:05:40,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:05:40,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:05:40,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:05:40,405 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:05:40,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:05:40,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:05:40,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:05:40,461 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:05:40,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:05:40,525 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['40849 memcached\n'], []) 2022-02-08 05:05:40,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 05:05:40,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:05:40,572 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['40849 memcached\n'], []) 2022-02-08 05:05:40,572 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:05:40,576 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:05:40,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:05:40,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:05:40,947 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:05:40,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:05:40,961 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:05:40,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:05:40,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:05:41,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:05:41,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:05:41,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:05:41,061 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:05:41,061 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:05:41,108 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['76893 memcached\n'], []) 2022-02-08 05:05:41,109 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 05:05:41,132 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:05:41,155 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['76893 memcached\n'], []) 2022-02-08 05:05:41,155 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:05:41,157 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:05:41,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:05:41,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:05:41,530 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:05:41,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:05:41,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:05:41,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:05:41,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:05:41,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:05:41,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:05:41,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:05:41,634 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:05:41,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:05:41,696 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['106585 memcached\n'], []) 2022-02-08 05:05:41,696 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 05:05:41,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:05:41,740 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['106585 memcached\n'], []) 2022-02-08 05:05:41,740 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:05:41,743 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:05:41,846 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:05:42,079 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:05:42,142 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:05:42,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:05:42,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:05:42,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:05:42,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:05:42,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:05:42,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:05:42,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:05:42,244 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:05:42,260 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 05:05:42,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:05:42,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:05:42,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:05:43,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:05:43,377 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:05:43,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:05:43,615 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 05:05:43,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:05:43,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:05:43,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:05:43,664 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 05:05:43,664 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:05:43,665 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:05:43,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:05:43,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:05:44,049 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:05:44,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:05:44,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:05:44,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:05:44,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:05:44,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:05:44,140 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:05:44,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:05:44,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:05:44,187 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 05:05:44,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:05:44,229 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:05:44,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:05:44,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:05:44,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:05:44,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:05:44,545 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 05:05:44,545 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:05:44,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:05:44,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:05:44,605 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 05:05:44,607 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:05:44,609 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:05:44,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:05:44,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:05:44,953 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:05:44,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:05:44,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:05:44,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:05:44,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:05:45,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:05:45,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:05:45,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:05:45,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:05:45,062 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 05:05:45,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:05:45,107 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:05:48,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:05:48,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:05:48,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:05:49,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:05:49,029 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 05:05:49,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:05:49,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:05:49,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:05:49,109 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 05:05:49,109 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:05:49,111 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 05:05:49,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 7012 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:05:49,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 05:05:49,482 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:05:49,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 05:05:49,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 05:05:49,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 05:05:49,522 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 05:05:49,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 05:05:49,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 05:05:49,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 05:05:49,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:05:49,591 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 05:05:49,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:05:49,630 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:05:49,645 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:05:49,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:05:49,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:05:49,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:05:49,703 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 05:05:49,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:05:49,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:05:49,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:05:49,744 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:49,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:05:49,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:49,803 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:49,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:49,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:49,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:49,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:49,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:49,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:49,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:49,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:05:49,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:49,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:49,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:50,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:50,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:50,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:50,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:50,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:50,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:50,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:05:50,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:50,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:50,147 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:50,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:50,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:50,191 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:50,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:50,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:50,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:50,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:05:50,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:50,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:50,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:50,313 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:50,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:50,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:50,367 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:50,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:50,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:50,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:05:50,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:50,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:50,474 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:50,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:50,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:50,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:50,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:50,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:50,582 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 05:05:50,584 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:05:50,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:05:50,905 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:05:50,992 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:05:50,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:05:51,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:05:51,020 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:05:51,035 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:05:51,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:05:51,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:05:51,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:05:51,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:05:51,111 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 05:05:51,137 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:05:51,163 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:05:53,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:05:53,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:05:54,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:05:54,318 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:05:54,335 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 05:05:54,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:05:54,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:05:54,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:05:54,730 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 05:05:54,730 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:05:54,732 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 05:05:54,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:05:55,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 05:05:55,148 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:05:55,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 05:05:55,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 05:05:55,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 05:05:55,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 05:05:55,211 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 05:05:55,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 05:05:55,242 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 05:05:55,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:05:55,273 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 05:05:55,395 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:05:55,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:05:55,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:05:55,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:05:55,464 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:05:55,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:05:55,494 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 05:05:55,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:05:55,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:05:55,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:05:55,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:55,563 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:05:55,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:55,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:55,630 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:55,650 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:55,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:55,693 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:55,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:55,739 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:05:55,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:55,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:05:55,798 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:55,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:55,834 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:55,862 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:55,882 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:55,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:55,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:55,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:05:55,974 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:56,017 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:05:56,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:56,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:56,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:56,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:56,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:56,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:56,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:56,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:05:56,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:56,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:05:56,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:56,211 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:56,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:56,262 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:56,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:56,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:56,308 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:56,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:05:56,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:56,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:05:56,405 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:56,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:56,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:56,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:56,477 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:56,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:56,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:56,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:05:56,542 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 05:05:56,543 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:05:56,640 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:05:56,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:05:56,914 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:05:56,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:05:56,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:05:56,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:05:56,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:05:56,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:05:56,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:05:56,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:05:57,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:05:57,025 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 05:05:57,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:05:57,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:05:59,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:05:59,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:00,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:00,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:06:00,359 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 05:06:00,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:06:00,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:06:00,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:06:00,424 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 05:06:00,424 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:06:00,427 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:06:00,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:06:00,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:06:00,819 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:06:00,821 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:06:00,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:06:00,854 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:06:00,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:06:00,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:06:00,910 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:06:00,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:06:00,946 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:06:00,964 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 05:06:00,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:06:01,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:06:02,015 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:06:02,415 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:02,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:02,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:06:02,898 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 05:06:02,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:06:02,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:06:02,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:06:03,002 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 05:06:03,002 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:06:03,003 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 05:06:03,086 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:06:03,308 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 05:06:03,374 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:06:03,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 05:06:03,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 05:06:03,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 05:06:03,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 05:06:03,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 05:06:03,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 05:06:03,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 05:06:03,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:06:03,493 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 05:06:03,522 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:06:03,535 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:06:03,548 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:06:03,561 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:03,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:03,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:06:03,598 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 05:06:03,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:06:03,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:06:03,622 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:06:03,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:03,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:06:03,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:03,686 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:03,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:03,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:03,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:03,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:03,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:03,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:03,806 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:03,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:06:03,839 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:03,857 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:03,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:03,890 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:03,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:03,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:03,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:03,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:03,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:03,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:06:04,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:04,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:04,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:04,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:04,086 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:04,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:04,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:04,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:04,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:04,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:06:04,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:04,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:04,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:04,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:04,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:04,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:04,295 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:04,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:04,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:04,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:06:04,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:04,392 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:04,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:04,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:04,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:04,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:04,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:04,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:04,529 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 05:06:04,529 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 05:06:04,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:06:04,834 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 05:06:04,894 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:06:04,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 05:06:04,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 05:06:04,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 05:06:04,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 05:06:04,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 05:06:04,964 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 05:06:04,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 05:06:04,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:06:05,009 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 05:06:05,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:06:05,048 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:06:05,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:06:05,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:05,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:05,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:06:05,119 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 05:06:05,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:06:05,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:06:05,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:06:05,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:05,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:06:05,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:05,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:05,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:05,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:05,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:05,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:05,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:05,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:05,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:05,403 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:06:05,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:05,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:05,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:05,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:05,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:05,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:05,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:05,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:05,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:05,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:06:05,568 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:05,584 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:05,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:05,612 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:05,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:05,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:05,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:05,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:05,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:05,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:06:05,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:05,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:05,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:05,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:05,775 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:05,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:05,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:05,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:05,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:05,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:06:05,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:05,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:05,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:05,923 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:05,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:05,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:05,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:05,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:06,013 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 05:06:06,015 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:06:06,109 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:06:06,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:06:06,443 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:06:06,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:06:06,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:06:06,474 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:06:06,489 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:06:06,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:06:06,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:06:06,535 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:06:06,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:06:06,568 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 05:06:06,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:06:06,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:06:08,357 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:06:08,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:09,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:09,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:06:09,418 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 05:06:09,420 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:06:09,438 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:06:09,489 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:06:09,506 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 05:06:09,506 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:06:09,509 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:06:09,604 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:06:09,890 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:06:10,036 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:06:10,038 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:06:10,071 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:06:10,089 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:06:10,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:06:10,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:06:10,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:06:10,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:06:10,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:06:10,201 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 05:06:10,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:06:10,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:06:10,558 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:06:10,877 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:11,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:11,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:06:11,321 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 05:06:11,321 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:06:11,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:06:11,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:06:11,392 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 05:06:11,392 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:06:11,394 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:06:11,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:06:12,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:06:12,595 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:06:12,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:06:12,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:06:12,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:06:12,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:06:12,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:06:12,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:06:12,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:06:12,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:06:12,785 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 05:06:12,811 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:06:12,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:06:15,982 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:06:16,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:16,920 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:17,114 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:06:17,134 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 05:06:17,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:06:17,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:06:17,211 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:06:17,230 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 05:06:17,230 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:06:17,233 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:06:17,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:06:17,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:06:17,621 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:06:17,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:06:17,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:06:17,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:06:17,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:06:17,681 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:06:17,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:06:17,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:06:17,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:06:17,759 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 05:06:17,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:06:17,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:06:19,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:06:20,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:20,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:20,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:06:20,951 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 05:06:20,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:06:20,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:06:21,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:06:21,019 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 05:06:21,020 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:06:21,022 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 05:06:21,099 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:06:21,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 05:06:21,417 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:06:21,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 05:06:21,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 05:06:21,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 05:06:21,469 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 05:06:21,492 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 05:06:21,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 05:06:21,522 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 05:06:21,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:06:21,558 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 05:06:21,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:06:21,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:06:21,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:06:21,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:21,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:06:21,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:06:21,674 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 05:06:21,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:06:21,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:06:21,706 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:06:21,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:21,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:06:21,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:21,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:21,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:21,821 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:21,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:21,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:21,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:21,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:06:21,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:21,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:06:21,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:21,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:22,002 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:22,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:22,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:22,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:22,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:22,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:06:22,104 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:22,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:06:22,135 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:22,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:22,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:22,186 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:22,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:22,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:22,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:22,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:06:22,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:22,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:06:22,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:22,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:22,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:22,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:22,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:22,403 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:22,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:22,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:06:22,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:22,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:06:22,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:22,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:22,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:22,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:22,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:22,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:22,602 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:22,622 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:06:22,638 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 05:06:22,638 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:06:22,641 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:06:22,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:06:23,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:06:23,164 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:06:23,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:06:23,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:06:23,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:06:23,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:06:23,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:06:23,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:06:23,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:06:23,318 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:06:23,825 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:06:23,826 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.18799996376. 2022-02-08 05:06:23,826 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:06:23,828 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:06:23,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:06:24,147 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:06:24,210 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:06:24,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:06:24,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:06:24,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:06:24,252 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:06:24,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:06:24,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:06:24,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:06:24,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:06:24,457 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:06:24,459 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:06:26,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:06:26,618 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:06:26,618 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:06:28,619 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:06:28,759 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:06:28,759 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:06:30,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:06:30,904 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:06:30,904 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:06:32,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:06:33,055 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:06:33,056 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:06:35,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:06:35,323 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:06:35,323 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:06:37,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:06:37,640 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:06:37,641 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:06:39,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:06:39,871 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:06:39,871 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:06:41,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:06:42,115 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:06:42,115 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:06:44,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:06:44,392 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:06:44,392 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:06:46,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:06:46,624 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:06:46,624 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:06:48,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:06:49,046 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:06:49,046 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 7027 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:06:51,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:06:51,417 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:06:51,417 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:06:53,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:06:53,753 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:06:53,753 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:06:55,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:06:55,990 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:06:55,990 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:06:57,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:06:58,221 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:06:58,221 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:07:00,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:07:00,734 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:07:00,736 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:07:02,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:07:03,084 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:07:03,084 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:07:05,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:07:05,369 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:07:05,369 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:07:07,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:07:07,690 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:07:07,690 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:07:09,691 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:07:09,933 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:07:09,933 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:07:11,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:07:12,278 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:07:12,279 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:07:14,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:07:15,092 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:07:15,092 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 51.2660000324. 2022-02-08 05:07:15,092 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:07:15,095 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:07:15,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:07:16,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:07:16,539 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:07:16,540 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:07:16,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:07:16,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:07:16,691 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:07:16,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:07:16,767 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:07:16,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:07:16,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:07:18,099 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:07:18,099 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.00699996948. 2022-02-08 05:07:18,099 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:07:18,101 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:07:18,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:07:19,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:07:19,624 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:07:19,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:07:19,668 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:07:19,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:07:19,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:07:19,849 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:07:19,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:07:19,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:07:20,015 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:07:24,348 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:07:24,349 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.25. 2022-02-08 05:07:24,349 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:07:24,351 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:07:25,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:07:25,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:07:25,917 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:07:25,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:07:25,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:07:25,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:07:25,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:07:26,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:07:26,030 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:07:26,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:07:26,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:07:30,520 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:07:30,522 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.17200016975. 2022-02-08 05:07:30,522 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:07:30,523 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:07:31,303 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:07:31,719 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:07:31,845 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:07:31,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:07:31,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:07:31,905 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:07:31,926 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:07:31,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:07:31,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:07:32,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:07:32,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:07:32,602 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:07:32,602 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.08099985123. 2022-02-08 05:07:32,602 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:07:32,605 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:07:33,372 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:07:33,783 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:07:33,937 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:07:33,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:07:33,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:07:33,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:07:33,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:07:33,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:07:34,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:07:34,038 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:07:34,053 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:07:34,499 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:07:34,499 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.89600014687. 2022-02-08 05:07:34,499 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:07:34,500 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:07:35,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:07:35,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:07:36,154 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:07:36,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:07:36,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:07:36,196 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:07:36,214 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:07:36,242 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:07:36,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:07:36,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:07:36,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:07:39,138 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:07:39,138 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.63999986649. 2022-02-08 05:07:39,138 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:07:39,141 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:07:39,665 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:07:39,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:07:39,993 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:07:39,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:07:40,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:07:40,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:07:40,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:07:40,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:07:40,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:07:40,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:07:40,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:07:40,443 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:07:40,444 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.30500006676. 2022-02-08 05:07:40,444 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:07:40,446 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:07:40,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:07:41,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:07:41,378 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:07:41,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:07:41,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:07:41,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:07:41,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:07:41,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:07:41,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:07:41,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:07:41,540 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:07:42,118 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:07:42,118 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.67499995232. ############################################################ # # RAM FootPrint: 4372.5703125 # ############################################################ 2022-02-08 05:07:42,161 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 05:07:42,161 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 101 seconds. Reason: Iteration:11 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 7046 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 7071 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:09:23,165 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 05:09:23,165 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 05:09:28,168 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:09:28,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:28,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:09:28,832 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:28,834 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:09:28,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:09:28,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:09:28,897 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:09:28,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:09:28,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:09:28,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:09:28,990 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:09:28,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:09:29,072 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['130114 memcached\n'], []) 2022-02-08 05:09:29,073 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 05:09:29,102 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:09:29,131 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['130114 memcached\n'], []) 2022-02-08 05:09:29,131 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:09:29,134 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:09:29,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:29,910 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:09:29,992 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:29,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:09:30,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:09:30,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:09:30,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:09:30,073 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:09:30,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:09:30,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:09:30,125 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:09:30,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:09:30,180 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['43111 memcached\n'], []) 2022-02-08 05:09:30,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 05:09:30,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:09:30,223 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['43111 memcached\n'], []) 2022-02-08 05:09:30,223 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:09:30,226 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:09:30,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:30,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:09:30,605 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:30,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:09:30,630 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:09:30,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:09:30,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:09:30,681 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:09:30,696 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:09:30,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:09:30,730 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:09:30,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:09:30,779 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['30265 memcached\n'], []) 2022-02-08 05:09:30,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 05:09:30,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:09:30,831 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['30265 memcached\n'], []) 2022-02-08 05:09:30,831 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:09:30,832 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:09:30,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:31,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:09:31,187 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:31,188 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:09:31,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:09:31,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:09:31,229 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:09:31,244 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:09:31,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:09:31,270 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:09:31,285 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:09:31,285 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:09:31,328 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['64058 memcached\n'], []) 2022-02-08 05:09:31,329 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 05:09:31,354 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:09:31,380 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['64058 memcached\n'], []) 2022-02-08 05:09:31,380 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:09:31,382 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:09:31,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:31,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:09:31,727 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:31,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:09:31,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:09:31,755 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:09:31,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:09:31,783 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:09:31,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:09:31,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:09:31,821 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:09:31,821 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:09:31,865 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['57890 memcached\n'], []) 2022-02-08 05:09:31,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 05:09:31,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:09:31,907 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['57890 memcached\n'], []) 2022-02-08 05:09:31,907 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:09:31,908 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:09:31,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:32,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:09:32,260 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:32,262 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:09:32,275 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:09:32,295 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:09:32,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:09:32,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:09:32,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:09:32,364 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:09:32,378 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:09:32,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:09:32,421 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['92802 memcached\n'], []) 2022-02-08 05:09:32,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 05:09:32,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:09:32,464 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['92802 memcached\n'], []) 2022-02-08 05:09:32,466 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:09:32,469 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:09:32,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:32,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:09:32,858 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:32,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:09:32,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:09:32,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:09:32,897 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:09:32,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:09:32,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:09:32,936 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:09:32,947 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:09:32,948 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:09:33,009 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['26668 memcached\n'], []) 2022-02-08 05:09:33,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 05:09:33,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:09:33,055 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['26668 memcached\n'], []) 2022-02-08 05:09:33,055 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:09:33,058 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:09:33,137 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:33,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:09:33,447 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:33,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:09:33,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:09:33,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:09:33,484 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:09:33,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:09:33,512 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:09:33,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:09:33,538 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:09:33,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:09:33,607 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['41553 memcached\n'], []) 2022-02-08 05:09:33,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 05:09:33,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:09:33,661 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['41553 memcached\n'], []) 2022-02-08 05:09:33,661 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:09:33,671 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:09:33,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:33,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:09:34,049 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:34,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:09:34,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:09:34,076 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:09:34,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:09:34,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:09:34,135 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:09:34,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:09:34,167 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:09:34,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:09:34,213 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['77597 memcached\n'], []) 2022-02-08 05:09:34,214 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 05:09:34,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:09:34,262 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['77597 memcached\n'], []) 2022-02-08 05:09:34,262 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:09:34,263 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:09:34,341 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:34,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:09:34,642 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:34,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:09:34,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:09:34,670 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:09:34,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:09:34,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:09:34,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:09:34,724 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:09:34,736 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:09:34,737 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:09:34,779 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['107240 memcached\n'], []) 2022-02-08 05:09:34,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 05:09:34,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:09:34,822 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['107240 memcached\n'], []) 2022-02-08 05:09:34,822 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:09:34,825 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:09:34,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:35,145 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:09:35,210 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:35,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:09:35,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:09:35,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:09:35,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:09:35,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:09:35,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:09:35,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:09:35,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:09:35,331 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 05:09:35,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:09:35,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:38,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:09:39,104 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:39,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:39,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:09:39,634 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 05:09:39,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:09:39,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:39,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:09:39,724 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 05:09:39,724 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:09:39,727 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:09:39,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:40,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:09:40,092 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:40,094 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:09:40,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:09:40,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:09:40,137 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:09:40,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:09:40,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:09:40,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:09:40,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:09:40,213 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 05:09:40,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:09:40,275 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:40,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:09:40,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:40,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:40,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:09:40,594 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 05:09:40,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:09:40,612 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:40,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:09:40,654 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 05:09:40,654 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:09:40,657 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:09:40,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:40,948 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:09:41,015 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:41,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:09:41,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:09:41,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:09:41,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:09:41,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:09:41,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:09:41,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:09:41,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:09:41,122 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 05:09:41,147 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:09:41,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:41,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:09:42,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:42,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:42,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:09:42,792 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 05:09:42,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:09:42,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:42,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:09:42,884 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 05:09:42,884 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:09:42,885 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 05:09:42,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:43,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 05:09:43,250 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:43,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 05:09:43,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 05:09:43,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 05:09:43,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 05:09:43,308 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 05:09:43,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 05:09:43,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 05:09:43,349 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:09:43,365 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 05:09:43,403 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:09:43,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:43,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:09:43,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:43,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:43,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:09:43,506 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 05:09:43,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:09:43,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:43,535 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:09:43,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:43,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:09:43,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:43,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:43,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:43,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:43,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:43,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:43,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:43,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:43,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:43,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:09:43,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:43,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:43,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:43,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:43,880 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:43,897 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:43,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:43,931 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:43,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:43,964 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:09:43,982 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:43,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:44,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:44,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:44,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:44,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:44,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:44,091 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:44,104 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:44,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:09:44,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:44,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:44,168 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:44,186 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:44,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:44,221 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:44,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:44,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:44,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:44,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:09:44,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:44,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:44,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:44,354 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:44,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:44,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:44,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:44,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:44,440 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 05:09:44,443 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:09:44,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:44,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:09:44,944 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:44,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:09:44,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:09:45,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:09:45,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:09:45,071 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:09:45,109 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:09:45,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:09:45,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:09:45,233 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 05:09:45,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:09:45,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:46,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:09:46,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:46,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:47,066 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:09:47,082 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 05:09:47,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:09:47,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:47,137 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:09:47,151 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 05:09:47,151 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:09:47,153 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 05:09:47,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:47,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 05:09:47,513 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:47,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 05:09:47,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 05:09:47,540 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 05:09:47,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 05:09:47,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 05:09:47,594 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 05:09:47,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 05:09:47,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:09:47,641 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 05:09:47,665 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:09:47,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:47,698 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:09:47,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:47,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:47,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:09:47,757 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 05:09:47,757 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:09:47,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:47,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:09:47,803 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:47,826 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:09:47,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:47,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:47,900 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:47,918 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:47,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:47,961 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:47,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:48,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:48,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:48,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:09:48,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:48,086 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:48,104 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:48,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:48,137 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:48,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:48,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:48,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:48,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:48,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:09:48,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:48,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:48,280 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:48,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:48,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:48,334 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:48,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:48,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:48,385 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:48,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:09:48,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:48,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:48,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:48,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:48,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:48,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:48,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:48,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:48,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:48,594 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:09:48,612 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:48,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:48,650 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:48,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:48,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:48,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:48,721 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:48,739 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:48,756 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 05:09:48,757 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:09:48,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:49,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:09:49,101 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:49,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:09:49,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:09:49,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:09:49,140 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:09:49,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:09:49,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:09:49,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:09:49,191 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:09:49,206 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 05:09:49,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:09:49,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 7095 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:09:49,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:09:50,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:50,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:50,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:09:50,663 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 05:09:50,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:09:50,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:50,724 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:09:50,743 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 05:09:50,743 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:09:50,744 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:09:50,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:51,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:09:51,125 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:51,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:09:51,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:09:51,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:09:51,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:09:51,188 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:09:51,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:09:51,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:09:51,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:09:51,253 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 05:09:51,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:09:51,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:53,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:09:53,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:53,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:54,147 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:09:54,164 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 05:09:54,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:09:54,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:54,249 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:09:54,283 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 05:09:54,285 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:09:54,286 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 05:09:54,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:54,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 05:09:54,681 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:54,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 05:09:54,698 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 05:09:54,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 05:09:54,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 05:09:54,752 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 05:09:54,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 05:09:54,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 05:09:54,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:09:54,818 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 05:09:54,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:09:54,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:54,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:09:54,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:54,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:54,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:09:54,931 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 05:09:54,931 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:09:54,948 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:54,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:09:54,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:55,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:09:55,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:55,053 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:55,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:55,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:55,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:55,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:55,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:55,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:55,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:55,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:09:55,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:55,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:55,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:55,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:55,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:55,326 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:55,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:55,357 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:55,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:55,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:09:55,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:55,415 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:55,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:55,446 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:55,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:55,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:55,497 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:55,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:55,535 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:55,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:09:55,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:55,591 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:55,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:55,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:55,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:55,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:55,691 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:55,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:55,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:55,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:09:55,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:55,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:55,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:55,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:55,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:55,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:55,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:55,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:55,927 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 05:09:55,930 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 05:09:56,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:56,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 05:09:56,288 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:56,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 05:09:56,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 05:09:56,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 05:09:56,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 05:09:56,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 05:09:56,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 05:09:56,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 05:09:56,390 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:09:56,405 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 05:09:56,438 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:09:56,454 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:56,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:09:56,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:56,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:56,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:09:56,526 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 05:09:56,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:09:56,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:56,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:09:56,566 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:56,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:09:56,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:56,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:56,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:56,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:56,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:56,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:56,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:56,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:09:56,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:56,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:09:56,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:56,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:56,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:56,864 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:56,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:56,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:56,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:56,931 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:09:56,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:56,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:09:56,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:57,002 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:57,017 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:57,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:57,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:57,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:57,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:57,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:09:57,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:57,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:09:57,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:57,160 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:57,176 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:57,191 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:57,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:57,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:57,242 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:57,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:09:57,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:57,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:09:57,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:57,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:57,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:57,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:57,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:57,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:57,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:57,433 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:09:57,448 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 05:09:57,450 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:09:57,543 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:09:57,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:09:57,928 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:09:57,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:09:57,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:09:57,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:09:58,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:09:58,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:09:58,066 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:09:58,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:09:58,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:09:58,111 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 05:09:58,140 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:09:58,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:58,586 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:09:59,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:59,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:09:59,658 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:09:59,677 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 05:09:59,678 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:09:59,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:09:59,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:09:59,780 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 05:09:59,780 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:09:59,786 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:09:59,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:10:00,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:10:00,278 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:10:00,278 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:10:00,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:10:00,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:10:00,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:10:00,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:10:00,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:10:00,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:10:00,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:10:00,437 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 05:10:00,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:10:00,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:10:03,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:10:03,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:10:04,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:10:04,336 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:10:04,354 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 05:10:04,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:10:04,372 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:10:04,408 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:10:04,424 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 05:10:04,424 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:10:04,427 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:10:04,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:10:05,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:10:05,246 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:10:05,247 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:10:05,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:10:05,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:10:05,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:10:05,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:10:05,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:10:05,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:10:05,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:10:05,691 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 05:10:05,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:10:05,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:10:08,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:10:09,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:10:09,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:10:09,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:10:09,526 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 05:10:09,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:10:09,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:10:09,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:10:09,601 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 05:10:09,601 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:10:09,602 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:10:09,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:10:09,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:10:10,019 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:10:10,020 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:10:10,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:10:10,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:10:10,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:10:10,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:10:10,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:10:10,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:10:10,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:10:10,161 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 05:10:10,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:10:10,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:10:10,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:10:11,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:10:12,094 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:10:12,492 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:10:12,509 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 05:10:12,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:10:12,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:10:12,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:10:12,707 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 05:10:12,707 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:10:12,710 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 05:10:12,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:10:13,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 05:10:13,124 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:10:13,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 05:10:13,140 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 05:10:13,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 05:10:13,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 05:10:13,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 05:10:13,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 05:10:13,229 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 05:10:13,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:10:13,266 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 05:10:13,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:10:13,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:10:13,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:10:13,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:10:13,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:10:13,377 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:10:13,394 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 05:10:13,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:10:13,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:10:13,438 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:10:13,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:10:13,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:10:13,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:10:13,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:10:13,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:10:13,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:10:13,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:10:13,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:10:13,645 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:10:13,670 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:10:13,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:10:13,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:10:13,727 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:10:13,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:10:13,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:10:13,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:10:13,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:10:13,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:10:13,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:10:13,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:10:13,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:10:13,905 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:10:13,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:10:13,946 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:10:13,964 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:10:13,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:10:14,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:10:14,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:10:14,038 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:10:14,061 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:10:14,079 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:10:14,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:10:14,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:10:14,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:10:14,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:10:14,165 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:10:14,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:10:14,201 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:10:14,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:10:14,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:10:14,255 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:10:14,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:10:14,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:10:14,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:10:14,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:10:14,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:10:14,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:10:14,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:10:14,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:10:14,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:10:14,467 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 05:10:14,467 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:10:14,470 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:10:14,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:10:14,974 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:10:15,079 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:10:15,079 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:10:15,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:10:15,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:10:15,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:10:15,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:10:15,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:10:15,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:10:15,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:15,802 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:10:15,802 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.33500003815. 2022-02-08 05:10:15,802 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:10:15,805 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:10:15,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:10:16,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:10:16,369 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:10:16,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:10:16,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:10:16,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:10:16,431 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:10:16,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:10:16,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:10:16,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:10:16,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:16,799 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:10:16,799 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:10:18,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:18,983 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:10:18,984 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:10:20,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:21,150 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:10:21,150 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:10:23,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:23,469 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:10:23,469 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:10:25,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:25,759 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:10:25,759 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:10:27,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:28,055 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:10:28,055 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:10:30,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:30,295 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:10:30,296 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:10:32,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:32,496 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:10:32,496 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:10:34,497 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:34,767 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:10:34,767 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:10:36,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:37,053 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:10:37,053 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:10:39,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:39,380 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:10:39,380 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:10:41,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:41,650 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:10:41,650 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:10:43,650 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:43,987 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:10:43,987 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:10:45,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:46,171 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:10:46,171 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:10:48,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:48,387 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:10:48,387 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 7095 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:10:50,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:50,596 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:10:50,598 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 34.7950000763. 2022-02-08 05:10:50,598 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:10:50,599 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:10:51,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:10:51,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:10:52,051 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:10:52,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:10:52,102 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:10:52,132 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:10:52,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:10:52,186 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:10:52,216 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:10:52,278 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:10:52,303 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:52,957 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:10:52,957 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 05:10:54,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:55,141 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:10:55,142 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.54399991035. 2022-02-08 05:10:55,142 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:10:55,144 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:10:55,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:10:56,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:10:56,539 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:10:56,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:10:56,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:10:56,584 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:10:56,612 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:10:56,637 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:10:56,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:10:56,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:10:56,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:57,525 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:10:57,526 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.38300013542. 2022-02-08 05:10:57,526 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:10:57,528 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:10:58,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:10:58,586 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:10:58,720 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:10:58,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:10:58,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:10:58,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:10:58,790 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:10:58,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:10:58,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:10:58,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:10:58,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:10:59,509 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:10:59,510 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.9849998951. 2022-02-08 05:10:59,510 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:10:59,513 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:11:00,221 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:11:00,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:11:00,737 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:11:00,739 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:11:00,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:11:00,783 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:11:00,806 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:11:00,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:11:00,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:11:00,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:11:00,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:11:01,448 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:11:01,450 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.94000005722. 2022-02-08 05:11:01,450 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:11:01,453 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:11:02,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:11:02,474 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:11:02,621 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:11:02,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:11:02,642 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:11:02,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:11:02,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:11:02,696 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:11:02,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:11:02,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:11:02,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:11:03,259 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:11:03,259 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.80900001526. 2022-02-08 05:11:03,259 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:11:03,262 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:11:03,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:11:04,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:11:04,391 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:11:04,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:11:04,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:11:04,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:11:04,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:11:04,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:11:04,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:11:04,553 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:11:04,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:11:05,158 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:11:05,158 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.89899992943. 2022-02-08 05:11:05,160 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:11:05,161 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:11:05,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:11:06,079 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:11:06,190 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:11:06,191 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:11:06,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:11:06,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:11:06,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:11:06,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:11:06,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:11:06,303 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:11:06,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:11:06,818 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:11:06,818 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.65799999237. 2022-02-08 05:11:06,818 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:11:06,819 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:11:07,849 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:11:08,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:11:09,141 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:11:09,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:11:09,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:11:09,270 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:11:09,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:11:09,576 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:11:09,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:11:09,844 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:11:09,961 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:11:10,868 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:11:10,868 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 05:11:12,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:11:13,049 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:11:13,049 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.23199987411. ############################################################ # # RAM FootPrint: 4381.61328125 # ############################################################ 2022-02-08 05:11:13,089 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 05:11:13,089 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 62 seconds. Reason: Iteration:12 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 7111 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:12:15,315 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 05:12:15,316 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 05:12:20,394 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:12:20,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:21,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:12:21,411 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:21,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:12:21,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:12:21,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:12:21,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:12:21,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:12:21,494 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:12:21,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:12:21,539 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:12:21,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:12:21,607 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['130838 memcached\n'], []) 2022-02-08 05:12:21,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 05:12:21,645 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:12:21,673 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['130838 memcached\n'], []) 2022-02-08 05:12:21,673 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:12:21,676 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:12:22,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:22,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:12:22,631 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:22,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:12:22,645 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:12:22,665 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:12:22,678 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:12:22,696 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:12:22,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:12:22,739 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:12:22,753 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:12:22,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:12:22,809 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['44097 memcached\n'], []) 2022-02-08 05:12:22,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 05:12:22,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:12:22,852 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['44097 memcached\n'], []) 2022-02-08 05:12:22,852 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:12:22,855 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:12:22,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:23,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:12:23,233 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:23,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:12:23,249 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:12:23,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:12:23,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:12:23,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:12:23,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:12:23,326 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:12:23,342 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:12:23,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:12:23,405 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['31079 memcached\n'], []) 2022-02-08 05:12:23,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 05:12:23,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:12:23,457 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['31079 memcached\n'], []) 2022-02-08 05:12:23,459 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:12:23,460 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:12:23,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:23,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:12:23,812 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:23,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:12:23,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:12:23,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:12:23,849 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:12:23,880 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:12:23,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:12:23,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:12:23,917 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:12:23,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:12:23,967 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['64779 memcached\n'], []) 2022-02-08 05:12:23,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 05:12:23,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:12:24,012 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['64779 memcached\n'], []) 2022-02-08 05:12:24,013 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:12:24,015 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:12:24,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:24,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:12:24,365 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:24,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:12:24,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:12:24,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:12:24,403 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:12:24,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:12:24,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:12:24,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:12:24,459 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:12:24,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:12:24,502 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['58612 memcached\n'], []) 2022-02-08 05:12:24,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 05:12:24,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:12:24,551 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['58612 memcached\n'], []) 2022-02-08 05:12:24,551 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:12:24,552 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:12:24,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:24,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:12:24,913 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:24,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:12:24,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:12:24,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:12:24,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:12:24,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:12:24,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:12:24,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:12:25,013 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:12:25,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:12:25,078 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['93521 memcached\n'], []) 2022-02-08 05:12:25,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 05:12:25,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:12:25,122 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['93521 memcached\n'], []) 2022-02-08 05:12:25,122 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:12:25,125 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:12:25,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:25,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:12:25,471 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:25,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:12:25,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:12:25,497 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:12:25,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:12:25,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:12:25,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:12:25,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:12:25,565 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:12:25,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:12:25,608 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['27391 memcached\n'], []) 2022-02-08 05:12:25,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 05:12:25,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:12:25,654 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['27391 memcached\n'], []) 2022-02-08 05:12:25,655 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:12:25,657 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:12:25,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:25,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:12:26,032 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:26,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:12:26,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:12:26,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:12:26,071 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:12:26,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:12:26,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:12:26,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:12:26,128 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:12:26,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:12:26,187 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['42276 memcached\n'], []) 2022-02-08 05:12:26,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 05:12:26,211 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:12:26,234 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['42276 memcached\n'], []) 2022-02-08 05:12:26,234 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:12:26,237 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:12:26,321 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:26,548 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:12:26,618 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:26,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:12:26,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:12:26,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:12:26,655 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:12:26,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:12:26,686 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:12:26,698 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:12:26,711 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:12:26,711 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:12:26,762 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['78320 memcached\n'], []) 2022-02-08 05:12:26,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 05:12:26,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:12:26,808 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['78320 memcached\n'], []) 2022-02-08 05:12:26,808 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:12:26,809 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:12:26,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:27,107 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:12:27,168 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:27,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:12:27,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:12:27,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:12:27,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:12:27,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:12:27,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:12:27,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:12:27,266 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:12:27,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:12:27,315 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['107964 memcached\n'], []) 2022-02-08 05:12:27,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 05:12:27,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:12:27,359 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['107964 memcached\n'], []) 2022-02-08 05:12:27,361 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:12:27,362 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:12:27,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:27,686 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:12:27,750 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:27,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:12:27,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:12:27,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:12:27,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:12:27,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:12:27,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:12:27,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:12:27,864 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:12:27,881 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 05:12:27,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:12:27,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:28,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:12:28,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:29,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:29,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:12:29,298 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 05:12:29,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:12:29,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:29,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:12:29,380 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 05:12:29,380 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:12:29,381 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:12:29,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:29,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:12:29,802 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:29,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:12:29,816 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:12:29,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:12:29,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:12:29,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:12:29,882 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:12:29,897 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:12:29,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:12:29,928 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 05:12:29,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:12:29,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:30,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:12:30,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:30,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:30,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:12:30,266 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 05:12:30,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:12:30,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:30,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:12:30,319 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 05:12:30,319 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:12:30,321 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:12:30,395 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:30,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:12:30,670 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:30,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:12:30,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:12:30,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:12:30,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:12:30,724 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:12:30,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:12:30,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:12:30,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:12:30,782 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 05:12:30,811 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:12:30,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:32,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:12:32,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:32,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:33,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:12:33,108 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 05:12:33,109 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:12:33,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:33,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:12:33,200 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 05:12:33,200 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:12:33,201 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 05:12:33,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:33,489 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 05:12:33,546 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:33,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 05:12:33,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 05:12:33,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 05:12:33,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 05:12:33,602 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 05:12:33,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 05:12:33,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 05:12:33,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:12:33,677 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 05:12:33,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:12:33,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:33,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:12:33,744 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:33,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:33,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:12:33,785 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 05:12:33,785 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:12:33,798 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:33,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:12:33,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:33,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:12:33,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:33,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:33,905 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:33,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:33,943 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:33,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:33,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:33,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:34,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:34,030 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:12:34,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:34,061 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:34,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:34,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:34,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:34,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:34,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:34,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:34,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:34,186 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:12:34,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:34,216 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:34,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:34,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:34,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:34,283 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:34,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:34,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:34,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:34,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:12:34,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:34,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:34,385 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:34,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:34,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:34,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:34,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:34,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:34,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:34,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:12:34,494 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:34,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:34,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:34,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:34,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:34,573 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:34,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:34,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:34,625 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 05:12:34,627 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:12:34,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:34,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:12:34,987 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:34,989 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:12:35,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:12:35,015 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:12:35,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:12:35,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:12:35,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:12:35,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:12:35,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:12:35,099 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 05:12:35,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:12:35,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:35,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:12:35,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:35,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:36,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:12:36,055 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 05:12:36,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:12:36,066 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:36,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:12:36,128 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 05:12:36,130 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:12:36,131 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 05:12:36,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:36,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 05:12:36,480 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:36,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 05:12:36,494 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 05:12:36,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 05:12:36,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 05:12:36,545 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 05:12:36,558 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 05:12:36,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 05:12:36,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:12:36,601 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 05:12:36,630 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:12:36,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:36,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:12:36,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:36,701 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:36,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:12:36,729 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 05:12:36,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:12:36,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:36,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:12:36,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:36,793 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:12:36,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:36,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:36,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:36,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:36,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:36,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:36,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:36,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:37,015 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:37,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:12:37,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:37,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:37,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:37,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:37,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:37,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:37,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:37,214 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:37,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:37,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:12:37,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:37,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:37,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:37,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:37,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:37,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:37,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:37,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:37,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:37,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:12:37,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:37,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:37,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:37,477 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:37,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:37,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:37,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:37,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:37,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:37,576 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:12:37,599 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:37,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:37,640 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:37,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:37,676 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:37,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:37,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:37,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:37,753 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 05:12:37,757 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:12:37,829 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:38,035 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:12:38,096 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:38,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:12:38,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:12:38,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:12:38,135 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:12:38,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:12:38,163 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:12:38,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:12:38,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:12:38,203 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 05:12:38,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:12:38,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:40,655 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:12:41,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:41,403 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:41,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:12:41,683 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 05:12:41,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:12:41,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:41,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:12:41,759 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 05:12:41,760 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:12:41,763 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:12:41,839 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:42,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:12:42,127 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:42,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:12:42,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:12:42,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:12:42,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:12:42,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:12:42,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:12:42,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:12:42,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:12:42,262 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 05:12:42,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:12:42,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:44,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:12:45,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:45,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:45,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:12:45,911 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 05:12:45,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:12:45,926 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:45,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:12:46,003 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 05:12:46,003 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:12:46,005 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 05:12:46,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:46,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 05:12:46,361 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:46,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 05:12:46,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 05:12:46,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 05:12:46,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 05:12:46,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 05:12:46,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 05:12:46,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 05:12:46,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:12:46,470 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 05:12:46,494 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:12:46,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:46,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:12:46,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:46,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:46,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:12:46,592 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 05:12:46,594 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:12:46,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:46,619 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:12:46,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:46,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:12:46,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:46,696 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:46,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:46,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:46,755 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:46,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:46,793 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:46,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:46,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:46,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:12:46,864 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:46,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:46,892 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:46,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:46,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:46,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:46,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:46,969 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:46,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:46,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:12:47,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:47,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:47,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:47,053 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:47,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:47,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:47,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:47,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:47,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:47,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:12:47,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:47,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:47,188 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:47,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:47,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:47,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:47,247 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:47,262 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:47,275 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:47,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:12:47,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:47,321 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:47,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:47,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:47,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:47,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:47,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:47,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:47,427 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 05:12:47,428 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 05:12:47,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:47,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 05:12:47,815 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:47,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 05:12:47,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 05:12:47,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 05:12:47,857 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 05:12:47,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 05:12:47,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 05:12:47,900 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 05:12:47,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:12:47,927 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 05:12:47,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:12:47,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:47,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:12:47,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:48,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:48,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:12:48,033 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 05:12:48,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:12:48,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:48,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:12:48,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:48,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:12:48,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:48,135 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:48,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:48,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:48,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:48,211 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:48,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:48,270 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:12:48,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:48,308 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:12:48,323 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:48,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:48,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:48,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:48,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:48,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:48,433 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:48,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:12:48,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:48,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:12:48,497 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:48,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:48,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:48,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:48,556 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:48,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:48,586 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:48,602 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:12:48,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:48,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:12:48,658 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:48,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:48,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:48,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:48,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:48,734 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:48,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:48,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:12:48,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:48,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:12:48,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:48,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:48,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:48,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:48,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:48,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:48,918 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:48,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:12:48,953 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 05:12:48,954 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:12:49,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:49,255 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:12:49,315 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:49,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:12:49,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:12:49,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:12:49,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 5 queries submitted, 0 failed, 7134 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:12:49,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:12:49,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:12:49,392 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:12:49,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:12:49,428 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 05:12:49,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:12:49,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:50,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:12:51,232 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:51,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:51,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:12:51,779 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 05:12:51,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:12:51,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:51,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:12:51,857 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 05:12:51,857 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:12:51,859 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:12:51,943 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:52,163 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:12:52,220 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:52,221 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:12:52,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:12:52,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:12:52,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:12:52,280 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:12:52,295 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:12:52,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:12:52,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:12:52,334 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 05:12:52,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:12:52,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:52,752 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:12:53,094 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:53,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:53,515 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:12:53,530 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 05:12:53,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:12:53,545 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:53,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:12:53,598 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 05:12:53,598 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:12:53,601 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:12:53,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:53,943 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:12:54,006 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:54,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:12:54,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:12:54,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:12:54,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:12:54,071 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:12:54,089 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:12:54,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:12:54,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:12:54,135 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 05:12:54,160 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:12:54,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:54,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:12:54,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:55,073 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:55,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:12:55,286 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 05:12:55,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:12:55,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:55,334 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:12:55,351 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 05:12:55,351 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:12:55,352 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:12:55,433 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:12:55,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:12:55,779 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:12:55,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:12:55,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:12:55,821 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:12:55,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:12:55,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:12:55,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:12:55,931 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:12:55,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:12:55,983 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 05:12:56,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:12:56,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:12:58,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:12:59,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:59,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:12:59,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:12:59,960 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 05:12:59,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:12:59,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:13:00,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:13:00,055 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 05:13:00,055 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:13:00,058 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 05:13:00,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:13:00,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 05:13:00,444 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:13:00,446 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 05:13:00,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 05:13:00,474 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 05:13:00,489 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 05:13:00,507 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 05:13:00,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 05:13:00,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 05:13:00,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:13:00,568 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 05:13:00,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:13:00,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:13:00,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:13:00,645 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:13:00,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:13:00,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:13:00,703 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 05:13:00,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:13:00,721 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:13:00,739 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:13:00,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:13:00,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:13:00,803 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:13:00,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:13:00,846 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:13:00,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:13:00,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:13:00,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:13:00,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:13:00,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:13:00,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:13:01,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:13:01,028 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:13:01,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:13:01,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:13:01,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:13:01,102 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:13:01,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:13:01,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:13:01,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:13:01,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:13:01,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:13:01,209 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:13:01,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:13:01,265 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:13:01,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:13:01,308 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:13:01,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:13:01,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:13:01,362 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:13:01,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:13:01,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:13:01,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:13:01,436 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:13:01,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:13:01,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:13:01,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:13:01,507 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:13:01,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:13:01,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:13:01,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:13:01,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:13:01,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:13:01,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:13:01,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:13:01,655 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:13:01,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:13:01,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:13:01,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:13:01,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:13:01,752 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 05:13:01,752 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:13:01,753 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:13:01,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:13:02,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:13:02,877 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:13:02,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:13:03,089 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:13:03,280 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:13:03,438 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:13:03,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:13:03,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:13:03,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:13:03,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:04,920 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:04,920 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 05:13:06,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:07,115 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:07,115 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 05:13:09,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:09,321 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:09,322 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 05:13:11,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:11,483 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:13:11,483 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 9.73199987411. 2022-02-08 05:13:11,483 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:13:11,486 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:13:11,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:13:11,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:13:12,019 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:13:12,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:13:12,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:13:12,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:13:12,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:13:12,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:13:12,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:13:12,176 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:13:12,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:12,375 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:12,375 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:14,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:14,677 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:14,677 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:16,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:16,900 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:16,901 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:18,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:19,184 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:19,184 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:21,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:21,395 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:21,395 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:23,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:23,562 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:23,562 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:25,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:25,743 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:25,743 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:27,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:28,033 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:28,033 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:30,035 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:30,538 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:30,539 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:32,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:33,073 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:33,073 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:35,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:35,335 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:35,335 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:37,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:37,585 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:37,585 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:39,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:39,953 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:39,953 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:41,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:42,158 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:42,158 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:44,160 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:44,546 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:44,546 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:46,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:46,851 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:46,851 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:48,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:49,068 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:49,068 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 7136 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:13:51,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:51,397 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:51,398 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:53,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:53,625 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:53,625 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:13:55,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:55,805 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:13:55,805 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 44.3220000267. 2022-02-08 05:13:55,806 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:13:55,809 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:13:56,637 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:13:57,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:13:57,517 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:13:57,517 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:13:57,573 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:13:57,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:13:57,645 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:13:57,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:13:57,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:13:57,803 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:13:57,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:13:58,380 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:13:58,381 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 05:14:00,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:14:00,698 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:14:00,698 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 05:14:02,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:14:03,286 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:14:03,286 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 05:14:05,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:14:05,744 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:14:05,744 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 05:14:07,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:14:08,088 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:14:08,088 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 12.2820000648. 2022-02-08 05:14:08,088 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:14:08,092 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:14:08,910 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:14:09,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:14:09,512 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:14:09,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:14:09,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:14:09,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:14:09,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:14:09,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:14:09,670 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:14:09,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:14:09,757 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:14:11,417 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:14:11,417 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.32800006866. 2022-02-08 05:14:11,417 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:14:11,418 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:14:12,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:14:12,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:14:12,951 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:14:12,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:14:13,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:14:13,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:14:13,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:14:13,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:14:13,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:14:13,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:14:13,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:14:21,474 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:14:21,474 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 10.0579998493. 2022-02-08 05:14:21,474 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:14:21,477 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:14:22,591 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:14:23,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:14:23,322 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:14:23,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:14:23,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:14:23,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:14:23,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:14:23,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:14:23,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:14:23,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:14:23,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:14:29,711 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:14:29,713 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 8.23699998856. 2022-02-08 05:14:29,713 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:14:29,714 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:14:30,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:14:30,892 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:14:31,072 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:14:31,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:14:31,086 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:14:31,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:14:31,114 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:14:31,130 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:14:31,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:14:31,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:14:31,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:14:31,653 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:14:31,654 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.94099998474. 2022-02-08 05:14:31,654 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:14:31,655 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:14:32,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:14:32,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:14:32,503 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:14:32,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:14:32,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:14:32,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:14:32,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:14:32,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:14:32,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:14:32,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:14:32,655 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:14:33,138 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:14:33,138 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.4849998951. 2022-02-08 05:14:33,138 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:14:33,141 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:14:33,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:14:34,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:14:34,486 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:14:34,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:14:34,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:14:34,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:14:34,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:14:34,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:14:34,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:14:34,586 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:14:34,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:14:35,068 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:14:35,068 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.92900013924. 2022-02-08 05:14:35,068 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:14:35,069 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:14:35,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:14:35,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:14:35,821 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:14:35,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:14:35,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:14:35,859 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:14:35,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:14:35,900 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:14:35,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:14:35,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:14:35,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:14:36,430 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:14:36,430 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.36299991608. ############################################################ # # RAM FootPrint: 4365.2734375 # ############################################################ 2022-02-08 05:14:36,607 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 05:14:36,608 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 70 seconds. Reason: Iteration:13 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 7154 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:15:46,644 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 05:15:46,644 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 5 queries submitted, 0 failed, 7180 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:15:51,648 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:15:52,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:15:52,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:15:52,506 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:15:52,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:15:52,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:15:52,558 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:15:52,573 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:15:52,591 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:15:52,604 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:15:52,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:15:52,644 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:15:52,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:15:52,700 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['697 memcached\n'], []) 2022-02-08 05:15:52,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 05:15:52,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:15:52,755 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['697 memcached\n'], []) 2022-02-08 05:15:52,756 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:15:52,756 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:15:53,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:15:53,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:15:53,664 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:15:53,665 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:15:53,678 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:15:53,698 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:15:53,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:15:53,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:15:53,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:15:53,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:15:53,786 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:15:53,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:15:53,842 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['44902 memcached\n'], []) 2022-02-08 05:15:53,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 05:15:53,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:15:53,900 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['44902 memcached\n'], []) 2022-02-08 05:15:53,900 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:15:53,903 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:15:53,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:15:54,201 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:15:54,265 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:15:54,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:15:54,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:15:54,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:15:54,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:15:54,329 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:15:54,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:15:54,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:15:54,374 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:15:54,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:15:54,427 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['31818 memcached\n'], []) 2022-02-08 05:15:54,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 05:15:54,454 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:15:54,476 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['31818 memcached\n'], []) 2022-02-08 05:15:54,477 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:15:54,480 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:15:54,556 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:15:54,762 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:15:54,828 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:15:54,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:15:54,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:15:54,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:15:54,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:15:54,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:15:54,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:15:54,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:15:54,923 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:15:54,923 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:15:54,970 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['65372 memcached\n'], []) 2022-02-08 05:15:54,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 05:15:54,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:15:55,015 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['65372 memcached\n'], []) 2022-02-08 05:15:55,015 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:15:55,017 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:15:55,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:15:55,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:15:55,354 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:15:55,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:15:55,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:15:55,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:15:55,392 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:15:55,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:15:55,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:15:55,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:15:55,447 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:15:55,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:15:55,509 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['59236 memcached\n'], []) 2022-02-08 05:15:55,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 05:15:55,535 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:15:55,555 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['59236 memcached\n'], []) 2022-02-08 05:15:55,555 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:15:55,558 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:15:55,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:15:55,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:15:55,931 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:15:55,933 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:15:55,946 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:15:55,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:15:55,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:15:55,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:15:56,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:15:56,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:15:56,026 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:15:56,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:15:56,084 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['94147 memcached\n'], []) 2022-02-08 05:15:56,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 05:15:56,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:15:56,131 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['94147 memcached\n'], []) 2022-02-08 05:15:56,131 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:15:56,132 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:15:56,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:15:56,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:15:56,473 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:15:56,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:15:56,489 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:15:56,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:15:56,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:15:56,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:15:56,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:15:56,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:15:56,569 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:15:56,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:15:56,660 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['27991 memcached\n'], []) 2022-02-08 05:15:56,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 05:15:56,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:15:56,704 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['27991 memcached\n'], []) 2022-02-08 05:15:56,706 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:15:56,707 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:15:56,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:15:57,012 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:15:57,078 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:15:57,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:15:57,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:15:57,107 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:15:57,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:15:57,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:15:57,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:15:57,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:15:57,177 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:15:57,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:15:57,299 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['42913 memcached\n'], []) 2022-02-08 05:15:57,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 05:15:57,323 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:15:57,345 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['42913 memcached\n'], []) 2022-02-08 05:15:57,345 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:15:57,346 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:15:57,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:15:57,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:15:57,700 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:15:57,701 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:15:57,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:15:57,727 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:15:57,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:15:57,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:15:57,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:15:57,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:15:57,795 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:15:57,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:15:57,849 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['78945 memcached\n'], []) 2022-02-08 05:15:57,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 05:15:57,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:15:57,907 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['78945 memcached\n'], []) 2022-02-08 05:15:57,907 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:15:57,914 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:15:57,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:15:58,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:15:58,276 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:15:58,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:15:58,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:15:58,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:15:58,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:15:58,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:15:58,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:15:58,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:15:58,371 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:15:58,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:15:58,414 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['108621 memcached\n'], []) 2022-02-08 05:15:58,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 05:15:58,446 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:15:58,467 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['108621 memcached\n'], []) 2022-02-08 05:15:58,467 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:15:58,470 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:15:58,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:15:58,788 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:15:58,851 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:15:58,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:15:58,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:15:58,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:15:58,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:15:58,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:15:58,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:15:58,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:15:58,948 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:15:58,963 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 05:15:58,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:15:59,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:15:59,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:16:00,188 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:00,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:00,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:16:00,716 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 05:16:00,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:16:00,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:00,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:16:00,878 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 05:16:00,880 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:16:00,881 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:16:00,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:16:01,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:16:01,240 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:16:01,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:16:01,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:16:01,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:16:01,278 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:16:01,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:16:01,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:16:01,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:16:01,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:16:01,348 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 05:16:01,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:16:01,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:02,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:16:02,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:02,349 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:02,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:16:02,414 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 05:16:02,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:16:02,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:02,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:16:02,466 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 05:16:02,466 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:16:02,467 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:16:02,548 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:16:02,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:16:02,819 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:16:02,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:16:02,834 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:16:02,846 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:16:02,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:16:02,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:16:02,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:16:02,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:16:02,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:16:02,937 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 05:16:02,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:16:02,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:04,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:16:04,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:05,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:05,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:16:05,388 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 05:16:05,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:16:05,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:05,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:16:05,471 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 05:16:05,473 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:16:05,473 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 05:16:05,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:16:05,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 05:16:05,878 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:16:05,880 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 05:16:05,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 05:16:05,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 05:16:05,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 05:16:05,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 05:16:05,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 05:16:05,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 05:16:05,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:16:06,000 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 05:16:06,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:16:06,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:06,053 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:16:06,071 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:06,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:06,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:16:06,111 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 05:16:06,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:16:06,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:06,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:16:06,158 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:06,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:16:06,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:06,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:06,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:06,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:06,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:06,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:06,326 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:06,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:06,367 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:06,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:16:06,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:06,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:06,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:06,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:06,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:06,474 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:06,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:06,505 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:06,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:06,540 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:16:06,556 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:06,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:06,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:06,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:06,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:06,637 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:06,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:06,668 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:06,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:06,698 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:16:06,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:06,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:06,744 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:06,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:06,785 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:06,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:06,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:06,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:06,846 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:06,862 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:16:06,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:06,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:06,910 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:06,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:06,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:06,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:06,971 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:06,989 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:07,003 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 05:16:07,006 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:16:07,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:16:07,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:16:07,385 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:16:07,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:16:07,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:16:07,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:16:07,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:16:07,441 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:16:07,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:16:07,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:16:07,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:16:07,496 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 05:16:07,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:16:07,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:08,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:16:08,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:08,650 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:08,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:16:08,923 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 05:16:08,923 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:16:08,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:09,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:16:09,016 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 05:16:09,016 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:16:09,017 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 05:16:09,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:16:09,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 05:16:09,404 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:16:09,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 05:16:09,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 05:16:09,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 05:16:09,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 05:16:09,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 05:16:09,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 05:16:09,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 05:16:09,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:16:09,526 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 05:16:09,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:16:09,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:09,586 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:16:09,602 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:09,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:09,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:16:09,644 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 05:16:09,645 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:16:09,658 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:09,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:16:09,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:09,711 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:16:09,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:09,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:09,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:09,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:09,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:09,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:09,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:09,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:09,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:09,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:16:09,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:09,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:09,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:10,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:10,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:10,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:10,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:10,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:10,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:10,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:16:10,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:10,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:10,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:10,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:10,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:10,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:10,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:10,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:10,270 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:10,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:16:10,303 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:10,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:10,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:10,352 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:10,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:10,385 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:10,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:10,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:10,436 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:10,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:16:10,474 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:10,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:10,512 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:10,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:10,548 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:10,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:10,584 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:10,604 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:10,621 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 05:16:10,625 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:16:10,696 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:16:10,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:16:10,951 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:16:10,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:16:10,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:16:10,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:16:10,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:16:11,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:16:11,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:16:11,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:16:11,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:16:11,059 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 05:16:11,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:16:11,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:13,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:16:13,755 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:14,040 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:14,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:16:14,312 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 05:16:14,313 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:16:14,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:14,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:16:14,382 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 05:16:14,382 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:16:14,384 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:16:14,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:16:14,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:16:14,753 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:16:14,755 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:16:14,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:16:14,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:16:14,806 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:16:14,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:16:14,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:16:14,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:16:14,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:16:14,894 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 05:16:14,923 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:16:14,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:17,086 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:16:17,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:17,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:18,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:16:18,023 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 05:16:18,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:16:18,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:18,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:16:18,099 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 05:16:18,101 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:16:18,101 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 05:16:18,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:16:18,408 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 05:16:18,463 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:16:18,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 05:16:18,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 05:16:18,489 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 05:16:18,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 05:16:18,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 05:16:18,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 05:16:18,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 05:16:18,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:16:18,575 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 05:16:18,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:16:18,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:18,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:16:18,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:18,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:18,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:16:18,703 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 05:16:18,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:16:18,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:18,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:16:18,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:18,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:16:18,788 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:18,806 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:18,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:18,854 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:18,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:18,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:18,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:18,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:18,974 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:18,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:16:19,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:19,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:19,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:19,051 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:19,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:19,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:19,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:19,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:19,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:19,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:16:19,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:19,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:19,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:19,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:19,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:19,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:19,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:19,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:19,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:19,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:16:19,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:19,318 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:19,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:19,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:19,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:19,377 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:19,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:19,405 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:19,420 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:19,436 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:16:19,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:19,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:19,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:19,507 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:19,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:19,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:19,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:19,602 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:19,619 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 05:16:19,621 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 05:16:19,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:16:19,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 05:16:20,009 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:16:20,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 05:16:20,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 05:16:20,038 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 05:16:20,051 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 05:16:20,066 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 05:16:20,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 05:16:20,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 05:16:20,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:16:20,124 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 05:16:20,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:16:20,165 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:20,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:16:20,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:20,214 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:20,229 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:16:20,243 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 05:16:20,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:16:20,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:20,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:16:20,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:20,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:16:20,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:20,357 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:20,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:20,403 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:20,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:20,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:20,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:20,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:20,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:20,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:16:20,558 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:20,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:20,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:20,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:20,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:20,647 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:20,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:20,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:20,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:20,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:16:20,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:20,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:20,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:20,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:20,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:20,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:20,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:20,862 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:20,877 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:20,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:16:20,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:20,923 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:20,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:20,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:20,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:20,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:20,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:21,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:21,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:21,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:16:21,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:21,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:21,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:21,107 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:21,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:21,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:21,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:21,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:21,187 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 05:16:21,188 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:16:21,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:16:21,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:16:21,578 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:16:21,579 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:16:21,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:16:21,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:16:21,624 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:16:21,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:16:21,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:16:21,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:16:21,721 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:16:21,743 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 05:16:21,775 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:16:21,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:23,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:16:24,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:24,507 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:24,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:16:24,769 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 05:16:24,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:16:24,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:24,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:16:24,859 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 05:16:24,861 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:16:24,862 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:16:24,999 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:16:25,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:16:25,387 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:16:25,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:16:25,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:16:25,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:16:25,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:16:25,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:16:25,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:16:25,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:16:25,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:16:25,526 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 05:16:25,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:16:25,568 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:25,900 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:16:26,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:26,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:26,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:16:26,756 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 05:16:26,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:16:26,793 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:26,846 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:16:26,884 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 05:16:26,885 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:16:26,888 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:16:27,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:16:27,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:16:28,079 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:16:28,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:16:28,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:16:28,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:16:28,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:16:28,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:16:28,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:16:28,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:16:28,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:16:28,780 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 05:16:28,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:16:28,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:32,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:16:32,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:32,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:32,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:16:33,104 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 05:16:33,104 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:16:33,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:33,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:16:33,296 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 05:16:33,296 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:16:33,298 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:16:33,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:16:33,619 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:16:33,681 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:16:33,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:16:33,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:16:33,711 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:16:33,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:16:33,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:16:33,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:16:33,778 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:16:33,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:16:33,813 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 05:16:33,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:16:33,854 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:36,395 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:16:36,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:37,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:37,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:16:37,474 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 05:16:37,474 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:16:37,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:37,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:16:37,556 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 05:16:37,558 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:16:37,559 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 05:16:37,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:16:37,905 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 05:16:37,974 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:16:37,974 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 05:16:37,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 05:16:38,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 05:16:38,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 05:16:38,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 05:16:38,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 05:16:38,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 05:16:38,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:16:38,111 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 05:16:38,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:16:38,158 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:38,176 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:16:38,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:38,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:16:38,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:16:38,240 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 05:16:38,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:16:38,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:16:38,275 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:16:38,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:38,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:16:38,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:38,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:38,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:38,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:38,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:38,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:38,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:38,517 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:16:38,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:38,561 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:16:38,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:38,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:38,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:38,642 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:38,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:38,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:38,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:38,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:16:38,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:38,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:16:38,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:38,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:38,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:38,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:38,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:38,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:38,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:38,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:16:38,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:38,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:16:38,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:38,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:38,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:39,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:39,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:39,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:39,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:39,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:16:39,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:39,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:16:39,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:39,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:39,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:39,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:39,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:39,244 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:39,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:39,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:16:39,301 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 05:16:39,302 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:16:39,305 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:16:39,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:16:39,665 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:16:39,740 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:16:39,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:16:39,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:16:39,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:16:39,785 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:16:39,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:16:39,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:16:39,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:16:39,867 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:16:40,499 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:16:40,500 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.19899988174. 2022-02-08 05:16:40,500 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:16:40,503 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:16:40,599 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:16:40,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:16:40,921 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:16:40,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:16:40,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:16:40,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:16:40,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:16:40,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:16:41,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:16:41,017 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:16:41,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:16:41,223 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:16:41,223 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:16:43,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:16:43,378 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:16:43,378 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:16:45,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:16:45,572 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:16:45,572 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:16:47,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:16:47,746 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:16:47,746 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 7191 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:16:49,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:16:49,914 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:16:49,915 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:16:51,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:16:52,062 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:16:52,062 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:16:54,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:16:54,240 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:16:54,242 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:16:56,242 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:16:56,418 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:16:56,418 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:16:58,420 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:16:58,815 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:16:58,815 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:00,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:01,101 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:01,101 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:03,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:03,315 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:03,315 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:05,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:05,654 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:05,654 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:07,655 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:07,967 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:07,969 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:09,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:10,280 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:10,280 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:12,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:12,476 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:12,476 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:14,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:14,671 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:14,671 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:16,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:16,907 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:16,907 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:18,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:19,168 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:19,170 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:21,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:21,381 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:21,382 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:23,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:23,601 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:23,601 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:25,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:25,838 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:25,838 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:27,839 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:28,012 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:28,012 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:30,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:30,193 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:30,193 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:32,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:32,411 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:32,411 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:34,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:34,671 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:34,671 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:36,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:37,013 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:37,013 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:39,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:39,266 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:17:39,266 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:17:41,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:41,453 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:17:41,454 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 60.9530000687. 2022-02-08 05:17:41,454 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:17:41,457 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:17:41,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:17:42,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:17:42,332 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:17:42,334 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:17:42,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:17:42,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:17:42,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:17:42,446 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:17:42,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:17:42,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:17:42,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:42,862 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:17:42,864 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.40900015831. 2022-02-08 05:17:42,864 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:17:42,865 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:17:43,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:17:44,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:17:44,398 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:17:44,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:17:44,433 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:17:44,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:17:44,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:17:44,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:17:44,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:17:44,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:17:44,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:45,221 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:17:45,223 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.35899996758. 2022-02-08 05:17:45,223 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:17:45,224 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:17:46,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:17:46,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:17:46,665 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:17:46,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:17:46,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:17:46,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:17:46,727 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:17:46,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:17:46,783 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:17:46,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:17:46,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:47,461 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:17:47,461 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.2389998436. 2022-02-08 05:17:47,461 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:17:47,463 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:17:48,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:17:48,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:17:49,082 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:17:49,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:17:49,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:17:49,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:17:49,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:17:49,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:17:49,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:17:49,247 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:17:49,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 7193 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:17:50,683 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:17:50,683 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.22100019455. 2022-02-08 05:17:50,684 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:17:50,684 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:17:51,323 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:17:51,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:17:51,767 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:17:51,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:17:51,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:17:51,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:17:51,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:17:51,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:17:51,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:17:51,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:17:51,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:52,790 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:17:52,790 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.10699987411. 2022-02-08 05:17:52,792 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:17:52,793 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:17:53,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:17:53,892 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:17:54,099 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:17:54,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:17:54,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:17:54,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:17:54,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:17:54,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:17:54,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:17:54,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:17:54,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:56,091 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:17:56,092 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.29999995232. 2022-02-08 05:17:56,092 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:17:56,094 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:17:57,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:17:57,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:17:57,759 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:17:57,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:17:57,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:17:57,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:17:57,834 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:17:57,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:17:57,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:17:57,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:17:57,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:17:59,608 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:17:59,608 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.51699995995. 2022-02-08 05:17:59,608 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:17:59,611 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:17:59,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:18:00,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:18:00,358 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:18:00,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:18:00,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:18:00,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:18:00,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:18:00,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:18:00,484 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:18:00,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:18:00,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:18:01,296 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:18:01,298 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.6890001297. ############################################################ # # RAM FootPrint: 4366.53125 # ############################################################ 2022-02-08 05:18:01,335 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 05:18:01,335 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 86 seconds. Reason: Iteration:14 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 7212 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:19:27,342 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 05:19:27,342 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 05:19:32,345 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:19:33,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:19:33,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:19:33,831 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:19:33,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:19:33,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:19:33,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:19:33,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:19:33,920 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:19:33,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:19:34,053 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:19:34,092 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:19:34,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:19:34,178 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['1471 memcached\n'], []) 2022-02-08 05:19:34,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 05:19:34,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:19:34,259 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['1471 memcached\n'], []) 2022-02-08 05:19:34,259 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:19:34,262 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:19:35,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:19:35,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:19:36,081 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:19:36,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:19:36,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:19:36,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:19:36,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:19:36,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:19:36,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:19:36,265 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:19:36,289 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:19:36,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:19:36,512 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['45802 memcached\n'], []) 2022-02-08 05:19:36,512 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 05:19:36,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:19:36,667 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['45802 memcached\n'], []) 2022-02-08 05:19:36,667 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:19:36,670 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:19:37,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:19:37,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:19:37,848 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:19:37,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:19:37,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:19:37,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:19:38,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:19:38,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:19:38,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:19:38,079 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:19:38,105 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:19:38,107 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:19:38,196 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['32639 memcached\n'], []) 2022-02-08 05:19:38,196 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 05:19:38,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:19:38,293 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['32639 memcached\n'], []) 2022-02-08 05:19:38,295 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:19:38,296 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:19:39,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:19:39,775 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:19:39,872 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:19:39,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:19:39,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:19:39,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:19:39,943 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:19:39,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:19:39,989 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:19:40,015 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:19:40,032 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:19:40,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:19:40,137 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['66073 memcached\n'], []) 2022-02-08 05:19:40,137 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 05:19:40,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:19:40,200 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['66073 memcached\n'], []) 2022-02-08 05:19:40,200 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:19:40,201 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:19:42,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:19:42,826 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:19:43,048 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:19:43,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:19:43,076 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:19:43,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:19:43,130 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:19:43,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:19:43,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:19:43,211 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:19:43,226 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:19:43,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:19:43,299 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['59908 memcached\n'], []) 2022-02-08 05:19:43,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 05:19:43,352 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:19:43,377 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['59908 memcached\n'], []) 2022-02-08 05:19:43,377 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:19:43,380 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:19:44,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:19:45,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:19:45,443 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:19:45,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:19:45,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:19:45,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:19:45,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:19:45,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:19:45,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:19:45,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:19:45,614 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:19:45,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:19:45,680 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['94817 memcached\n'], []) 2022-02-08 05:19:45,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 05:19:45,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:19:45,734 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['94817 memcached\n'], []) 2022-02-08 05:19:45,736 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:19:45,739 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:19:46,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:19:47,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:19:47,414 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:19:47,415 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:19:47,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:19:47,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:19:47,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:19:47,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:19:47,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:19:47,624 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:19:47,638 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:19:47,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:19:47,693 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['28694 memcached\n'], []) 2022-02-08 05:19:47,693 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 05:19:47,724 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:19:47,746 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['28694 memcached\n'], []) 2022-02-08 05:19:47,746 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:19:47,749 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:19:48,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:19:48,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:19:48,627 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:19:48,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:19:48,642 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:19:48,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:19:48,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:19:48,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:19:48,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:19:48,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:19:48,780 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:19:48,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:19:48,838 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['43571 memcached\n'], []) 2022-02-08 05:19:48,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 05:19:48,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:19:48,888 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['43571 memcached\n'], []) 2022-02-08 05:19:48,888 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:19:48,891 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 5 queries submitted, 0 failed, 7236 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:19:49,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:19:50,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:19:50,667 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:19:50,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:19:50,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:19:50,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:19:50,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:19:50,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:19:50,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:19:50,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:19:50,822 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:19:50,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:19:50,881 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['79618 memcached\n'], []) 2022-02-08 05:19:50,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 05:19:50,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:19:50,934 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['79618 memcached\n'], []) 2022-02-08 05:19:50,936 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:19:50,938 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:19:51,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:19:51,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:19:51,934 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:19:51,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:19:51,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:19:51,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:19:51,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:19:52,040 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:19:52,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:19:52,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:19:52,105 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:19:52,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:19:52,167 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['109276 memcached\n'], []) 2022-02-08 05:19:52,168 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 05:19:52,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:19:52,213 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['109276 memcached\n'], []) 2022-02-08 05:19:52,213 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:19:52,216 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:19:52,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:19:52,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:19:52,582 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:19:52,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:19:52,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:19:52,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:19:52,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:19:52,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:19:52,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:19:52,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:19:52,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:19:52,697 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 05:19:52,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:19:52,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:19:56,464 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:19:56,936 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:19:57,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:19:57,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:19:57,492 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 05:19:57,492 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:19:57,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:19:57,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:19:57,572 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 05:19:57,573 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:19:57,576 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:19:57,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:19:57,885 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:19:57,953 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:19:57,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:19:57,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:19:57,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:19:57,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:19:58,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:19:58,035 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:19:58,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:19:58,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:19:58,076 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 05:19:58,104 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:19:58,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:19:58,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:19:59,094 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:19:59,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:19:59,211 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:19:59,226 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 05:19:59,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:19:59,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:19:59,265 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:19:59,279 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 05:19:59,279 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:19:59,282 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:19:59,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:19:59,576 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:19:59,637 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:19:59,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:19:59,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:19:59,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:19:59,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:19:59,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:19:59,706 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:19:59,719 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:19:59,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:19:59,747 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 05:19:59,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:19:59,793 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:04,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:20:05,420 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:05,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:05,964 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:20:05,979 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 05:20:05,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:20:05,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:06,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:20:06,075 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 05:20:06,075 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:20:06,078 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 05:20:06,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:20:06,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 05:20:06,460 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:20:06,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 05:20:06,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 05:20:06,487 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 05:20:06,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 05:20:06,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 05:20:06,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 05:20:06,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 05:20:06,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:20:06,591 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 05:20:06,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:20:06,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:06,650 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:20:06,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:06,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:06,693 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:20:06,707 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 05:20:06,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:20:06,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:06,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:20:06,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:06,783 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:20:06,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:06,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:06,864 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:06,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:06,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:06,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:06,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:06,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:07,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:07,035 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:20:07,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:07,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:07,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:07,099 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:07,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:07,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:07,147 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:07,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:07,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:07,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:20:07,209 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:07,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:07,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:07,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:07,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:07,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:07,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:07,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:07,336 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:07,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:20:07,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:07,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:07,403 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:07,420 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:07,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:07,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:07,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:07,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:07,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:07,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:20:07,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:07,548 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:07,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:07,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:07,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:07,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:07,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:07,650 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:07,665 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 05:20:07,668 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:20:07,744 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:20:07,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:20:08,025 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:20:08,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:20:08,040 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:20:08,053 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:20:08,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:20:08,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:20:08,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:20:08,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:20:08,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:20:08,153 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 05:20:08,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:20:08,196 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:10,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:20:11,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:11,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:12,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:20:12,173 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 05:20:12,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:20:12,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:12,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:20:12,351 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 05:20:12,351 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:20:12,354 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 05:20:12,436 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:20:12,678 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 05:20:12,746 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:20:12,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 05:20:12,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 05:20:12,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 05:20:12,788 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 05:20:12,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 05:20:12,821 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 05:20:12,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 05:20:12,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:20:12,868 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 05:20:12,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:20:12,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:12,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:20:12,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:12,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:13,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:20:13,023 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 05:20:13,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:20:13,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:13,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:20:13,086 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:13,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:20:13,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:13,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:13,201 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:13,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:13,255 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:13,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:13,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:13,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:13,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:13,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:20:13,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:13,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:13,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:13,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:13,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:13,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:13,512 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:13,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:13,553 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:13,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:20:13,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:13,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:13,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:13,637 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:13,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:13,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:13,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:13,706 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:13,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:13,739 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:20:13,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:13,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:13,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:13,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:13,826 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:13,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:13,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:13,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:13,897 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:13,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:20:13,936 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:13,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:13,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:13,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:14,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:14,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:14,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:14,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:14,095 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 05:20:14,096 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:20:14,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:20:14,438 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:20:14,515 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:20:14,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:20:14,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:20:14,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:20:14,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:20:14,579 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:20:14,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:20:14,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:20:14,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:20:14,648 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 05:20:14,678 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:20:14,698 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:18,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:20:18,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:19,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:19,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:20:19,509 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 05:20:19,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:20:19,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:19,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:20:19,586 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 05:20:19,588 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:20:19,589 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:20:19,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:20:20,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:20:20,174 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:20:20,176 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:20:20,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:20:20,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:20:20,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:20:20,405 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:20:20,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:20:20,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:20:20,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:20:20,700 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 05:20:20,727 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:20:20,755 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:24,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:20:25,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:25,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:26,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:20:26,223 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 05:20:26,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:20:26,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:26,346 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:20:26,371 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 05:20:26,371 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:20:26,374 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 05:20:26,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:20:26,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 05:20:26,836 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:20:26,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 05:20:26,857 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 05:20:26,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 05:20:26,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 05:20:26,920 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 05:20:26,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 05:20:26,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 05:20:26,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:20:26,990 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 05:20:27,015 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:20:27,030 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:27,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:20:27,066 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:27,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:27,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:20:27,108 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 05:20:27,109 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:20:27,124 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:27,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:20:27,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:27,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:20:27,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:27,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:27,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:27,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:27,329 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:27,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:27,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:27,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:27,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:27,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:20:27,533 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:27,556 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:27,579 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:27,602 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:27,624 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:27,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:27,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:27,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:27,706 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:27,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:20:27,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:27,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:27,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:27,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:27,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:27,849 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:27,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:27,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:27,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:27,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:20:27,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:27,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:28,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:28,020 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:28,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:28,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:28,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:28,109 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:28,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:28,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:20:28,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:28,201 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:28,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:28,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:28,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:28,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:28,313 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:28,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:28,361 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 05:20:28,364 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 05:20:28,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:20:28,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 05:20:28,811 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:20:28,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 05:20:28,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 05:20:28,846 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 05:20:28,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 05:20:28,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 05:20:28,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 05:20:28,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 05:20:28,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:20:28,947 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 05:20:28,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:20:28,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:29,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:20:29,028 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:29,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:29,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:20:29,078 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 05:20:29,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:20:29,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:29,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:20:29,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:29,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:20:29,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:29,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:29,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:29,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:29,303 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:29,334 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:29,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:29,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:29,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:29,448 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:20:29,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:29,487 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:29,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:29,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:29,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:29,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:29,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:29,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:29,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:29,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:20:29,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:29,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:29,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:29,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:29,724 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:29,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:29,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:29,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:29,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:29,816 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:20:29,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:29,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:29,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:29,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:29,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:29,923 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:29,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:29,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:29,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:29,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:20:30,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:30,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:30,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:30,073 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:30,094 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:30,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:30,132 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:30,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:30,171 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 05:20:30,174 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:20:30,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:20:30,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:20:30,732 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:20:30,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:20:30,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:20:30,798 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:20:30,816 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:20:30,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:20:30,859 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:20:30,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:20:30,926 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:20:30,943 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 05:20:30,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:20:31,017 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:34,910 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:20:35,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:35,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:36,079 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:20:36,138 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 05:20:36,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:20:36,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:36,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:20:36,260 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 05:20:36,260 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:20:36,263 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:20:36,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:20:36,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:20:37,005 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:20:37,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:20:37,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:20:37,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:20:37,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:20:37,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:20:37,140 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:20:37,186 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:20:37,201 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:20:37,229 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 05:20:37,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:20:37,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:40,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:20:41,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:41,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:41,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:20:41,644 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 05:20:41,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:20:41,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:41,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:20:41,756 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 05:20:41,756 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:20:41,757 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:20:42,499 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:20:42,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:20:43,035 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:20:43,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:20:43,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:20:43,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:20:43,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:20:43,114 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:20:43,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:20:43,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:20:43,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:20:43,200 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 05:20:43,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:20:43,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:46,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:20:46,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:46,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:47,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:20:47,183 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 05:20:47,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:20:47,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:47,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:20:47,276 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 05:20:47,276 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:20:47,278 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:20:47,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:20:47,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:20:47,707 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:20:47,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:20:47,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:20:47,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:20:47,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:20:47,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:20:47,806 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:20:47,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:20:47,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:20:47,865 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 05:20:47,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:20:47,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 7260 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:20:52,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:20:52,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:53,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:53,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:20:53,627 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 05:20:53,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:20:53,645 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:53,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:20:53,717 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 05:20:53,719 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:20:53,720 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 05:20:53,811 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:20:54,051 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 05:20:54,117 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:20:54,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 05:20:54,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 05:20:54,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 05:20:54,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 05:20:54,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 05:20:54,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 05:20:54,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 05:20:54,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:20:54,246 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 05:20:54,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:20:54,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:54,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:20:54,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:54,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:20:54,354 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:20:54,368 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 05:20:54,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:20:54,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:20:54,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:20:54,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:54,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:20:54,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:54,522 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:54,553 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:54,584 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:54,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:54,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:54,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:54,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:20:54,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:54,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:20:54,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:54,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:54,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:54,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:54,867 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:54,885 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:54,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:54,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:20:54,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:54,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:20:54,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:55,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:55,020 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:55,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:55,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:55,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:55,099 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:55,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:20:55,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:55,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:20:55,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:55,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:55,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:55,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:55,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:55,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:55,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:55,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:20:55,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:55,329 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:20:55,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:55,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:55,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:55,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:55,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:55,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:55,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:55,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:20:55,502 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 05:20:55,502 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:20:55,503 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:20:55,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:20:55,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:20:55,941 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:20:55,943 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:20:55,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:20:55,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:20:55,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:20:56,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:20:56,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:20:56,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:20:56,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:20:56,579 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:20:56,581 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.07899999619. 2022-02-08 05:20:56,581 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:20:56,584 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:20:57,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:20:57,931 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:20:58,132 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:20:58,132 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:20:58,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:20:58,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:20:58,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:20:58,278 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:20:58,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:20:58,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:20:58,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:20:59,094 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:20:59,094 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:21:01,094 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:01,437 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:21:01,438 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:21:03,438 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:03,601 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:21:03,601 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:21:05,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:05,757 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:21:05,757 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 9.17699980736. 2022-02-08 05:21:05,759 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:21:05,760 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:21:06,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:21:07,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:21:07,298 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:21:07,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:21:07,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:21:07,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:21:07,377 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:21:07,408 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:21:07,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:21:07,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:21:07,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:08,401 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:21:08,401 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.64300012589. 2022-02-08 05:21:08,401 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:21:08,403 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:21:09,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:21:09,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:21:09,711 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:21:09,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:21:09,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:21:09,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:21:09,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:21:09,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:21:09,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:21:09,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:21:09,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:10,500 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:21:10,502 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.10100007057. 2022-02-08 05:21:10,502 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:21:10,503 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:21:11,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:21:11,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:21:11,753 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:21:11,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:21:11,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:21:11,793 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:21:11,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:21:11,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:21:11,867 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:21:11,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:21:11,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:12,548 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:21:12,548 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.04600000381. 2022-02-08 05:21:12,549 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:21:12,549 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:21:12,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:21:13,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:21:13,517 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:21:13,517 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:21:13,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:21:13,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:21:13,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:21:13,591 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:21:13,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:21:13,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:21:13,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:14,548 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:21:14,549 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 05:21:16,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:16,992 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:21:16,992 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 05:21:18,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:19,460 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:21:19,460 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 05:21:21,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:21,937 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:21:21,937 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 05:21:23,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:24,371 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:21:24,371 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 05:21:26,372 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:26,714 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:21:26,716 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 05:21:28,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:29,019 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:21:29,019 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 05:21:31,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:31,345 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:21:31,345 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 05:21:33,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:33,520 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:21:33,522 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 20.9739999771. 2022-02-08 05:21:33,522 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:21:33,525 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:21:34,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:21:34,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:21:34,588 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:21:34,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:21:34,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:21:34,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:21:34,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:21:34,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:21:34,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:21:34,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:21:34,727 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:35,269 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:21:35,270 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.74899983406. 2022-02-08 05:21:35,270 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:21:35,273 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:21:35,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:21:36,209 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:21:36,381 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:21:36,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:21:36,403 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:21:36,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:21:36,446 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:21:36,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:21:36,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:21:36,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:21:36,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:37,236 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:21:37,240 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.96900010109. 2022-02-08 05:21:37,240 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:21:37,243 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:21:37,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:21:38,221 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:21:38,335 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:21:38,336 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:21:38,352 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:21:38,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:21:38,385 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:21:38,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:21:38,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:21:38,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:21:38,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:39,039 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:21:39,039 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.79800009727. 2022-02-08 05:21:39,039 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:21:39,040 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:21:40,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:21:40,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:21:40,861 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:21:40,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:21:40,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:21:41,114 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:21:41,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:21:41,372 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:21:41,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:21:41,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:21:41,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:42,608 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:21:42,608 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 05:21:44,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:21:44,780 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:21:44,782 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.74300003052. ############################################################ # # RAM FootPrint: 4367.55078125 # ############################################################ 2022-02-08 05:21:44,822 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 05:21:44,822 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 67 seconds. Reason: Iteration:15 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 7285 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 7310 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:22:51,831 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 05:22:51,831 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 05:22:56,834 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:22:57,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:22:58,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:22:58,292 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:22:58,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:22:58,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:22:58,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:22:58,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:22:58,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:22:58,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:22:58,693 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:22:58,721 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:22:58,721 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:22:58,835 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['2189 memcached\n'], []) 2022-02-08 05:22:58,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 05:22:58,897 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:22:58,923 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['2189 memcached\n'], []) 2022-02-08 05:22:58,923 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:22:58,924 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:22:59,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:00,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:23:00,286 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:00,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:23:00,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:23:00,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:23:00,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:23:00,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:23:00,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:23:00,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:23:00,490 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:23:00,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:23:00,549 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['46837 memcached\n'], []) 2022-02-08 05:23:00,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 05:23:00,584 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:23:00,612 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['46837 memcached\n'], []) 2022-02-08 05:23:00,614 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:23:00,615 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:23:01,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:01,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:23:02,204 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:02,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:23:02,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:23:02,255 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:23:02,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:23:02,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:23:02,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:23:02,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:23:02,394 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:23:02,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:23:02,463 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['33443 memcached\n'], []) 2022-02-08 05:23:02,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 05:23:02,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:23:02,529 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['33443 memcached\n'], []) 2022-02-08 05:23:02,529 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:23:02,532 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:23:03,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:03,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:23:04,028 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:04,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:23:04,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:23:04,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:23:04,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:23:04,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:23:04,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:23:04,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:23:04,176 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:23:04,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:23:04,259 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['66783 memcached\n'], []) 2022-02-08 05:23:04,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 05:23:04,295 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:23:04,319 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['66783 memcached\n'], []) 2022-02-08 05:23:04,319 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:23:04,321 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:23:05,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:05,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:23:05,877 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:05,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:23:05,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:23:05,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:23:05,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:23:05,982 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:23:06,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:23:06,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:23:06,056 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:23:06,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:23:06,118 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['60621 memcached\n'], []) 2022-02-08 05:23:06,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 05:23:06,160 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:23:06,184 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['60621 memcached\n'], []) 2022-02-08 05:23:06,186 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:23:06,187 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:23:07,354 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:07,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:23:08,049 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:08,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:23:08,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:23:08,086 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:23:08,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:23:08,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:23:08,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:23:08,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:23:08,214 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:23:08,214 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:23:08,279 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['95534 memcached\n'], []) 2022-02-08 05:23:08,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 05:23:08,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:23:08,329 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['95534 memcached\n'], []) 2022-02-08 05:23:08,329 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:23:08,332 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:23:09,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:09,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:23:09,650 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:09,650 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:23:09,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:23:09,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:23:09,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:23:09,734 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:23:09,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:23:09,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:23:09,788 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:23:09,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:23:09,848 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['29414 memcached\n'], []) 2022-02-08 05:23:09,849 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 05:23:09,877 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:23:09,898 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['29414 memcached\n'], []) 2022-02-08 05:23:09,898 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:23:09,901 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:23:10,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:11,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:23:11,380 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:11,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:23:11,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:23:11,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:23:11,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:23:11,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:23:11,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:23:11,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:23:11,551 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:23:11,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:23:11,608 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['44292 memcached\n'], []) 2022-02-08 05:23:11,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 05:23:11,637 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:23:11,661 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['44292 memcached\n'], []) 2022-02-08 05:23:11,661 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:23:11,663 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:23:12,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:13,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:23:13,213 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:13,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:23:13,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:23:13,255 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:23:13,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:23:13,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:23:13,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:23:13,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:23:13,400 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:23:13,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:23:13,456 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['80339 memcached\n'], []) 2022-02-08 05:23:13,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 05:23:13,484 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:23:13,509 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['80339 memcached\n'], []) 2022-02-08 05:23:13,510 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:23:13,513 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:23:14,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:14,438 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:23:14,628 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:14,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:23:14,642 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:23:14,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:23:14,698 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:23:14,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:23:14,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:23:14,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:23:14,766 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:23:14,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:23:14,815 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['109996 memcached\n'], []) 2022-02-08 05:23:14,816 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 05:23:14,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:23:14,861 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['109996 memcached\n'], []) 2022-02-08 05:23:14,861 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:23:14,864 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:23:14,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:15,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:23:15,266 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:15,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:23:15,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:23:15,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:23:15,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:23:15,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:23:15,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:23:15,362 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:23:15,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:23:15,394 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 05:23:15,420 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:23:15,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:18,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:23:19,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:19,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:19,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:23:19,668 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 05:23:19,668 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:23:19,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:19,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:23:19,753 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 05:23:19,755 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:23:19,756 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:23:19,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:20,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:23:20,137 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:20,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:23:20,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:23:20,165 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:23:20,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:23:20,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:23:20,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:23:20,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:23:20,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:23:20,259 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 05:23:20,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:23:20,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:21,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:23:21,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:21,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:21,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:23:21,394 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 05:23:21,395 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:23:21,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:21,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:23:21,456 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 05:23:21,456 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:23:21,457 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:23:21,543 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:21,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:23:21,812 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:21,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:23:21,826 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:23:21,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:23:21,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:23:21,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:23:21,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:23:21,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:23:21,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:23:21,931 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 05:23:21,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:23:21,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:26,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:23:26,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:26,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:27,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:23:27,134 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 05:23:27,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:23:27,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:27,209 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:23:27,220 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 05:23:27,221 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:23:27,223 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 05:23:27,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:27,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 05:23:27,588 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:27,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 05:23:27,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 05:23:27,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 05:23:27,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 05:23:27,647 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 05:23:27,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 05:23:27,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 05:23:27,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:23:27,707 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 05:23:27,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:23:27,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:27,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:23:27,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:27,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:27,798 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:23:27,812 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 05:23:27,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:23:27,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:27,839 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:23:27,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:27,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:23:27,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:27,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:27,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:28,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:28,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:28,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:28,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:28,145 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:28,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:28,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:23:28,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:28,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:28,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:28,260 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:28,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:28,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:28,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:28,334 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:28,349 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:28,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:23:28,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:28,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:28,415 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:28,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:28,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:28,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:28,489 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:28,505 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:28,520 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:28,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:23:28,553 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:28,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:28,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:28,602 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:28,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:28,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:28,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:28,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:28,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:28,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:23:28,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:28,734 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:28,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:28,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:28,788 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:28,806 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:28,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:28,846 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:28,864 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 05:23:28,865 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:23:29,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:29,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:23:29,614 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:29,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:23:29,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:23:29,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:23:29,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:23:29,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:23:29,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:23:29,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:23:29,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:23:29,734 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 05:23:29,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:23:29,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:32,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:23:33,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:33,232 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:33,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:23:33,407 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 05:23:33,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:23:33,423 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:33,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:23:33,484 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 05:23:33,484 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:23:33,486 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 05:23:33,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:33,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 05:23:33,908 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:33,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 05:23:33,926 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 05:23:33,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 05:23:33,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 05:23:33,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 05:23:33,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 05:23:34,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 05:23:34,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:23:34,049 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 05:23:34,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:23:34,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:34,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:23:34,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:34,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:34,158 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:23:34,174 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 05:23:34,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:23:34,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:34,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:23:34,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:34,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:23:34,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:34,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:34,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:34,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:34,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:34,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:34,499 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:34,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:34,563 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:34,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:23:34,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:34,619 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:34,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:34,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:34,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:34,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:34,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:34,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:34,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:34,767 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:23:34,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:34,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:34,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:34,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:34,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:34,877 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:34,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:34,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:34,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:34,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:23:34,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:34,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:35,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:35,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:35,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:35,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:35,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:35,099 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:35,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:35,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:23:35,158 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:35,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:35,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:35,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:35,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:35,255 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:35,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:35,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:35,311 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 05:23:35,312 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:23:35,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:35,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:23:35,648 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:35,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:23:35,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:23:35,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:23:35,686 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:23:35,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:23:35,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:23:35,727 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:23:35,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:23:35,757 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 05:23:35,788 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:23:35,803 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:39,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:23:40,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:40,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:40,744 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:23:40,881 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 05:23:40,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:23:40,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:40,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:23:40,961 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 05:23:40,961 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:23:40,963 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:23:41,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:41,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:23:41,734 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:41,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:23:41,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:23:41,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:23:41,844 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:23:41,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:23:41,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:23:41,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:23:41,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:23:42,000 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 05:23:42,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:23:42,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:46,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:23:46,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:46,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:47,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:23:47,180 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 05:23:47,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:23:47,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:47,265 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:23:47,282 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 05:23:47,282 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:23:47,283 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 05:23:47,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:47,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 05:23:47,671 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:47,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 05:23:47,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 05:23:47,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 05:23:47,719 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 05:23:47,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 05:23:47,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 05:23:47,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 05:23:47,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:23:47,793 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 05:23:47,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:23:47,846 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:47,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:23:47,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:47,890 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:47,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:23:47,920 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 05:23:47,920 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:23:47,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:47,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:23:47,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:47,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:23:48,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:48,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:48,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:48,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:48,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:48,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:48,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:48,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:48,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:48,285 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:23:48,303 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:48,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:48,334 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:48,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:48,364 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:48,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:48,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:48,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:48,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:48,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:23:48,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:48,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:48,489 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:48,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:48,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:48,533 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:48,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:48,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:48,591 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:48,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:23:48,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:48,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:48,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:48,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:48,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:48,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:48,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:48,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:48,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:48,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:23:48,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:48,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:48,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:48,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:48,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:48,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:48,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:48,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:48,930 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 05:23:48,934 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 05:23:49,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:49,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 05:23:49,348 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:49,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 05:23:49,362 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 5 queries submitted, 0 failed, 7334 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:23:49,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 05:23:49,390 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 05:23:49,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 05:23:49,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 05:23:49,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 05:23:49,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:23:49,464 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 05:23:49,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:23:49,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:49,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:23:49,540 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:49,556 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:49,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:23:49,588 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 05:23:49,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:23:49,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:49,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:23:49,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:49,665 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:23:49,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:49,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:49,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:49,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:49,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:49,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:49,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:49,936 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:23:49,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:49,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:23:50,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:50,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:50,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:50,053 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:50,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:50,086 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:50,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:50,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:23:50,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:50,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:23:50,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:50,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:50,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:50,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:50,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:50,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:50,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:50,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:23:50,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:50,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:23:50,349 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:50,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:50,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:50,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:50,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:50,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:50,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:50,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:23:50,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:50,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:23:50,545 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:50,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:50,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:50,599 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:50,619 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:50,635 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:50,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:50,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:23:50,693 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 05:23:50,694 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:23:50,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:51,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:23:51,167 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:51,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:23:51,188 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:23:51,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:23:51,216 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:23:51,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:23:51,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:23:51,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:23:51,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:23:51,289 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 05:23:51,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:23:51,341 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:54,821 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:23:55,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:55,775 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:23:56,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:23:56,065 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 05:23:56,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:23:56,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:23:56,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:23:56,188 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 05:23:56,188 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:23:56,190 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:23:56,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:23:56,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:23:56,622 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:23:56,624 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:23:56,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:23:56,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:23:56,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:23:56,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:23:56,724 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:23:56,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:23:56,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:23:56,786 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 05:23:56,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:23:56,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:24:00,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:24:00,392 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:24:00,635 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:24:00,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:24:00,868 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 05:24:00,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:24:00,882 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:24:00,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:24:00,934 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 05:24:00,934 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:24:00,937 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:24:01,668 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:24:02,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:24:02,348 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:24:02,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:24:02,364 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:24:02,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:24:02,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:24:02,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:24:02,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:24:02,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:24:02,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:24:02,562 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 05:24:02,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:24:02,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:24:05,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:24:06,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:24:06,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:24:06,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:24:06,763 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 05:24:06,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:24:06,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:24:06,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:24:06,839 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 05:24:06,839 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:24:06,842 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:24:06,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:24:07,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:24:07,371 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:24:07,372 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:24:07,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:24:07,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:24:07,477 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:24:07,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:24:07,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:24:07,545 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:24:07,566 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:24:07,588 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 05:24:07,612 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:24:07,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:24:08,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:24:09,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:24:09,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:24:09,905 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:24:09,987 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 05:24:09,989 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:24:10,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:24:10,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:24:10,160 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 05:24:10,161 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:24:10,163 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 05:24:10,313 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:24:10,563 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 05:24:10,638 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:24:10,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 05:24:10,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 05:24:10,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 05:24:10,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 05:24:10,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 05:24:10,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 05:24:10,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 05:24:10,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:24:10,780 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 05:24:10,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:24:10,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:24:10,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:24:10,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:24:10,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:24:10,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:24:10,901 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 05:24:10,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:24:10,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:24:10,933 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:24:10,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:24:10,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:24:11,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:24:11,040 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:24:11,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:24:11,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:24:11,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:24:11,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:24:11,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:24:11,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:24:11,280 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:24:11,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:24:11,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:24:11,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:24:11,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:24:11,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:24:11,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:24:11,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:24:11,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:24:11,446 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:24:11,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:24:11,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:24:11,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:24:11,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:24:11,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:24:11,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:24:11,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:24:11,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:24:11,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:24:11,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:24:11,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:24:11,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:24:11,678 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:24:11,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:24:11,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:24:11,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:24:11,747 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:24:11,767 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:24:11,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:24:11,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:24:11,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:24:11,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:24:11,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:24:11,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:24:11,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:24:11,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:24:11,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:24:11,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:24:11,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:24:12,002 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:24:12,020 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 05:24:12,020 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:24:12,023 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:24:12,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:24:12,367 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:24:12,476 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:24:12,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:24:12,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:24:12,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:24:12,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:24:12,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:24:12,576 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:24:12,591 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:24:12,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:13,095 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:24:13,096 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.0759999752. 2022-02-08 05:24:13,096 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:24:13,098 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:24:13,897 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:24:14,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:24:14,517 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:24:14,517 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:24:14,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:24:14,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:24:14,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:24:14,737 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:24:14,755 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:24:14,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:24:14,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:15,707 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:24:15,707 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:24:17,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:17,913 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:24:17,913 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:24:19,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:20,187 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:24:20,188 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:24:22,188 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:22,395 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:24:22,397 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:24:24,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:24,647 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:24:24,647 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:24:26,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:26,865 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:24:26,865 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:24:28,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:29,269 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:24:29,269 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:24:31,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:31,433 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:24:31,433 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:24:33,436 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:33,618 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:24:33,618 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:24:35,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:35,779 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:24:35,780 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 22.6840000153. 2022-02-08 05:24:35,780 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:24:35,782 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:24:36,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:24:36,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:24:36,970 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:24:36,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:24:36,999 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:24:37,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:24:37,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:24:37,099 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:24:37,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:24:37,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:24:37,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:38,276 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:24:38,278 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 05:24:40,278 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:40,496 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:24:40,496 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.71600008011. 2022-02-08 05:24:40,497 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:24:40,499 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:24:41,262 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:24:41,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:24:41,742 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:24:41,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:24:41,765 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:24:41,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:24:41,813 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:24:41,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:24:41,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:24:41,910 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:24:41,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:42,739 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:24:42,739 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.242000103. 2022-02-08 05:24:42,739 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:24:42,740 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:24:43,553 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:24:43,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:24:44,030 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:24:44,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:24:44,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:24:44,073 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:24:44,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:24:44,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:24:44,145 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:24:44,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:24:44,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:44,910 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:24:44,910 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.17100000381. 2022-02-08 05:24:44,910 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:24:44,913 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:24:45,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:24:46,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:24:46,210 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:24:46,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:24:46,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:24:46,252 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:24:46,275 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:24:46,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:24:46,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:24:46,392 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:24:46,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:47,138 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:24:47,138 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.22899985313. 2022-02-08 05:24:47,138 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:24:47,141 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:24:47,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:24:48,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:24:48,194 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:24:48,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:24:48,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:24:48,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:24:48,249 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:24:48,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:24:48,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:24:48,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:24:48,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:48,851 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:24:48,852 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.71399998665. 2022-02-08 05:24:48,852 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:24:48,855 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:24:49,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 7354 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:24:49,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:24:49,608 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:24:49,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:24:49,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:24:49,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:24:49,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:24:50,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:24:50,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:24:50,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:24:50,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:51,121 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:24:51,122 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 05:24:53,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:53,658 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:24:53,660 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 05:24:55,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:56,190 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:24:56,190 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 05:24:58,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:24:58,674 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:24:58,674 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 05:25:00,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:25:01,279 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:25:01,280 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 05:25:03,280 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:25:03,477 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:25:03,477 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 14.625. 2022-02-08 05:25:03,477 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:25:03,480 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:25:04,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:25:04,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:25:04,480 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:25:04,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:25:04,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:25:04,520 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:25:04,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:25:04,573 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:25:04,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:25:04,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:25:04,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:25:05,217 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:25:05,217 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.73900008202. 2022-02-08 05:25:05,217 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:25:05,219 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:25:05,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:25:05,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:25:06,216 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:25:06,216 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:25:06,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:25:06,520 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:25:06,568 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:25:06,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:25:06,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:25:06,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:25:06,839 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:25:08,359 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:25:08,361 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 05:25:10,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:25:10,658 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:25:10,660 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 05:25:12,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:25:12,842 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:25:12,842 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.625. ############################################################ # # RAM FootPrint: 4365.26953125 # ############################################################ 2022-02-08 05:25:12,875 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 05:25:12,875 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 74 seconds. Reason: Iteration:16 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 7380 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:26:26,881 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 05:26:26,882 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 05:26:31,885 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:26:32,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:26:33,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:26:33,345 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:26:33,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:26:33,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:26:33,487 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:26:33,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:26:33,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:26:33,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:26:33,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:26:33,786 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:26:33,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:26:33,861 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['2845 memcached\n'], []) 2022-02-08 05:26:33,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 05:26:33,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:26:33,917 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['2845 memcached\n'], []) 2022-02-08 05:26:33,917 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:26:33,920 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:26:34,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:26:34,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:26:35,062 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:26:35,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:26:35,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:26:35,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:26:35,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:26:35,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:26:35,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:26:35,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:26:35,239 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:26:35,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:26:35,309 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['47525 memcached\n'], []) 2022-02-08 05:26:35,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 05:26:35,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:26:35,380 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['47525 memcached\n'], []) 2022-02-08 05:26:35,380 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:26:35,381 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:26:36,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:26:37,086 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:26:37,345 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:26:37,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:26:37,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:26:37,403 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:26:37,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:26:37,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:26:37,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:26:37,517 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:26:37,542 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:26:37,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:26:37,608 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['34197 memcached\n'], []) 2022-02-08 05:26:37,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 05:26:37,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:26:37,701 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['34197 memcached\n'], []) 2022-02-08 05:26:37,701 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:26:37,704 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:26:38,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:26:39,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:26:39,289 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:26:39,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:26:39,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:26:39,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:26:39,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:26:39,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:26:39,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:26:39,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:26:39,448 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:26:39,448 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:26:39,515 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['67447 memcached\n'], []) 2022-02-08 05:26:39,515 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 05:26:39,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:26:39,569 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['67447 memcached\n'], []) 2022-02-08 05:26:39,569 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:26:39,572 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:26:40,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:26:41,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:26:41,325 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:26:41,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:26:41,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:26:41,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:26:41,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:26:41,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:26:41,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:26:41,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:26:41,476 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:26:41,477 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:26:41,539 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['61274 memcached\n'], []) 2022-02-08 05:26:41,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 05:26:41,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:26:41,591 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['61274 memcached\n'], []) 2022-02-08 05:26:41,592 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:26:41,594 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:26:42,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:26:43,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:26:43,148 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:26:43,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:26:43,163 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:26:43,252 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:26:43,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:26:43,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:26:43,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:26:43,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:26:43,365 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:26:43,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:26:43,423 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['96282 memcached\n'], []) 2022-02-08 05:26:43,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 05:26:43,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:26:43,483 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['96282 memcached\n'], []) 2022-02-08 05:26:43,483 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:26:43,486 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:26:44,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:26:44,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:26:44,730 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:26:44,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:26:44,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:26:44,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:26:44,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:26:44,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:26:44,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:26:44,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:26:44,869 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:26:44,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:26:44,927 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['30069 memcached\n'], []) 2022-02-08 05:26:44,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 05:26:44,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:26:44,976 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['30069 memcached\n'], []) 2022-02-08 05:26:44,977 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:26:44,980 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:26:45,806 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:26:46,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:26:46,305 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:26:46,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:26:46,336 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:26:46,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:26:46,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:26:46,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:26:46,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:26:46,484 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:26:46,502 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:26:46,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:26:46,565 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['44953 memcached\n'], []) 2022-02-08 05:26:46,566 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 05:26:46,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:26:46,619 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['44953 memcached\n'], []) 2022-02-08 05:26:46,619 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:26:46,622 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:26:47,517 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:26:47,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:26:48,157 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:26:48,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:26:48,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:26:48,191 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:26:48,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:26:48,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:26:48,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:26:48,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:26:48,282 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:26:48,283 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:26:48,335 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['80989 memcached\n'], []) 2022-02-08 05:26:48,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 05:26:48,364 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:26:48,387 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['80989 memcached\n'], []) 2022-02-08 05:26:48,387 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:26:48,390 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:26:49,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 7402 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:26:49,403 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:26:49,549 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:26:49,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:26:49,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:26:49,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:26:49,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:26:49,630 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:26:49,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:26:49,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:26:49,688 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:26:49,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:26:49,743 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['110711 memcached\n'], []) 2022-02-08 05:26:49,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 05:26:49,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:26:49,790 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['110711 memcached\n'], []) 2022-02-08 05:26:49,790 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:26:49,792 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:26:49,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:26:50,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:26:50,170 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:26:50,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:26:50,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:26:50,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:26:50,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:26:50,242 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:26:50,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:26:50,270 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:26:50,285 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:26:50,301 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 05:26:50,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:26:50,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:26:53,926 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:26:54,385 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:26:54,655 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:26:54,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:26:54,936 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 05:26:54,936 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:26:54,948 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:26:55,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:26:55,016 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 05:26:55,016 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:26:55,019 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:26:55,104 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:26:55,334 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:26:55,408 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:26:55,408 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:26:55,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:26:55,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:26:55,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:26:55,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:26:55,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:26:55,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:26:55,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:26:55,526 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 05:26:55,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:26:55,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:26:56,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:26:56,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:26:56,604 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:26:56,670 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:26:56,686 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 05:26:56,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:26:56,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:26:56,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:26:56,744 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 05:26:56,746 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:26:56,747 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:26:56,816 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:26:57,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:26:57,088 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:26:57,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:26:57,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:26:57,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:26:57,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:26:57,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:26:57,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:26:57,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:26:57,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:26:57,198 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 05:26:57,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:26:57,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:01,329 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:27:01,798 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:02,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:02,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:27:02,331 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 05:27:02,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:27:02,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:02,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:27:02,420 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 05:27:02,420 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:27:02,421 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 05:27:02,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:27:02,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 05:27:02,832 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:27:02,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 05:27:02,849 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 05:27:02,864 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 05:27:02,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 05:27:02,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 05:27:02,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 05:27:02,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 05:27:02,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:27:02,961 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 05:27:02,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:27:03,005 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:03,022 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:27:03,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:03,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:03,071 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:27:03,089 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 05:27:03,089 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:27:03,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:03,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:27:03,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:03,176 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:27:03,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:03,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:03,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:03,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:03,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:03,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:03,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:03,433 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:03,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:03,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:27:03,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:03,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:03,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:03,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:03,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:03,591 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:03,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:03,635 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:03,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:03,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:27:03,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:03,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:03,727 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:03,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:03,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:03,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:03,803 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:03,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:03,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:03,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:27:03,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:03,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:03,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:03,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:03,943 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:03,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:03,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:03,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:04,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:04,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:27:04,048 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:04,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:04,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:04,099 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:04,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:04,135 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:04,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:04,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:04,188 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 05:27:04,190 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:27:04,415 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:27:04,767 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:27:04,832 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:27:04,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:27:04,849 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:27:04,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:27:04,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:27:04,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:27:04,982 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:27:04,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:27:05,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:27:05,130 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 05:27:05,158 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:27:05,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:08,252 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:27:09,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:09,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:09,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:27:09,658 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 05:27:09,658 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:27:09,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:09,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:27:09,786 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 05:27:09,786 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:27:09,789 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 05:27:09,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:27:10,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 05:27:10,204 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:27:10,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 05:27:10,221 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 05:27:10,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 05:27:10,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 05:27:10,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 05:27:10,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 05:27:10,308 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 05:27:10,323 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:27:10,342 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 05:27:10,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:27:10,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:10,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:27:10,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:10,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:10,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:27:10,456 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 05:27:10,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:27:10,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:10,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:27:10,505 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:10,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:27:10,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:10,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:10,635 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:10,665 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:10,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:10,734 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:10,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:10,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:10,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:10,857 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:27:10,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:10,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:10,910 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:10,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:10,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:10,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:10,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:10,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:11,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:11,030 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:27:11,051 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:11,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:11,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:11,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:11,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:11,135 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:11,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:11,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:11,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:11,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:27:11,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:11,252 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:11,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:11,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:11,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:11,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:11,341 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:11,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:11,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:11,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:27:11,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:11,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:11,464 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:11,484 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:11,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:11,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:11,545 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:11,566 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:11,588 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 05:27:11,592 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:27:11,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:27:11,923 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:27:11,977 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:27:11,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:27:11,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:27:12,002 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:27:12,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:27:12,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:27:12,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:27:12,053 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:27:12,066 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:27:12,084 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 05:27:12,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:27:12,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:16,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:27:16,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:17,124 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:17,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:27:17,398 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 05:27:17,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:27:17,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:17,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:27:17,476 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 05:27:17,476 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:27:17,477 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:27:17,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:27:17,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:27:17,920 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:27:17,920 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:27:17,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:27:17,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:27:17,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:27:18,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:27:18,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:27:18,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:27:18,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:27:18,150 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 05:27:18,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:27:18,191 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:22,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:27:23,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:23,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:24,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:27:24,309 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 05:27:24,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:27:24,326 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:24,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:27:24,417 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 05:27:24,417 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:27:24,421 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 05:27:24,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:27:24,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 05:27:24,836 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:27:24,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 05:27:24,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 05:27:24,867 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 05:27:24,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 05:27:24,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 05:27:24,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 05:27:24,933 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 05:27:24,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:27:24,973 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 05:27:24,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:27:25,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:25,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:27:25,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:25,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:25,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:27:25,099 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 05:27:25,099 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:27:25,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:25,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:27:25,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:25,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:27:25,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:25,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:25,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:25,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:25,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:25,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:25,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:25,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:25,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:25,533 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:27:25,553 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:25,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:25,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:25,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:25,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:25,650 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:25,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:25,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:25,711 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:25,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:27:25,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:25,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:25,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:25,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:25,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:25,854 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:25,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:25,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:25,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:25,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:27:25,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:25,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:25,989 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:26,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:26,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:26,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:26,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:26,091 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:26,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:26,135 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:27:26,160 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:26,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:26,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:26,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:26,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:26,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:26,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:26,321 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:26,342 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 05:27:26,345 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 05:27:26,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:27:26,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 05:27:26,769 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:27:26,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 05:27:26,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 05:27:26,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 05:27:26,813 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 05:27:26,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 05:27:26,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 05:27:26,862 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 05:27:26,880 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:27:26,898 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 05:27:26,926 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:27:26,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:26,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:27:26,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:26,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:27,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:27:27,023 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 05:27:27,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:27:27,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:27,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:27:27,071 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:27,107 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:27:27,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:27,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:27,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:27,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:27,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:27,313 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:27,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:27,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:27,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:27,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:27:27,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:27,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:27,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:27,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:27,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:27,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:27,566 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:27,586 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:27,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:27,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:27:27,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:27,676 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:27,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:27,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:27,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:27,752 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:27,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:27,790 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:27,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:27,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:27:27,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:27,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:27,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:27,900 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:27,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:27,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:27,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:27,971 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:27,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:28,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:27:28,028 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:28,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:28,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:28,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:28,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:28,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:28,145 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:28,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:28,181 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 05:27:28,184 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:27:28,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:27:28,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:27:28,631 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:27:28,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:27:28,658 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:27:28,676 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:27:28,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:27:28,767 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:27:28,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:27:28,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:27:28,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:27:28,844 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 05:27:28,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:27:28,885 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:32,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:27:33,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:33,691 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:33,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:27:33,993 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 05:27:33,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:27:34,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:34,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:27:34,134 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 05:27:34,135 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:27:34,138 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:27:34,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:27:34,487 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:27:34,558 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:27:34,558 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:27:34,576 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:27:34,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:27:34,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:27:34,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:27:34,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:27:34,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:27:34,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:27:34,819 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 05:27:34,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:27:34,936 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:37,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:27:38,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:38,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:38,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:27:38,519 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 05:27:38,520 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:27:38,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:38,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:27:38,594 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 05:27:38,595 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:27:38,596 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:27:39,295 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:27:39,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:27:39,778 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:27:39,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:27:39,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:27:39,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:27:39,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:27:39,867 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:27:39,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:27:39,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:27:39,946 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:27:39,964 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 05:27:39,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:27:40,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:43,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:27:43,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:43,665 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:43,882 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:27:43,898 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 05:27:43,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:27:43,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:43,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:27:43,979 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 05:27:43,979 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:27:43,982 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:27:44,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:27:44,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:27:44,433 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:27:44,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:27:44,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:27:44,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:27:44,492 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:27:44,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:27:44,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:27:44,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:27:44,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:27:44,681 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 05:27:44,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:27:44,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 7427 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:27:49,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:27:50,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:50,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:50,719 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:27:50,746 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 05:27:50,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:27:50,770 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:50,821 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:27:50,851 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 05:27:50,851 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:27:50,854 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 05:27:50,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:27:51,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 05:27:51,275 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:27:51,275 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 05:27:51,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 05:27:51,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 05:27:51,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 05:27:51,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 05:27:51,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 05:27:51,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 05:27:51,385 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:27:51,404 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 05:27:51,431 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:27:51,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:51,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:27:51,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:51,502 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:27:51,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:27:51,532 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 05:27:51,533 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:27:51,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:27:51,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:27:51,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:51,619 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:27:51,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:51,691 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:51,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:51,767 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:51,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:51,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:51,880 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:51,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:27:51,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:51,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:27:52,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:52,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:52,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:52,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:52,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:52,109 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:52,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:52,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:27:52,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:52,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:27:52,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:52,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:52,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:52,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:52,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:52,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:52,308 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:52,326 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:27:52,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:52,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:27:52,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:52,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:52,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:52,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:52,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:52,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:52,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:52,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:27:52,522 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:52,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:27:52,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:52,579 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:52,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:52,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:52,635 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:52,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:52,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:52,691 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:27:52,710 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 05:27:52,710 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:27:52,713 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:27:52,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:27:53,051 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:27:53,128 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:27:53,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:27:53,163 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:27:53,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:27:53,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:27:53,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:27:53,252 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:27:53,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:27:53,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:27:53,885 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:27:53,885 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.17499995232. 2022-02-08 05:27:53,887 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:27:53,888 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:27:54,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:27:55,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:27:55,414 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:27:55,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:27:55,431 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:27:55,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:27:55,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:27:55,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:27:55,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:27:55,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:27:55,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:27:56,509 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:27:56,509 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:27:58,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:27:58,753 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:27:58,755 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:28:00,755 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:00,967 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:00,967 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:28:02,969 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:03,160 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:03,161 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:28:05,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:05,332 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:05,332 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:28:07,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:07,532 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:07,532 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:28:09,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:09,895 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:09,895 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:28:11,897 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:12,111 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:12,112 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:28:14,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:14,292 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:14,293 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:28:16,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:16,493 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:16,494 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:28:18,494 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:18,664 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:28:18,664 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 24.7780001163. 2022-02-08 05:28:18,664 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:28:18,665 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:28:19,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:28:19,767 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:28:19,913 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:28:19,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:28:19,943 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:28:19,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:28:20,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:28:20,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:28:20,076 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:28:20,160 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:28:20,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:21,091 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:21,091 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 05:28:23,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:23,269 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:28:23,269 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.60599994659. 2022-02-08 05:28:23,269 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:28:23,272 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:28:24,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:28:25,002 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:28:25,155 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:28:25,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:28:25,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:28:25,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:28:25,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:28:25,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:28:25,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:28:25,308 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:28:25,326 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:25,951 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:28:25,951 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.6819999218. 2022-02-08 05:28:25,953 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:28:25,954 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:28:26,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:28:27,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:28:27,378 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:28:27,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:28:27,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:28:27,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:28:27,446 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:28:27,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:28:27,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:28:27,520 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:28:27,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:28,176 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:28,176 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 05:28:30,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:30,430 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:28:30,430 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.47799992561. 2022-02-08 05:28:30,430 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:28:30,433 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:28:30,619 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:28:30,943 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:28:31,088 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:28:31,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:28:31,132 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:28:31,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:28:31,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:28:31,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:28:31,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:28:31,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:28:31,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:32,009 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:28:32,009 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.57800006866. 2022-02-08 05:28:32,009 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:28:32,010 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:28:32,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:28:32,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:28:33,016 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:28:33,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:28:33,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:28:33,053 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:28:33,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:28:33,086 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:28:33,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:28:33,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:28:33,147 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:33,673 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:28:33,673 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.66300010681. 2022-02-08 05:28:33,673 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:28:33,674 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:28:33,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:28:34,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:28:34,411 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:28:34,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:28:34,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:28:34,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:28:34,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:28:34,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:28:34,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:28:34,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:28:34,579 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:35,176 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:28:35,177 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.5039999485. 2022-02-08 05:28:35,177 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:28:35,178 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:28:35,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:28:36,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:28:36,280 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:28:36,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:28:36,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:28:36,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:28:36,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:28:36,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:28:36,372 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:28:36,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:28:36,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:36,974 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:28:36,974 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.79799985886. 2022-02-08 05:28:36,974 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:28:36,976 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:28:37,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:28:37,839 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:28:38,461 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:28:38,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:28:38,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:28:38,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:28:38,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:28:38,785 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:28:38,885 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:28:38,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:28:39,043 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:39,805 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:39,805 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 05:28:41,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:42,084 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:42,084 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 05:28:44,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:44,342 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:44,344 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 05:28:46,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:46,793 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:46,795 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 05:28:48,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:49,187 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:49,187 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 5 queries submitted, 0 failed, 7453 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:28:51,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:51,796 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:51,796 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 05:28:53,798 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:54,437 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:54,437 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 05:28:56,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:56,829 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:56,831 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 05:28:58,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:28:59,072 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:28:59,073 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 05:29:01,073 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:29:01,242 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:29:01,243 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 24.2670001984. ############################################################ # # RAM FootPrint: 4362.015625 # ############################################################ 2022-02-08 05:29:01,276 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 05:29:01,276 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 107 seconds. Reason: Iteration:17 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 7477 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:30:48,282 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 05:30:48,282 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 5 queries submitted, 0 failed, 7496 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:30:53,285 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:30:54,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:30:54,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:30:55,055 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:30:55,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:30:55,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:30:55,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:30:55,114 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:30:55,135 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:30:55,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:30:55,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:30:55,214 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:30:55,214 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:30:55,309 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['3554 memcached\n'], []) 2022-02-08 05:30:55,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 05:30:55,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:30:55,380 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['3554 memcached\n'], []) 2022-02-08 05:30:55,380 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:30:55,381 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:30:56,385 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:30:56,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:30:57,194 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:30:57,196 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:30:57,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:30:57,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:30:57,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:30:57,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:30:57,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:30:57,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:30:57,380 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:30:57,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:30:57,469 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['48341 memcached\n'], []) 2022-02-08 05:30:57,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 05:30:57,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:30:57,546 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['48341 memcached\n'], []) 2022-02-08 05:30:57,546 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:30:57,549 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:30:58,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:30:58,507 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:30:58,642 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:30:58,642 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:30:58,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:30:58,701 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:30:58,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:30:58,744 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:30:58,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:30:58,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:30:58,895 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:30:58,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:30:59,016 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['35011 memcached\n'], []) 2022-02-08 05:30:59,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 05:30:59,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:30:59,089 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['35011 memcached\n'], []) 2022-02-08 05:30:59,091 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:30:59,092 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:30:59,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:00,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:31:00,938 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:00,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:31:00,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:31:00,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:31:01,002 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:31:01,020 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:31:01,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:31:01,066 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:31:01,085 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:31:01,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:31:01,184 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['68154 memcached\n'], []) 2022-02-08 05:31:01,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 05:31:01,216 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:31:01,243 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['68154 memcached\n'], []) 2022-02-08 05:31:01,243 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:31:01,246 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:31:02,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:02,931 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:31:03,117 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:03,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:31:03,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:31:03,160 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:31:03,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:31:03,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:31:03,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:31:03,249 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:31:03,286 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:31:03,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:31:03,349 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['61982 memcached\n'], []) 2022-02-08 05:31:03,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 05:31:03,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:31:03,405 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['61982 memcached\n'], []) 2022-02-08 05:31:03,407 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:31:03,407 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:31:04,533 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:05,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:31:05,299 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:05,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:31:05,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:31:05,336 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:31:05,362 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:31:05,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:31:05,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:31:05,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:31:05,448 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:31:05,448 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:31:05,523 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['96987 memcached\n'], []) 2022-02-08 05:31:05,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 05:31:05,553 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:31:05,575 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['96987 memcached\n'], []) 2022-02-08 05:31:05,575 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:31:05,578 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:31:06,392 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:06,829 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:31:07,049 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:07,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:31:07,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:31:07,091 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:31:07,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:31:07,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:31:07,137 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:31:07,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:31:07,177 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:31:07,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:31:07,232 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['30773 memcached\n'], []) 2022-02-08 05:31:07,232 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 05:31:07,260 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:31:07,282 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['30773 memcached\n'], []) 2022-02-08 05:31:07,282 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:31:07,285 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:31:08,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:08,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:31:08,437 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:08,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:31:08,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:31:08,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:31:08,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:31:08,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:31:08,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:31:08,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:31:08,569 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:31:08,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:31:08,624 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['45721 memcached\n'], []) 2022-02-08 05:31:08,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 05:31:08,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:31:08,674 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['45721 memcached\n'], []) 2022-02-08 05:31:08,674 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:31:08,677 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:31:09,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:10,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:31:10,450 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:10,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:31:10,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:31:10,487 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:31:10,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:31:10,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:31:10,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:31:10,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:31:10,596 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:31:10,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:31:10,644 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['81694 memcached\n'], []) 2022-02-08 05:31:10,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 05:31:10,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:31:10,693 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['81694 memcached\n'], []) 2022-02-08 05:31:10,694 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:31:10,697 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:31:11,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:11,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:31:12,013 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:12,015 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:31:12,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:31:12,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:31:12,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:31:12,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:31:12,089 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:31:12,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:31:12,144 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:31:12,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:31:12,233 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['111411 memcached\n'], []) 2022-02-08 05:31:12,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 05:31:12,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:31:12,282 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['111411 memcached\n'], []) 2022-02-08 05:31:12,282 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:31:12,285 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:31:12,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:12,584 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:31:12,661 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:12,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:31:12,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:31:12,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:31:12,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:31:12,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:31:12,752 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:31:12,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:31:12,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:31:12,799 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 05:31:12,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:31:12,839 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:17,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:31:17,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:17,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:18,165 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:31:18,190 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 05:31:18,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:31:18,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:18,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:31:18,278 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 05:31:18,279 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:31:18,280 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:31:18,364 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:18,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:31:18,668 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:18,670 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:31:18,681 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:31:18,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:31:18,706 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:31:18,721 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:31:18,734 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:31:18,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:31:18,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:31:18,779 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 05:31:18,803 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:31:18,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:19,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:31:19,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:19,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:19,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:31:20,005 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 05:31:20,005 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:31:20,017 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:20,040 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:31:20,053 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 05:31:20,053 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:31:20,055 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:31:20,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:20,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:31:20,398 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:20,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:31:20,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:31:20,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:31:20,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:31:20,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:31:20,464 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:31:20,477 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:31:20,492 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:31:20,507 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 05:31:20,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:31:20,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:24,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:31:25,377 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:25,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:25,910 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:31:25,926 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 05:31:25,926 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:31:25,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:25,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:31:26,009 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 05:31:26,010 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:31:26,013 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 05:31:26,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:26,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 05:31:26,380 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:26,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 05:31:26,392 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 05:31:26,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 05:31:26,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 05:31:26,431 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 05:31:26,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 05:31:26,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 05:31:26,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:31:26,484 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 05:31:26,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:31:26,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:26,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:31:26,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:26,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:26,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:31:26,605 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 05:31:26,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:31:26,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:26,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:31:26,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:26,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:31:26,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:26,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:26,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:26,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:26,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:26,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:26,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:26,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:26,974 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:26,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:31:27,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:27,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:27,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:27,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:27,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:27,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:27,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:27,124 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:27,140 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:27,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:31:27,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:27,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:27,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:27,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:27,232 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:27,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:27,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:27,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:27,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:27,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:31:27,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:27,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:27,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:27,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:27,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:27,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:27,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:27,441 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:27,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:27,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:31:27,487 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:27,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:27,520 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:27,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:27,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:27,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:27,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:27,604 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:27,619 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 05:31:27,621 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:31:27,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:27,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:31:27,986 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:27,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:31:28,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:31:28,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:31:28,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:31:28,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:31:28,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:31:28,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:31:28,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:31:28,104 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 05:31:28,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:31:28,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:31,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:31:31,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:31,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:31,933 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:31:32,051 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 05:31:32,051 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:31:32,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:32,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:31:32,384 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 05:31:32,384 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:31:32,387 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 05:31:32,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:32,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 05:31:32,779 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:32,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 05:31:32,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 05:31:32,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 05:31:32,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 05:31:32,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 05:31:32,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 05:31:32,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 05:31:32,892 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:31:32,908 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 05:31:32,933 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:31:32,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:32,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:31:32,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:32,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:33,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:31:33,020 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 05:31:33,022 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:31:33,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:33,051 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:31:33,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:33,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:31:33,137 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:33,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:33,209 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:33,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:33,275 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:33,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:33,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:33,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:33,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:33,431 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:31:33,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:33,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:33,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:33,499 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:33,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:33,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:33,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:33,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:33,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:33,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:31:33,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:33,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:33,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:33,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:33,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:33,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:33,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:33,734 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:33,752 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:33,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:31:33,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:33,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:33,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:33,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:33,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:33,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:33,926 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:33,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:33,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:33,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:31:33,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:34,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:34,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:34,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:34,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:34,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:34,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:34,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:34,145 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 05:31:34,148 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:31:34,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:34,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:31:34,492 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:34,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:31:34,507 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:31:34,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:31:34,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:31:34,548 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:31:34,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:31:34,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:31:34,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:31:34,599 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 05:31:34,622 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:31:34,637 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:38,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:31:38,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:39,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:39,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:31:39,539 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 05:31:39,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:31:39,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:39,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:31:39,619 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 05:31:39,619 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:31:39,622 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:31:39,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:39,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:31:40,122 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:40,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:31:40,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:31:40,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:31:40,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:31:40,190 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:31:40,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:31:40,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:31:40,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:31:40,266 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 05:31:40,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:31:40,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:44,579 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:31:45,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:45,405 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:45,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:31:45,680 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 05:31:45,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:31:45,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:45,783 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:31:45,808 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 05:31:45,809 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:31:45,811 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 05:31:45,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:46,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 05:31:46,233 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:46,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 05:31:46,252 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 05:31:46,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 05:31:46,283 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 05:31:46,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 05:31:46,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 05:31:46,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 05:31:46,357 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:31:46,377 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 05:31:46,405 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:31:46,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:46,441 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:31:46,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:46,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:46,499 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:31:46,516 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 05:31:46,517 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:31:46,533 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:46,553 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:31:46,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:46,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:31:46,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:46,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:46,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:46,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:46,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:46,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:46,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:46,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:46,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:47,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:31:47,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:47,048 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:47,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:47,091 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:47,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:47,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:47,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:47,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:47,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:47,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:31:47,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:47,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:47,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:47,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:47,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:47,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:47,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:47,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:47,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:47,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:31:47,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:47,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:47,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:47,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:47,545 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:47,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:47,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:47,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:47,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:47,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:31:47,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:47,698 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:47,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:47,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:47,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:47,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:47,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:47,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:47,849 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 05:31:47,851 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 05:31:47,946 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:48,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 05:31:48,253 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:48,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 05:31:48,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 05:31:48,283 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 05:31:48,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 05:31:48,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 05:31:48,334 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 05:31:48,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 05:31:48,364 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:31:48,381 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 05:31:48,405 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:31:48,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:48,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:31:48,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:48,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:48,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:31:48,506 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 05:31:48,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:31:48,522 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:48,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:31:48,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:48,586 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:31:48,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:48,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:48,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:48,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:48,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:48,806 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:48,844 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:48,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:31:48,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:48,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:31:48,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:48,982 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:49,012 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:49,035 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:49,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:49,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:49,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:49,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:31:49,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:49,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:31:49,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:49,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:49,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:49,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:49,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:49,265 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:49,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:49,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:31:49,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:49,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:31:49,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:49,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 5 queries submitted, 0 failed, 7520 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:31:49,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:49,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:49,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:49,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:49,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:49,477 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:31:49,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:49,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:31:49,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:49,561 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:49,576 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:49,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:49,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:49,632 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:49,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:49,668 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:31:49,687 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 05:31:49,688 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:31:49,821 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:50,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:31:50,207 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:50,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:31:50,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:31:50,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:31:50,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:31:50,341 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:31:50,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:31:50,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:31:50,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:31:50,421 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 05:31:50,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:31:50,464 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:54,357 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:31:54,862 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:55,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:31:55,464 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:31:55,487 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 05:31:55,489 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:31:55,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:55,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:31:55,628 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 05:31:55,628 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:31:55,630 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:31:55,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:31:55,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:31:56,055 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:31:56,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:31:56,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:31:56,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:31:56,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:31:56,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:31:56,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:31:56,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:31:56,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:31:56,177 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 05:31:56,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:31:56,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:31:59,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:31:59,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:32:00,073 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:32:00,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:32:00,299 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 05:32:00,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:32:00,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:32:00,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:32:00,377 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 05:32:00,378 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:32:01,236 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:32:02,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:32:02,599 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:32:02,769 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:32:02,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:32:02,785 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:32:02,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:32:02,839 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:32:02,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:32:02,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:32:02,918 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:32:02,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:32:02,954 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 05:32:02,982 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:32:03,005 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:32:06,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:32:06,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:32:06,834 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:32:07,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:32:07,040 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 05:32:07,040 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:32:07,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:32:07,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:32:07,109 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 05:32:07,111 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:32:07,111 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:32:07,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:32:07,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:32:07,732 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:32:07,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:32:07,747 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:32:07,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:32:07,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:32:07,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:32:07,821 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:32:07,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:32:07,857 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:32:07,878 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 05:32:07,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:32:07,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:32:12,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:32:13,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:32:13,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:32:13,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:32:13,947 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 05:32:13,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:32:13,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:32:14,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:32:14,085 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 05:32:14,085 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:32:14,203 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 05:32:14,280 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:32:14,522 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 05:32:14,588 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:32:14,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 05:32:14,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 05:32:14,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 05:32:14,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 05:32:14,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 05:32:14,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 05:32:14,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 05:32:14,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:32:14,713 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 05:32:15,484 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:32:15,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:32:15,520 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:32:15,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:32:15,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:32:15,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:32:15,582 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 05:32:15,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:32:15,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:32:15,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:32:15,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:32:15,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:32:15,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:32:15,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:32:15,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:32:15,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:32:15,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:32:15,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:32:15,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:32:15,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:32:16,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:32:16,051 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:32:16,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:32:16,089 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:32:16,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:32:16,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:32:16,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:32:16,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:32:16,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:32:16,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:32:16,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:32:16,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:32:16,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:32:16,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:32:16,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:32:16,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:32:16,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:32:16,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:32:16,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:32:16,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:32:16,392 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:32:16,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:32:16,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:32:16,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:32:16,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:32:16,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:32:16,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:32:16,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:32:16,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:32:16,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:32:16,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:32:16,586 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:32:16,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:32:16,622 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:32:16,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:32:16,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:32:16,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:32:16,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:32:16,719 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:32:16,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:32:16,755 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 05:32:16,756 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:32:16,757 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:32:16,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:32:17,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:32:17,210 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:32:17,211 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:32:17,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:32:17,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:32:17,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:32:17,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:32:17,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:32:17,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:32:17,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:17,846 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:32:17,848 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.09200000763. 2022-02-08 05:32:17,848 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:32:17,849 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:32:18,604 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:32:18,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:32:19,065 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:32:19,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:32:19,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:32:19,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:32:19,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:32:19,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:32:19,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:32:19,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:32:19,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:20,348 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:32:20,348 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:32:22,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:22,542 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:32:22,542 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:32:24,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:24,752 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:32:24,752 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:32:26,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:26,959 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:32:26,959 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:32:28,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:29,148 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:32:29,148 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:32:31,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:31,354 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:32:31,355 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:32:33,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:33,549 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:32:33,549 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:32:35,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:35,904 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:32:35,904 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:32:37,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:38,164 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:32:38,164 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:32:40,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:40,381 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:32:40,381 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:32:42,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:42,555 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:32:42,556 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:32:44,556 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:44,734 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:32:44,734 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:32:46,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:46,924 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:32:46,926 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:32:48,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:49,144 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:32:49,144 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 31.2969999313. 2022-02-08 05:32:49,144 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:32:49,147 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 5 queries submitted, 0 failed, 7547 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:32:50,132 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:32:50,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:32:50,670 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:32:50,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:32:50,693 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:32:50,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:32:50,744 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:32:50,775 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:32:50,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:32:50,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:32:50,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:51,525 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:32:51,526 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.38100004196. 2022-02-08 05:32:51,526 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:32:51,526 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:32:52,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:32:52,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:32:53,007 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:32:53,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:32:53,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:32:53,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:32:53,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:32:53,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:32:53,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:32:53,260 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:32:53,283 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:54,131 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:32:54,131 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 05:32:56,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:56,338 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:32:56,338 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 05:32:58,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:32:58,536 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:32:58,536 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.01099991798. 2022-02-08 05:32:58,536 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:32:58,539 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:32:59,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:32:59,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:32:59,812 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:32:59,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:32:59,839 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:32:59,871 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:32:59,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:32:59,933 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:32:59,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:33:00,005 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:33:00,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:33:00,647 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:33:00,647 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.11000013351. 2022-02-08 05:33:00,648 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:33:00,648 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:33:01,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:33:01,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:33:01,967 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:33:01,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:33:01,989 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:33:02,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:33:02,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:33:02,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:33:02,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:33:02,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:33:02,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:33:02,709 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:33:02,709 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.06100010872. 2022-02-08 05:33:02,709 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:33:02,710 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:33:03,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:33:03,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:33:03,789 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:33:03,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:33:03,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:33:03,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:33:03,859 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:33:03,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:33:03,892 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:33:03,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:33:03,941 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:33:04,542 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:33:04,542 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.83399987221. 2022-02-08 05:33:04,542 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:33:04,545 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:33:05,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:33:05,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:33:05,746 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:33:05,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:33:05,767 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:33:05,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:33:05,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:33:05,854 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:33:05,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:33:05,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:33:05,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:33:06,503 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:33:06,505 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.96200013161. 2022-02-08 05:33:06,505 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:33:06,506 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:33:07,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:33:07,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:33:07,532 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:33:07,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:33:07,548 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:33:07,563 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:33:07,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:33:07,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:33:07,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:33:07,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:33:07,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:33:08,194 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:33:08,194 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.6899998188. 2022-02-08 05:33:08,194 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:33:08,197 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:33:09,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:33:09,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:33:10,102 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:33:10,102 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:33:10,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:33:10,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:33:10,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:33:10,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:33:10,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:33:10,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:33:11,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:33:12,161 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:33:12,161 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 05:33:14,163 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:33:14,581 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:33:14,581 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.3860001564. ############################################################ # # RAM FootPrint: 4366.39453125 # ############################################################ 2022-02-08 05:33:15,101 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 05:33:15,101 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 61 seconds. Reason: Iteration:18 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 7573 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:34:16,105 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 05:34:16,105 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 05:34:21,108 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:34:21,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:34:22,642 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:34:22,943 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:34:22,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:34:22,964 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:34:22,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:34:23,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:34:23,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:34:23,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:34:23,295 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:34:23,349 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:34:23,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:34:23,433 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['4294 memcached\n'], []) 2022-02-08 05:34:23,433 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 05:34:23,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:34:23,486 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['4294 memcached\n'], []) 2022-02-08 05:34:23,486 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:34:23,490 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:34:23,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:34:24,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:34:24,638 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:34:24,640 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:34:24,678 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:34:24,711 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:34:24,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:34:24,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:34:24,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:34:24,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:34:24,914 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:34:24,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:34:24,993 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['49203 memcached\n'], []) 2022-02-08 05:34:24,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 05:34:25,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:34:25,059 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['49203 memcached\n'], []) 2022-02-08 05:34:25,061 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:34:25,062 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:34:25,813 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:34:26,372 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:34:26,578 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:34:26,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:34:26,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:34:26,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:34:26,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:34:26,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:34:26,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:34:26,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:34:26,762 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:34:26,762 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:34:26,835 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['35866 memcached\n'], []) 2022-02-08 05:34:26,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 05:34:26,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:34:26,927 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['35866 memcached\n'], []) 2022-02-08 05:34:26,927 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:34:26,930 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:34:27,813 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:34:28,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:34:28,384 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:34:28,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:34:28,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:34:28,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:34:28,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:34:28,499 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:34:28,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:34:28,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:34:28,572 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:34:28,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:34:28,637 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['68896 memcached\n'], []) 2022-02-08 05:34:28,637 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 05:34:28,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:34:28,686 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['68896 memcached\n'], []) 2022-02-08 05:34:28,686 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:34:28,687 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:34:29,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:34:30,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:34:30,197 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:34:30,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:34:30,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:34:30,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:34:30,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:34:30,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:34:30,311 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:34:30,346 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:34:30,364 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:34:30,364 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:34:30,427 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['62738 memcached\n'], []) 2022-02-08 05:34:30,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 05:34:30,454 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:34:30,477 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['62738 memcached\n'], []) 2022-02-08 05:34:30,479 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:34:30,480 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:34:31,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:34:31,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:34:32,023 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:34:32,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:34:32,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:34:32,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:34:32,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:34:32,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:34:32,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:34:32,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:34:32,186 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:34:32,186 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:34:32,250 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['97729 memcached\n'], []) 2022-02-08 05:34:32,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 05:34:32,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:34:32,308 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['97729 memcached\n'], []) 2022-02-08 05:34:32,308 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:34:32,309 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:34:33,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:34:33,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:34:33,519 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:34:33,520 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:34:33,535 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:34:33,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:34:33,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:34:33,586 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:34:33,599 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:34:33,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:34:33,661 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:34:33,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:34:33,720 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['31523 memcached\n'], []) 2022-02-08 05:34:33,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 05:34:33,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:34:33,766 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['31523 memcached\n'], []) 2022-02-08 05:34:33,767 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:34:33,769 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:34:34,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:34:34,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:34:34,664 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:34:34,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:34:34,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:34:34,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:34:34,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:34:34,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:34:34,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:34:34,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:34:34,776 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:34:34,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:34:34,842 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['46464 memcached\n'], []) 2022-02-08 05:34:34,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 05:34:34,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:34:34,892 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['46464 memcached\n'], []) 2022-02-08 05:34:34,894 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:34:34,895 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:34:35,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:34:36,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:34:36,165 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:34:36,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:34:36,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:34:36,201 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:34:36,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:34:36,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:34:36,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:34:36,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:34:36,328 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:34:36,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:34:36,381 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['82438 memcached\n'], []) 2022-02-08 05:34:36,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 05:34:36,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:34:36,430 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['82438 memcached\n'], []) 2022-02-08 05:34:36,430 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:34:36,431 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:34:36,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:34:37,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:34:37,282 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:34:37,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:34:37,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:34:37,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:34:37,341 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:34:37,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:34:37,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:34:37,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:34:37,418 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:34:37,420 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:34:37,479 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['112268 memcached\n'], []) 2022-02-08 05:34:37,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 05:34:37,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:34:37,529 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['112268 memcached\n'], []) 2022-02-08 05:34:37,529 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:34:37,532 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:34:37,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:34:37,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:34:37,914 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:34:37,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:34:37,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:34:37,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:34:37,971 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:34:37,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:34:38,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:34:38,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:34:38,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:34:38,055 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 05:34:38,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:34:38,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:34:42,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:34:42,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:34:42,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:34:43,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:34:43,023 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 05:34:43,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:34:43,038 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:34:43,086 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:34:43,099 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 05:34:43,099 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:34:43,101 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:34:43,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:34:43,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:34:43,467 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:34:43,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:34:43,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:34:43,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:34:43,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:34:43,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:34:43,545 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:34:43,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:34:43,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:34:43,588 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 05:34:43,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:34:43,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:34:44,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:34:44,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:34:44,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:34:44,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:34:44,828 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 05:34:44,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:34:44,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:34:44,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:34:44,890 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 05:34:44,890 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:34:44,892 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:34:44,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:34:45,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:34:45,229 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:34:45,229 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:34:45,242 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:34:45,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:34:45,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:34:45,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:34:45,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:34:45,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:34:45,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:34:45,342 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 05:34:45,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:34:45,387 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 7598 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:34:49,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:34:50,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:34:50,377 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:34:50,624 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:34:50,638 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 05:34:50,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:34:50,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:34:50,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:34:50,723 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 05:34:50,724 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:34:50,726 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 05:34:50,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:34:51,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 05:34:51,109 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:34:51,109 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 05:34:51,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 05:34:51,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 05:34:51,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 05:34:51,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 05:34:51,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 05:34:51,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 05:34:51,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:34:51,223 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 05:34:51,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:34:51,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:34:51,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:34:51,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:34:51,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:34:51,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:34:51,344 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 05:34:51,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:34:51,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:34:51,377 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:34:51,390 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:51,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:34:51,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:51,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:51,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:51,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:51,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:51,637 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:51,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:51,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:51,744 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:51,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:34:51,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:51,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:51,813 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:51,846 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:51,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:51,897 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:51,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:51,936 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:51,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:51,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:34:51,989 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:52,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:52,022 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:52,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:52,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:52,073 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:52,089 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:52,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:52,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:52,145 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:34:52,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:52,176 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:52,191 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:52,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:52,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:52,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:52,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:52,275 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:52,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:52,308 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:34:52,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:52,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:52,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:52,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:52,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:52,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:52,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:52,441 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:52,457 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 05:34:52,459 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:34:52,586 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:34:53,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:34:53,381 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:34:53,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:34:53,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:34:53,624 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:34:53,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:34:53,706 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:34:53,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:34:53,737 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:34:53,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:34:53,858 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 05:34:53,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:34:54,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:34:56,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:34:57,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:34:57,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:34:57,698 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:34:57,747 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 05:34:57,747 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:34:57,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:34:57,867 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:34:57,904 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 05:34:57,904 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:34:57,907 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 05:34:58,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:34:58,278 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 05:34:58,346 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:34:58,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 05:34:58,362 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 05:34:58,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 05:34:58,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 05:34:58,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 05:34:58,430 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 05:34:58,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 05:34:58,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:34:58,482 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 05:34:58,505 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:34:58,522 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:34:58,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:34:58,558 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:34:58,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:34:58,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:34:58,605 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 05:34:58,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:34:58,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:34:58,642 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:34:58,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:58,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:34:58,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:58,778 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:58,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:58,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:58,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:58,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:58,971 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:59,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:34:59,048 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:59,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:34:59,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:59,107 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:59,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:59,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:59,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:59,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:59,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:59,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:34:59,244 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:59,262 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:34:59,280 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:59,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:59,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:59,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:59,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:59,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:59,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:59,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:34:59,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:59,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:34:59,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:59,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:59,487 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:59,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:59,520 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:59,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:59,556 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:59,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:34:59,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:59,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:34:59,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:59,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:59,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:59,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:59,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:59,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:59,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:59,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:34:59,786 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 05:34:59,789 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:34:59,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:35:00,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:35:00,137 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:35:00,137 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:35:00,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:35:00,165 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:35:00,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:35:00,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:35:00,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:35:00,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:35:00,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:35:00,255 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 05:35:00,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:35:00,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:03,946 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:35:04,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:04,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:05,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:35:05,049 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 05:35:05,049 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:35:05,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:05,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:35:05,135 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 05:35:05,135 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:35:05,138 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:35:05,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:35:05,752 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:35:05,990 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:35:05,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:35:06,130 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:35:06,283 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:35:06,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:35:06,354 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:35:06,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:35:06,390 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:35:06,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:35:06,428 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 05:35:06,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:35:06,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:10,789 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:35:11,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:11,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:11,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:35:11,822 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 05:35:11,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:35:11,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:11,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:35:11,928 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 05:35:11,928 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:35:11,930 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 05:35:12,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:35:12,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 05:35:12,345 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:35:12,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 05:35:12,364 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 05:35:12,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 05:35:12,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 05:35:12,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 05:35:12,433 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 05:35:12,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 05:35:12,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:35:12,480 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 05:35:12,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:35:12,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:12,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:35:12,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:12,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:12,586 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:35:12,604 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 05:35:12,604 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:35:12,619 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:12,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:35:12,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:12,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:35:12,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:12,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:12,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:12,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:12,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:12,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:12,974 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:13,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:13,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:13,072 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:35:13,091 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:13,109 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:13,130 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:13,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:13,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:13,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:13,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:13,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:13,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:13,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:35:13,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:13,295 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:13,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:13,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:13,349 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:13,368 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:13,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:13,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:13,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:13,433 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:35:13,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:13,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:13,484 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:13,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:13,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:13,533 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:13,551 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:13,568 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:13,584 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:13,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:35:13,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:13,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:13,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:13,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:13,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:13,724 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:13,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:13,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:13,806 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 05:35:13,809 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 05:35:13,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:35:14,145 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 05:35:14,209 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:35:14,209 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 05:35:14,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 05:35:14,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 05:35:14,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 05:35:14,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 05:35:14,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 05:35:14,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 05:35:14,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:35:14,335 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 05:35:14,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:35:14,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:14,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:35:14,413 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:14,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:14,446 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:35:14,463 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 05:35:14,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:35:14,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:14,494 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:35:14,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:14,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:35:14,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:14,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:14,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:14,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:14,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:14,790 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:14,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:14,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:14,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:14,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:35:14,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:14,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:14,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:14,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:15,015 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:15,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:15,065 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:15,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:15,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:15,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:35:15,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:15,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:15,163 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:15,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:15,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:15,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:15,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:15,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:15,262 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:15,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:35:15,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:15,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:15,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:15,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:15,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:15,377 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:15,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:15,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:15,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:15,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:35:15,464 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:15,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:15,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:15,517 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:15,535 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:15,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:15,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:15,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:15,605 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 05:35:15,608 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:35:15,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:35:15,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:35:16,032 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:35:16,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:35:16,048 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:35:16,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:35:16,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:35:16,094 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:35:16,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:35:16,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:35:16,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:35:16,155 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 05:35:16,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:35:16,196 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:20,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:35:20,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:21,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:21,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:35:21,398 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 05:35:21,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:35:21,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:21,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:35:21,493 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 05:35:21,493 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:35:21,496 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:35:21,604 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:35:21,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:35:21,927 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:35:21,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:35:21,948 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:35:21,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:35:21,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:35:22,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:35:22,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:35:22,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:35:22,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:35:22,092 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 05:35:22,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:35:22,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:25,079 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:35:25,448 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:25,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:25,877 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:35:25,894 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 05:35:25,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:35:25,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:25,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:35:25,974 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 05:35:25,976 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:35:25,977 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:35:26,778 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:35:27,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:35:27,312 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:35:27,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:35:27,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:35:27,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:35:27,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:35:27,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:35:27,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:35:27,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:35:27,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:35:27,490 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 05:35:27,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:35:27,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:30,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:35:31,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:31,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:31,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:35:31,546 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 05:35:31,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:35:31,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:31,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:35:31,628 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 05:35:31,628 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:35:31,631 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:35:31,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:35:31,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:35:32,059 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:35:32,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:35:32,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:35:32,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:35:32,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:35:32,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:35:32,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:35:32,163 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:35:32,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:35:32,204 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 05:35:32,229 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:35:32,255 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:34,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:35:34,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:35,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:35,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:35:35,596 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 05:35:35,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:35:35,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:35,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:35:35,710 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 05:35:35,711 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:35:35,713 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 05:35:35,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:35:36,051 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 05:35:36,118 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:35:36,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 05:35:36,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 05:35:36,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 05:35:36,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 05:35:36,186 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 05:35:36,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 05:35:36,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 05:35:36,234 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:35:36,252 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 05:35:36,278 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:35:36,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:36,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:35:36,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:36,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:35:36,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:35:36,377 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 05:35:36,377 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:35:36,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:35:36,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:35:36,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:36,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:35:36,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:36,545 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:36,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:36,624 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:36,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:36,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:36,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:36,788 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:35:36,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:36,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:35:36,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:36,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:36,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:36,918 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:36,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:36,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:36,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:37,005 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:35:37,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:37,048 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:35:37,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:37,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:37,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:37,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:37,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:37,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:37,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:37,201 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:35:37,221 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:37,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:35:37,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:37,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:37,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:37,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:37,354 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:37,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:37,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:37,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:35:37,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:37,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:35:37,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:37,497 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:37,517 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:37,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:37,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:37,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:37,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:37,612 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:35:37,630 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 05:35:37,631 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:35:37,632 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:35:37,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:35:37,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:35:38,036 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:35:38,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:35:38,053 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:35:38,066 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:35:38,079 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:35:38,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:35:38,109 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:35:38,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:35:38,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:35:38,650 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:35:38,650 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.01900005341. 2022-02-08 05:35:38,650 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:35:38,651 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:35:39,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:35:39,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:35:39,809 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:35:39,811 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:35:39,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:35:39,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:35:39,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:35:39,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:35:39,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:35:40,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:35:40,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:35:40,733 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:35:40,733 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:35:42,734 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:35:42,930 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:35:42,930 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:35:44,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:35:45,132 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:35:45,134 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:35:47,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:35:47,377 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:35:47,377 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:35:49,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 7621 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:35:49,592 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:35:49,594 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:35:51,594 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:35:51,822 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:35:51,822 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:35:53,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:35:54,006 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:35:54,006 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:35:56,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:35:56,427 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:35:56,427 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:35:58,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:35:58,707 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:35:58,707 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:36:00,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:36:00,986 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:36:00,986 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:36:02,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:36:03,532 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:36:03,532 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:36:05,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:36:05,734 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:36:05,736 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:36:07,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:36:07,936 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:36:07,936 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:36:09,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:36:10,130 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:36:10,130 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:36:12,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:36:12,322 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:36:12,322 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 33.6729998589. 2022-02-08 05:36:12,322 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:36:12,325 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:36:13,020 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:36:13,586 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:36:13,720 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:36:13,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:36:13,744 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:36:13,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:36:13,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:36:13,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:36:13,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:36:13,890 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:36:13,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:36:14,806 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:36:14,808 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.48500013351. 2022-02-08 05:36:14,808 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:36:14,809 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:36:16,163 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:36:16,655 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:36:16,848 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:36:16,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:36:16,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:36:16,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:36:16,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:36:16,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:36:16,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:36:17,002 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:36:17,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:36:17,838 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:36:17,838 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.03099989891. 2022-02-08 05:36:17,838 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:36:17,841 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:36:18,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:36:19,448 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:36:19,559 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:36:19,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:36:19,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:36:19,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:36:19,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:36:19,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:36:19,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:36:19,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:36:19,714 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:36:20,335 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:36:20,336 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.49799990654. 2022-02-08 05:36:20,336 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:36:20,338 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:36:21,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:36:21,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:36:21,710 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:36:21,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:36:21,729 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:36:21,747 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:36:21,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:36:21,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:36:21,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:36:21,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:36:21,892 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:36:22,483 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:36:22,483 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.14700007439. 2022-02-08 05:36:22,484 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:36:22,486 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:36:22,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:36:22,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:36:22,904 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:36:22,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:36:22,918 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:36:22,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:36:22,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:36:22,969 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:36:22,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:36:22,999 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:36:23,015 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:36:23,538 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:36:23,539 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.05300021172. 2022-02-08 05:36:23,539 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:36:23,540 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:36:24,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:36:24,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:36:25,108 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:36:25,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:36:25,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:36:25,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:36:25,209 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:36:25,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:36:25,249 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:36:25,283 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:36:25,303 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:36:25,884 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:36:25,884 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.34599995613. 2022-02-08 05:36:25,884 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:36:25,888 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:36:26,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:36:26,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:36:26,578 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:36:26,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:36:26,594 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:36:26,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:36:26,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:36:26,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:36:26,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:36:26,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:36:26,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:36:27,240 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:36:27,240 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.35599994659. 2022-02-08 05:36:27,242 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:36:27,243 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:36:28,285 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:36:28,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:36:29,019 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:36:29,020 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:36:29,048 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:36:29,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:36:29,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:36:29,130 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:36:29,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:36:29,186 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:36:29,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:36:29,927 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:36:29,927 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.6859998703. ############################################################ # # RAM FootPrint: 4365.80859375 # ############################################################ 2022-02-08 05:36:29,957 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 05:36:29,959 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 103 seconds. Reason: Iteration:19 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 7644 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 7667 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:38:12,967 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 05:38:12,967 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 05:38:17,971 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:38:18,816 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:38:19,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:38:19,726 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:38:19,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:38:19,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:38:19,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:38:19,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:38:19,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:38:19,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:38:19,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:38:19,908 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:38:19,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:38:19,980 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['4960 memcached\n'], []) 2022-02-08 05:38:19,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 05:38:20,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:38:20,030 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['4960 memcached\n'], []) 2022-02-08 05:38:20,032 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:38:20,035 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:38:20,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:38:21,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:38:21,595 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:38:21,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:38:21,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:38:21,640 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:38:21,655 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:38:21,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:38:21,707 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:38:21,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:38:21,785 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:38:21,785 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:38:21,865 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['50058 memcached\n'], []) 2022-02-08 05:38:21,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 05:38:21,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:38:21,927 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['50058 memcached\n'], []) 2022-02-08 05:38:21,927 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:38:21,930 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:38:22,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:38:23,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:38:23,732 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:38:23,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:38:23,752 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:38:23,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:38:23,793 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:38:23,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:38:23,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:38:23,872 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:38:23,892 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:38:23,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:38:23,961 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['36658 memcached\n'], []) 2022-02-08 05:38:23,961 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 05:38:23,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:38:24,029 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['36658 memcached\n'], []) 2022-02-08 05:38:24,029 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:38:24,032 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:38:24,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:38:25,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:38:25,697 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:38:25,698 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:38:25,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:38:25,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:38:25,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:38:25,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:38:25,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:38:25,877 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:38:25,898 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:38:25,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:38:25,969 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['69574 memcached\n'], []) 2022-02-08 05:38:25,969 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 05:38:26,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:38:26,032 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['69574 memcached\n'], []) 2022-02-08 05:38:26,032 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:38:26,035 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:38:26,931 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:38:27,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:38:27,961 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:38:27,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:38:27,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:38:28,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:38:28,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:38:28,099 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:38:28,114 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:38:28,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:38:28,174 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:38:28,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:38:28,233 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['63437 memcached\n'], []) 2022-02-08 05:38:28,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 05:38:28,265 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:38:28,289 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['63437 memcached\n'], []) 2022-02-08 05:38:28,290 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:38:28,292 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:38:29,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:38:29,678 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:38:29,829 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:38:29,829 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:38:29,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:38:29,890 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:38:29,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:38:29,926 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:38:29,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:38:29,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:38:29,990 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:38:29,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:38:30,056 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['98426 memcached\n'], []) 2022-02-08 05:38:30,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 05:38:30,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:38:30,111 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['98426 memcached\n'], []) 2022-02-08 05:38:30,111 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:38:30,112 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:38:30,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:38:31,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:38:31,595 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:38:31,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:38:31,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:38:31,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:38:31,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:38:31,691 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:38:31,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:38:31,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:38:31,766 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:38:31,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:38:31,828 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['32191 memcached\n'], []) 2022-02-08 05:38:31,829 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 05:38:31,859 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:38:31,882 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['32191 memcached\n'], []) 2022-02-08 05:38:31,884 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:38:31,885 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:38:32,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:38:33,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:38:33,128 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:38:33,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:38:33,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:38:33,165 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:38:33,183 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:38:33,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:38:33,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:38:33,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:38:33,295 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:38:33,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:38:33,351 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['47161 memcached\n'], []) 2022-02-08 05:38:33,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 05:38:33,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:38:33,398 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['47161 memcached\n'], []) 2022-02-08 05:38:33,398 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:38:33,401 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:38:34,132 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:38:34,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:38:34,673 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:38:34,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:38:34,690 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:38:34,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:38:34,734 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:38:34,765 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:38:34,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:38:34,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:38:34,835 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:38:34,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:38:34,903 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['83137 memcached\n'], []) 2022-02-08 05:38:34,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 05:38:34,933 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:38:34,957 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['83137 memcached\n'], []) 2022-02-08 05:38:34,959 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:38:34,960 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:38:35,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:38:35,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:38:35,786 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:38:35,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:38:35,803 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:38:35,822 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:38:35,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:38:35,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:38:35,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:38:35,910 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:38:35,931 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:38:35,933 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:38:35,990 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['112976 memcached\n'], []) 2022-02-08 05:38:35,992 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 05:38:36,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:38:36,055 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['112976 memcached\n'], []) 2022-02-08 05:38:36,055 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:38:36,059 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:38:36,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:38:36,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:38:36,437 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:38:36,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:38:36,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:38:36,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:38:36,480 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:38:36,499 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:38:36,512 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:38:36,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:38:36,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:38:36,553 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 05:38:36,579 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:38:36,594 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:38:40,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:38:41,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:38:41,303 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:38:41,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:38:41,579 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 05:38:41,581 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:38:41,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:38:41,647 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:38:41,661 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 05:38:41,661 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:38:41,664 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:38:41,785 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:38:42,026 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:38:42,089 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:38:42,091 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:38:42,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:38:42,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:38:42,132 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:38:42,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:38:42,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:38:42,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:38:42,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:38:42,219 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 05:38:42,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:38:42,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:38:43,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:38:43,354 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:38:43,426 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:38:43,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:38:43,516 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 05:38:43,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:38:43,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:38:43,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:38:43,576 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 05:38:43,578 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:38:43,578 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:38:43,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:38:43,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:38:43,936 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:38:43,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:38:43,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:38:43,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:38:43,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:38:43,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:38:44,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:38:44,017 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:38:44,030 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:38:44,046 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 05:38:44,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:38:44,082 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:38:48,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:38:49,085 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 7690 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:38:49,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:38:49,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:38:49,644 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 05:38:49,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:38:49,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:38:49,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:38:49,747 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 05:38:49,747 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:38:49,750 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 05:38:49,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:38:50,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 05:38:50,167 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:38:50,168 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 05:38:50,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 05:38:50,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 05:38:50,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 05:38:50,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 05:38:50,247 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 05:38:50,262 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 05:38:50,278 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:38:50,293 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 05:38:50,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:38:50,341 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:38:50,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:38:50,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:38:50,385 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:38:50,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:38:50,414 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 05:38:50,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:38:50,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:38:50,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:38:50,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:50,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:38:50,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:50,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:50,619 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:50,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:50,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:50,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:50,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:50,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:50,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:50,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:38:50,875 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:50,892 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:50,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:50,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:50,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:50,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:50,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:51,002 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:51,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:51,035 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:38:51,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:51,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:51,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:51,099 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:51,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:51,132 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:51,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:51,168 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:51,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:51,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:38:51,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:51,239 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:51,255 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:51,270 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:51,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:51,308 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:51,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:51,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:51,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:51,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:38:51,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:51,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:51,436 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:51,463 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:51,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:51,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:51,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:51,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:51,561 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 05:38:51,562 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:38:51,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:38:51,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:38:52,193 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:38:52,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:38:52,362 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:38:52,433 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:38:52,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:38:52,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:38:52,492 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:38:52,507 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:38:52,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:38:52,542 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 05:38:52,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:38:52,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:38:55,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:38:55,594 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:38:55,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:38:55,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:38:56,039 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 05:38:56,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:38:56,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:38:56,260 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:38:56,279 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 05:38:56,279 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:38:56,280 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 05:38:56,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:38:56,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 05:38:56,697 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:38:56,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 05:38:56,711 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 05:38:56,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 05:38:56,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 05:38:56,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 05:38:56,778 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 05:38:56,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 05:38:56,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:38:56,829 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 05:38:56,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:38:56,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:38:56,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:38:56,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:38:56,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:38:56,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:38:56,963 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 05:38:56,964 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:38:56,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:38:56,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:38:57,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:57,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:38:57,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:57,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:57,073 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:57,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:57,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:57,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:57,137 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:57,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:38:57,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:57,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:38:57,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:57,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:57,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:57,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:57,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:57,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:57,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:57,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:38:57,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:57,395 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:38:57,415 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:57,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:57,454 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:57,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:57,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:57,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:57,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:57,561 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:38:57,581 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:57,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:38:57,618 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:57,635 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:57,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:57,670 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:57,686 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:57,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:57,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:57,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:38:57,755 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:57,775 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:38:57,795 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:57,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:57,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:57,849 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:57,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:57,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:57,903 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:57,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:38:57,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:38:57,961 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 05:38:57,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:38:58,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:38:58,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:38:58,048 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:38:58,066 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:38:58,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:38:58,108 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:38:58,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:38:58,150 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 05:38:58,154 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:38:58,230 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:38:58,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:38:58,520 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:38:58,520 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:38:58,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:38:58,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:38:58,562 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:38:58,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:38:58,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:38:58,602 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:38:58,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:38:58,631 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 05:38:58,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:38:58,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:02,489 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:39:02,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:03,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:03,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:39:03,565 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 05:39:03,566 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:39:03,579 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:03,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:39:03,641 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 05:39:03,641 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:39:03,642 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:39:03,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:39:03,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:39:04,068 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:39:04,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:39:04,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:39:04,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:39:04,124 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:39:04,145 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:39:04,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:39:04,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:39:04,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:39:04,221 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 05:39:04,249 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:39:04,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:08,494 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:39:08,974 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:09,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:09,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:39:09,533 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 05:39:09,533 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:39:09,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:09,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:39:09,640 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 05:39:09,641 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:39:09,642 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 05:39:09,762 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:39:10,007 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 05:39:10,076 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:39:10,076 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 05:39:10,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 05:39:10,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 05:39:10,127 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 05:39:10,154 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 05:39:10,173 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 05:39:10,188 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 05:39:10,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:39:10,223 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 05:39:10,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:39:10,278 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:10,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:39:10,318 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:10,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:10,346 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:39:10,362 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 05:39:10,362 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:39:10,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:10,403 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:39:10,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:10,436 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:39:10,454 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:10,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:10,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:10,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:10,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:10,545 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:10,563 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:10,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:10,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:10,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:39:10,642 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:10,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:10,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:10,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:10,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:10,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:10,760 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:10,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:10,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:10,818 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:39:10,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:10,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:10,877 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:10,897 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:10,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:10,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:10,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:10,974 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:10,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:11,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:39:11,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:11,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:11,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:11,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:11,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:11,137 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:11,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:11,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:11,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:11,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:39:11,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:11,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:11,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:11,285 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:11,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:11,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:11,341 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:11,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:11,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:11,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 05:39:11,420 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:11,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:11,461 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:11,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:11,503 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:11,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:11,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:11,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:11,591 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 05:39:11,594 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 05:39:11,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:39:11,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 05:39:11,999 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:39:12,000 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 05:39:12,017 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 05:39:12,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 05:39:12,048 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 05:39:12,068 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 05:39:12,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 05:39:12,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 05:39:12,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:39:12,137 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 05:39:12,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:39:12,186 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:12,201 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:39:12,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:12,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:12,255 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:39:12,270 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 05:39:12,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:39:12,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:12,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:39:12,321 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:12,336 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:39:12,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:12,371 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:12,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:12,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:12,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:12,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:12,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:12,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:12,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:12,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:39:12,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:12,558 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:12,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:12,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:12,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:12,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:12,658 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:12,678 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:12,697 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:12,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:39:12,747 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:12,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:12,783 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:12,806 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:12,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:12,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:12,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:12,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:12,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:12,921 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:39:12,938 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:12,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:12,977 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:12,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:13,010 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:13,029 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:13,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:13,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:13,079 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:13,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:39:13,112 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:13,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:13,150 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:13,168 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:13,186 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:13,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:13,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:13,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:13,257 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:13,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 05:39:13,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:13,313 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:13,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:13,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:13,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:13,388 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:13,408 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:13,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:13,450 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 05:39:13,451 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:39:13,553 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:39:13,793 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:39:13,891 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:39:13,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:39:13,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:39:13,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:39:13,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:39:13,997 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:39:14,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:39:14,063 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:39:14,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:39:14,112 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 05:39:14,151 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:39:14,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:18,198 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:39:18,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:19,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:19,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:39:19,415 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 05:39:19,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:39:19,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:19,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:39:19,687 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 05:39:19,687 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:39:19,690 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:39:19,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:39:20,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:39:20,815 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:39:20,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:39:20,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:39:20,846 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:39:20,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:39:20,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:39:20,987 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:39:21,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:39:21,176 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:39:21,266 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 05:39:21,293 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:39:21,382 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:24,640 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:39:25,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:25,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:25,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:39:25,684 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 05:39:25,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:39:25,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:25,755 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:39:25,776 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 05:39:25,776 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:39:25,778 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:39:26,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:39:26,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:39:27,115 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:39:27,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:39:27,132 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:39:27,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:39:27,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:39:27,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:39:27,280 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:39:27,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:39:27,323 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:39:27,341 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 05:39:27,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:39:27,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:30,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:39:31,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:31,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:31,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:39:31,565 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 05:39:31,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:39:31,579 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:31,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:39:31,637 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 05:39:31,637 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:39:31,638 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:39:31,733 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:39:32,006 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:39:32,078 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:39:32,079 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:39:32,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:39:32,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:39:32,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:39:32,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:39:32,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:39:32,204 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:39:32,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:39:32,276 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 05:39:32,301 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:39:32,325 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:36,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:39:37,438 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:37,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:38,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:39:38,137 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 05:39:38,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:39:38,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:38,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:39:38,276 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 05:39:38,276 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:39:38,278 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 05:39:38,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:39:38,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 05:39:38,683 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:39:38,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 05:39:38,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 05:39:38,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 05:39:38,732 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 05:39:38,749 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 05:39:38,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 05:39:38,778 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 05:39:38,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:39:38,819 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 05:39:38,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:39:38,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:38,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:39:38,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:38,918 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:39:38,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:39:38,950 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 05:39:38,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:39:38,964 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:39:38,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:39:38,999 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:39,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:39:39,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:39,051 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:39,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:39,086 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:39,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:39,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:39,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:39,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:39:39,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:39,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:39:39,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:39,242 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:39,260 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:39,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:39,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:39,318 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:39,336 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:39,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:39:39,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:39,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:39:39,410 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:39,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:39,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:39,469 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:39,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:39,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:39,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:39,540 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:39:39,558 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:39,573 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:39:39,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:39,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:39,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:39,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:39,660 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:39,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:39,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:39,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:39:39,757 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:39,775 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:39:39,792 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:39,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:39,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:39,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:39,859 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:39,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:39,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:39,910 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:39:39,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:39,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 05:39:39,966 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:39,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:40,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:40,022 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:40,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:40,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:40,075 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:40,092 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:39:40,111 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 05:39:40,111 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:39:40,114 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:39:40,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:39:40,464 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:39:40,549 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:39:40,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:39:40,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:39:40,582 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:39:40,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:39:40,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:39:40,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:39:40,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:39:40,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:39:41,194 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:39:41,194 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.08299994469. 2022-02-08 05:39:41,194 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:39:41,197 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:39:41,328 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:39:41,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:39:41,736 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:39:41,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:39:41,775 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:39:41,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:39:41,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:39:41,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:39:41,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:39:41,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:39:41,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:39:42,641 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:39:42,641 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:39:44,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:39:44,825 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:39:44,826 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:39:46,826 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:39:47,023 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:39:47,023 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:39:49,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:39:49,236 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:39:49,236 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 7710 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:39:51,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:39:51,420 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:39:51,421 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:39:53,421 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:39:53,611 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:39:53,611 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:39:55,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:39:55,815 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:39:55,815 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:39:57,816 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:39:58,017 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:39:58,017 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:40:00,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:40:00,290 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:40:00,290 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:40:02,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:40:02,792 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:40:02,792 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:40:04,793 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:40:04,983 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:40:04,983 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:40:06,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:40:07,194 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:40:07,196 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:40:09,196 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:40:09,400 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:40:09,401 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:40:11,401 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:40:11,586 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:40:11,586 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:40:13,586 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:40:13,772 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:40:13,772 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:40:15,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:40:15,943 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:40:15,944 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 34.7490000725. 2022-02-08 05:40:15,944 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:40:15,947 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:40:16,418 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:40:16,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:40:17,029 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:40:17,030 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:40:17,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:40:17,109 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:40:17,138 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:40:17,177 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:40:17,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:40:17,237 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:40:17,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:40:17,931 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:40:17,933 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 05:40:19,933 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:40:20,127 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:40:20,127 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.18300008774. 2022-02-08 05:40:20,127 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:40:20,128 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:40:20,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:40:21,305 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:40:21,427 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:40:21,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:40:21,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:40:21,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:40:21,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:40:21,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:40:21,579 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:40:21,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:40:21,650 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:40:22,384 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:40:22,384 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.257999897. 2022-02-08 05:40:22,384 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:40:22,387 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:40:23,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:40:23,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:40:23,571 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:40:23,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:40:23,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:40:23,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:40:23,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:40:23,653 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:40:23,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:40:23,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:40:23,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:40:24,342 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:40:24,344 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.95900011063. 2022-02-08 05:40:24,344 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:40:24,345 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:40:25,163 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:40:25,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:40:25,631 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:40:25,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:40:25,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:40:25,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:40:25,693 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:40:25,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:40:25,746 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:40:25,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:40:25,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:40:26,378 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:40:26,384 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.04099988937. 2022-02-08 05:40:26,384 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:40:26,387 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:40:26,846 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:40:27,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:40:27,219 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:40:27,220 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:40:27,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:40:27,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:40:27,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:40:27,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:40:27,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:40:27,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:40:27,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:40:27,940 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:40:27,940 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.55599999428. 2022-02-08 05:40:27,941 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:40:27,944 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:40:28,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:40:29,249 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:40:29,466 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:40:29,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:40:29,497 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:40:29,525 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:40:29,556 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:40:29,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:40:29,595 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:40:29,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:40:29,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:40:30,232 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:40:30,233 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.29000020027. 2022-02-08 05:40:30,233 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:40:30,234 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:40:30,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:40:31,174 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:40:31,289 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:40:31,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:40:31,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:40:31,339 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:40:31,357 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:40:31,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:40:31,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:40:31,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:40:31,443 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:40:32,025 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:40:32,025 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.79200005531. 2022-02-08 05:40:32,026 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:40:32,028 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:40:32,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:40:33,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:40:33,267 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:40:33,267 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:40:33,288 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:40:33,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:40:33,344 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:40:33,374 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:40:33,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:40:33,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:40:33,444 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:40:33,996 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:40:33,996 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.97000002861. ############################################################ # # RAM FootPrint: 4366.3359375 # ############################################################ 2022-02-08 05:40:34,032 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 05:40:34,032 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 71 seconds. Reason: Iteration:20 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 7733 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:41:45,038 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 05:41:45,039 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 5 queries submitted, 0 failed, 7755 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:41:50,042 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:41:50,857 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:41:51,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:41:51,704 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:41:51,704 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:41:51,772 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:41:51,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:41:51,880 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:41:51,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:41:51,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:41:51,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:41:52,036 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:41:52,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:41:52,155 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['5682 memcached\n'], []) 2022-02-08 05:41:52,155 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 05:41:52,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 05:41:52,236 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['5682 memcached\n'], []) 2022-02-08 05:41:52,236 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:41:52,237 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:41:53,176 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:41:53,601 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:41:53,740 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:41:53,742 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:41:53,765 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:41:53,788 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:41:53,805 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:41:53,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:41:53,845 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:41:53,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:41:53,898 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:41:53,898 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:41:53,970 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['50948 memcached\n'], []) 2022-02-08 05:41:53,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 05:41:54,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 05:41:54,035 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['50948 memcached\n'], []) 2022-02-08 05:41:54,035 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:41:54,038 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:41:54,711 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:41:55,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:41:55,474 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:41:55,474 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:41:55,505 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:41:55,526 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:41:55,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:41:55,573 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:41:55,591 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:41:55,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:41:55,661 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:41:55,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:41:55,742 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['37476 memcached\n'], []) 2022-02-08 05:41:55,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 05:41:55,778 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 05:41:55,805 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['37476 memcached\n'], []) 2022-02-08 05:41:55,806 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:41:55,809 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:41:56,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:41:57,216 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:41:57,424 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:41:57,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:41:57,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:41:57,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:41:57,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:41:57,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:41:57,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:41:57,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:41:57,604 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:41:57,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:41:57,676 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['70295 memcached\n'], []) 2022-02-08 05:41:57,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 05:41:57,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 05:41:57,739 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['70295 memcached\n'], []) 2022-02-08 05:41:57,739 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:41:57,742 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:41:58,693 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:41:59,176 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:41:59,359 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:41:59,361 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:41:59,408 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:41:59,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:41:59,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:41:59,492 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:41:59,517 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:41:59,561 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:41:59,578 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:41:59,578 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:41:59,641 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['64159 memcached\n'], []) 2022-02-08 05:41:59,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 05:41:59,673 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 05:41:59,696 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['64159 memcached\n'], []) 2022-02-08 05:41:59,697 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:41:59,698 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:42:00,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:42:01,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:42:01,528 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:42:01,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:42:01,546 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:42:01,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:42:01,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:42:01,607 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:42:01,624 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:42:01,657 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:42:01,674 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:42:01,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:42:01,730 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['99148 memcached\n'], []) 2022-02-08 05:42:01,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 05:42:01,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 05:42:01,780 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['99148 memcached\n'], []) 2022-02-08 05:42:01,782 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:42:01,783 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:42:02,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:42:02,900 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:42:03,115 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:42:03,117 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:42:03,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:42:03,153 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:42:03,170 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:42:03,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:42:03,207 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:42:03,240 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:42:03,256 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:42:03,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:42:03,318 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['32913 memcached\n'], []) 2022-02-08 05:42:03,318 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 05:42:03,346 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 05:42:03,371 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['32913 memcached\n'], []) 2022-02-08 05:42:03,371 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:42:03,375 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:42:04,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:42:04,642 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:42:04,772 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:42:04,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:42:04,788 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:42:04,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:42:04,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:42:04,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:42:04,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:42:04,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:42:04,904 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:42:04,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:42:04,966 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['47887 memcached\n'], []) 2022-02-08 05:42:04,967 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 05:42:04,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 05:42:05,028 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['47887 memcached\n'], []) 2022-02-08 05:42:05,029 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:42:05,030 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:42:05,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:42:06,069 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:42:06,211 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:42:06,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:42:06,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:42:06,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:42:06,260 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:42:06,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:42:06,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:42:06,331 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:42:06,358 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.121.136 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:42:06,359 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:42:06,427 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.121.136 - Process info before sending signal: (['83860 memcached\n'], []) 2022-02-08 05:42:06,428 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: kill -9 $(pgrep memcached) 2022-02-08 05:42:06,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: pgrep -l memcached 2022-02-08 05:42:06,482 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.121.136 - Process info after sending signal: (['83860 memcached\n'], []) 2022-02-08 05:42:06,482 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:42:06,484 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:42:07,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:42:07,665 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:42:07,861 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:42:07,862 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:42:07,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:42:07,900 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:42:07,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:42:07,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:42:07,951 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:42:07,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:42:07,994 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:42:07,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:42:08,052 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['113699 memcached\n'], []) 2022-02-08 05:42:08,052 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 05:42:08,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:42:08,108 | infra | DEBUG | Thread-8010 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['113699 memcached\n'], []) 2022-02-08 05:42:08,108 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:42:08,111 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:42:08,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:42:08,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:42:08,568 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:42:08,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:42:08,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:42:08,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:42:08,617 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:42:08,635 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:42:08,648 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:42:08,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:42:08,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:42:08,691 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 05:42:08,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:42:08,739 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:12,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:42:12,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:13,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:13,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:42:13,313 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 05:42:13,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:42:13,332 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:13,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:42:13,395 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 05:42:13,397 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:42:13,400 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:42:13,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:42:13,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:42:13,802 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:42:13,802 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:42:13,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:42:13,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:42:13,852 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:42:13,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:42:13,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:42:13,897 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:42:13,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:42:13,927 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 05:42:13,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:42:13,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:14,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:42:15,062 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:15,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:15,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:42:15,229 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 05:42:15,229 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:42:15,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:15,286 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:42:15,312 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 05:42:15,312 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:42:15,315 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:42:15,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:42:15,638 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:42:15,709 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:42:15,709 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:42:15,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:42:15,737 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:42:15,750 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:42:15,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:42:15,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:42:15,793 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:42:15,808 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:42:15,823 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 05:42:15,859 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:42:15,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:20,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:42:20,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:20,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:21,188 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:42:21,203 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 05:42:21,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:42:21,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:21,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:42:21,289 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 05:42:21,289 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:42:21,293 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 05:42:21,380 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:42:21,615 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 05:42:21,687 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:42:21,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 05:42:21,701 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 05:42:21,716 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 05:42:21,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 05:42:21,747 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 05:42:21,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 05:42:21,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 05:42:21,790 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:42:21,808 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 05:42:21,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:42:21,849 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:21,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:42:21,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:21,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:21,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:42:21,926 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 05:42:21,927 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:42:21,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:21,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:42:21,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:21,984 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:42:22,002 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:22,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:22,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:22,059 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:22,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:22,094 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:22,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:22,128 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:22,147 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:22,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:42:22,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:22,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:22,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:22,242 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:22,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:22,276 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:22,296 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:22,315 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:22,334 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:22,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:42:22,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:22,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:22,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:22,434 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:22,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:22,469 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:22,486 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:22,505 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:22,520 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:22,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:42:22,556 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:22,573 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:22,592 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:22,609 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:22,627 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:22,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:22,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:22,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:22,696 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:22,713 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:42:22,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:22,747 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:22,765 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:22,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:22,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:22,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:22,829 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:22,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:22,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:22,885 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 05:42:22,905 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:22,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:22,940 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:22,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:22,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:22,994 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:23,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:23,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:23,059 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 05:42:23,062 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:42:23,279 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:42:23,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:42:24,177 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:42:24,178 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:42:24,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:42:24,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:42:24,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:42:24,625 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:42:24,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:42:24,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:42:24,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:42:24,711 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 05:42:24,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:42:24,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:27,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:42:28,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:28,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:28,650 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:42:28,665 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 05:42:28,665 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:42:28,681 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:28,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:42:28,746 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 05:42:28,746 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:42:28,747 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 05:42:28,838 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:42:29,076 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 05:42:29,142 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:42:29,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 05:42:29,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 05:42:29,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 05:42:29,186 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 05:42:29,203 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 05:42:29,219 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 05:42:29,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 05:42:29,250 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:42:29,266 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 05:42:29,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:42:29,318 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:29,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:42:29,352 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:29,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:29,385 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:42:29,405 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 05:42:29,407 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:42:29,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:29,441 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:42:29,459 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:29,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:42:29,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:29,516 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:29,533 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:29,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:29,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:29,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:29,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:29,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:29,644 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:29,663 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:42:29,683 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:29,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:29,721 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:29,740 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:29,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:29,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:29,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:29,823 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:29,841 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:29,858 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:42:29,877 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:29,894 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:29,911 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:29,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:29,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:29,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:29,980 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:29,999 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:30,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:30,035 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:42:30,053 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:30,071 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:30,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:30,104 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:30,119 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:30,140 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:30,157 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:30,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:30,209 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:30,227 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:42:30,249 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:30,266 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:30,285 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:30,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:30,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:30,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:30,357 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:30,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:30,392 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:30,411 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 05:42:30,431 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:30,451 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:30,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:30,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:30,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:30,529 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:30,549 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:30,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:30,586 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 05:42:30,588 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:42:30,664 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:42:30,874 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:42:30,934 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:42:30,936 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:42:30,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:42:30,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:42:30,979 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:42:30,999 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:42:31,016 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:42:31,030 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:42:31,046 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:42:31,062 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 05:42:31,105 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:42:31,122 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:35,098 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:42:35,611 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:35,937 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:36,233 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:42:36,252 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 05:42:36,252 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:42:36,270 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:36,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:42:36,357 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 05:42:36,357 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:42:36,358 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:42:36,571 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:42:37,273 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:42:37,470 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:42:37,471 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:42:37,681 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:42:37,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:42:37,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:42:37,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:42:37,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:42:37,857 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:42:37,954 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:42:38,131 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 05:42:38,165 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:42:38,398 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:42,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:42:43,142 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:43,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:43,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:42:43,816 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 05:42:43,816 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:42:43,865 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:43,959 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:42:43,979 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 05:42:43,980 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:42:43,983 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 05:42:44,088 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:42:44,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 05:42:44,454 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:42:44,456 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 05:42:44,476 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 05:42:44,493 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 05:42:44,510 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 05:42:44,533 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 05:42:44,552 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 05:42:44,569 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 05:42:44,586 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:42:44,605 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 05:42:44,634 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:42:44,655 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:44,674 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:42:44,694 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:44,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:44,726 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:42:44,744 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 05:42:44,744 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:42:44,763 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:44,782 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:42:44,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:44,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:42:44,842 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:44,862 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:44,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:44,904 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:44,931 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:44,953 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:44,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:44,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:45,015 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:45,033 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:42:45,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:45,078 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:45,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:45,124 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:45,145 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:45,165 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:45,187 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:45,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:45,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:45,242 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:42:45,262 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:45,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:45,302 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:45,322 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:45,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:45,365 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:45,404 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:45,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:45,447 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:45,467 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:42:45,487 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:45,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:45,532 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:45,555 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:45,572 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:45,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:45,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:45,628 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:45,647 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:45,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:42:45,686 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:45,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:45,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:45,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:45,762 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:45,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:45,799 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:45,819 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:45,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:45,855 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 05:42:45,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:45,897 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:45,915 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:45,936 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:45,956 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:45,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:45,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:46,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:46,032 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 05:42:46,033 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 05:42:46,125 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:42:46,369 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 05:42:46,447 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:42:46,448 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 05:42:46,466 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 05:42:46,483 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 05:42:46,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 05:42:46,517 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 05:42:46,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 05:42:46,558 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 05:42:46,575 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:42:46,592 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 05:42:46,621 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:42:46,637 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:46,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:42:46,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:46,687 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:46,701 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:42:46,720 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 05:42:46,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:42:46,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:46,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:42:46,769 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:46,786 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:42:46,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:46,828 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:46,848 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:46,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:46,887 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:46,908 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:46,931 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:46,960 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:42:46,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:47,003 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:42:47,022 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:47,042 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:47,061 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:47,081 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:47,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:47,121 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:47,141 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:47,163 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:42:47,180 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:47,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:42:47,216 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:47,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:47,253 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:47,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:47,289 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:47,306 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:47,326 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:47,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:42:47,364 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:47,381 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:42:47,403 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:47,424 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:47,440 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:47,460 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:47,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:47,496 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:47,523 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:47,542 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:42:47,559 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:47,579 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:42:47,598 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:47,614 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:47,631 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:47,651 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:47,667 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:47,684 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:47,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:47,720 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:42:47,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:47,756 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 05:42:47,779 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:47,798 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:47,815 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:47,832 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:47,849 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:47,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:47,885 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:47,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:42:47,927 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 05:42:47,930 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:42:48,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:42:48,351 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:42:48,451 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:42:48,453 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:42:48,473 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:42:48,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:42:48,506 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:42:48,538 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:42:48,561 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:42:48,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:42:48,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:42:48,628 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 05:42:48,658 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:42:48,677 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 7780 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:42:52,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:42:53,058 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:53,391 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:53,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:42:53,743 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 05:42:53,743 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:42:53,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:53,880 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:42:53,911 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 05:42:53,911 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:42:53,914 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:42:54,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:42:54,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:42:54,395 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:42:54,397 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:42:54,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:42:54,436 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:42:54,457 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:42:54,482 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:42:54,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:42:54,519 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:42:54,536 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:42:54,561 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 05:42:54,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:42:54,608 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:57,884 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:42:58,263 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:58,528 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:42:58,775 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:42:58,835 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 05:42:58,835 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:42:58,859 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:42:58,914 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:42:58,933 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 05:42:58,933 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:42:58,936 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:42:59,816 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:43:00,188 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:43:00,282 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:43:00,282 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:43:00,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:43:00,384 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:43:00,563 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:43:00,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:43:00,630 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:43:00,671 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:43:00,688 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:43:00,706 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 05:43:00,739 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:43:00,753 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:43:04,171 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:43:04,589 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:43:04,812 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:43:05,032 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:43:05,055 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 05:43:05,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:43:05,073 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:43:05,118 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:43:05,132 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 05:43:05,134 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:43:05,137 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:43:05,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:43:05,703 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:43:05,782 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:43:05,783 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:43:05,801 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:43:05,825 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:43:05,844 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:43:05,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:43:05,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:43:05,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:43:05,924 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:43:05,947 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 05:43:05,973 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:43:05,993 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:43:08,969 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:43:09,604 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:43:09,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:43:10,299 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:43:10,321 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 05:43:10,321 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:43:10,348 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:43:10,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:43:10,443 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 05:43:10,444 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:43:10,446 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 05:43:10,565 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:43:10,809 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 05:43:10,878 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:43:10,878 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 05:43:10,895 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 05:43:10,913 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 05:43:10,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 05:43:10,948 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 05:43:10,963 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 05:43:10,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 05:43:10,990 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:43:11,009 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 05:43:11,040 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:43:11,056 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:43:11,073 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:43:11,095 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:43:11,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:43:11,193 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:43:11,210 | test | DEBUG | Thread-8010 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 05:43:11,210 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:43:11,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:43:11,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:43:11,259 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:43:11,278 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:43:11,298 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:43:11,316 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:43:11,338 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:43:11,355 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:43:11,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:43:11,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:43:11,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:43:11,433 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:43:11,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:43:11,470 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:43:11,490 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:43:11,509 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:43:11,530 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:43:11,548 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:43:11,566 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:43:11,585 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:43:11,605 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:43:11,624 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:43:11,641 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:43:11,661 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:43:11,680 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:43:11,700 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:43:11,717 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:43:11,736 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:43:11,755 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:43:11,773 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:43:11,790 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:43:11,811 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:43:11,831 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:43:11,849 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:43:11,869 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:43:11,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:43:11,907 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:43:11,933 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:43:11,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:43:11,964 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:43:11,983 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:43:12,002 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:43:12,019 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:43:12,036 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:43:12,055 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:43:12,076 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:43:12,096 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:43:12,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:43:12,131 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:43:12,148 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:43:12,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:43:12,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:43:12,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:43:12,217 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 05:43:12,236 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:43:12,256 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:43:12,275 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:43:12,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:43:12,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:43:12,345 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:43:12,375 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:43:12,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:43:12,413 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 05:43:12,414 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:43:12,415 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:43:12,512 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:43:12,744 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:43:12,816 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:43:12,816 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:43:12,836 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:43:12,851 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:43:12,881 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:43:12,901 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:43:12,917 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:43:12,933 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:43:12,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:13,474 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:43:13,474 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.0609998703. 2022-02-08 05:43:13,476 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:43:13,479 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:43:14,197 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:43:14,558 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:43:14,721 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:43:14,723 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:43:14,766 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:43:14,868 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:43:14,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:43:15,009 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:43:15,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:43:15,184 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:43:15,224 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:16,164 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:43:16,165 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:43:18,167 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:18,437 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:43:18,437 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:43:20,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:20,653 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:43:20,653 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:43:22,654 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:22,887 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:43:22,888 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:43:24,888 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:25,180 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:43:25,181 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:43:27,181 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:27,427 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:43:27,427 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:43:29,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:29,729 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:43:29,730 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:43:31,730 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:31,934 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:43:31,934 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:43:33,934 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:34,131 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:43:34,131 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 05:43:36,134 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:36,335 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:43:36,335 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 22.8599998951. 2022-02-08 05:43:36,336 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:43:36,338 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:43:36,759 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:43:37,213 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:43:37,309 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:43:37,309 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:43:37,334 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:43:37,450 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:43:37,474 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:43:37,513 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:43:37,539 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:43:37,556 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:43:37,584 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:38,572 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:43:38,573 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.23600006104. 2022-02-08 05:43:38,573 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:43:38,575 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:43:39,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:43:39,780 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:43:39,928 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:43:39,928 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:43:39,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:43:39,976 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:43:39,996 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:43:40,023 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:43:40,045 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:43:40,073 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:43:40,111 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:40,793 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:43:40,795 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.22100019455. 2022-02-08 05:43:40,796 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:43:40,799 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:43:41,710 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:43:42,161 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:43:42,318 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:43:42,319 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:43:42,342 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:43:42,378 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:43:42,400 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:43:42,427 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:43:42,446 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:43:42,479 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:43:42,500 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:43,099 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:43:43,101 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.30500006676. 2022-02-08 05:43:43,101 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:43:43,104 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:43:44,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:43:45,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:43:45,796 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:43:45,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:43:45,944 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:43:46,025 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:43:46,115 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:43:46,200 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:43:46,246 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:43:46,312 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:43:46,437 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:47,417 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 05:43:47,417 | test | INFO | Thread-8010 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 5 queries submitted, 0 failed, 7805 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:43:49,417 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:49,595 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:43:49,595 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.49499988556. 2022-02-08 05:43:49,595 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:43:49,599 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:43:50,507 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:43:50,796 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:43:50,928 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:43:50,930 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:43:50,957 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:43:50,986 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:43:51,013 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:43:51,038 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:43:51,053 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:43:51,084 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:43:51,101 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:51,687 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:43:51,688 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.09299993515. 2022-02-08 05:43:51,690 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:43:51,693 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:43:52,588 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:43:53,028 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:43:53,193 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:43:53,194 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:43:53,223 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:43:53,269 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:43:53,292 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:43:53,335 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:43:53,358 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:43:53,394 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:43:53,414 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:54,109 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:43:54,111 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.42100000381. 2022-02-08 05:43:54,111 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:43:54,114 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 05:43:54,752 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:43:55,039 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 05:43:55,144 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:43:55,144 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 05:43:55,164 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 05:43:55,188 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 05:43:55,206 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 05:43:55,226 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 05:43:55,243 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 05:43:55,272 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 05:43:55,290 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:43:55,865 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 05:43:55,865 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.75499987602. 2022-02-08 05:43:55,867 | test | DEBUG | Thread-8010 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 05:43:55,868 | infra | DEBUG | Thread-8010 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:43:56,950 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:43:57,596 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:43:57,775 | infra | DEBUG | Thread-8010 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:43:57,776 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:43:57,793 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:43:57,844 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:43:57,861 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:43:57,891 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:43:57,918 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:43:57,947 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:43:57,970 | infra | DEBUG | Thread-8010 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 05:44:00,051 | infra | DEBUG | Thread-8010 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:44:00,052 | test | INFO | Thread-8010 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.18600010872. 2022-02-08 05:44:00,056 | test | DEBUG | Thread-8010 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L836 2022-02-08 05:44:00,056 | test | DEBUG | Thread-8010 | [common_lib:sleep:24] Sleep 300 seconds. Reason: None 5 queries submitted, 0 failed, 7830 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 7857 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 7882 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 7907 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 7931 passed, 0 rejected, 0 cancelled, 0 timeout ############################################################ # # Step 10: Swap with Loading of docs # ############################################################ KV nodes in cluster: ['172.23.121.116', '172.23.121.123', '172.23.121.124', '172.23.121.126', '172.23.121.127', '172.23.121.129', '172.23.121.135', '172.23.121.115', '172.23.121.136'] CBAS nodes in cluster: [] INDEX nodes in cluster: ['172.23.121.130', '172.23.121.131', '172.23.121.132', '172.23.121.133', '172.23.121.134'] FTS nodes in cluster: [] QUERY nodes in cluster: ['172.23.121.130', '172.23.121.131', '172.23.121.132', '172.23.121.133', '172.23.121.134'] EVENTING nodes in cluster: [] AVAILABLE nodes for cluster: ['172.23.121.128'] Servers coming in : ['172.23.121.128'] with services: ['kv'] Servers going out : ['172.23.121.136'] NEW AVAILABLE nodes for cluster: ['172.23.121.136'] 2022-02-08 05:49:00,213 | infra | DEBUG | MainThread | [task:__init__:249] Last known rebalance status_id: 91eec0441c576c2ba19373594a56144d 2022-02-08 05:49:00,230 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[172.23.121.128]_OUT=[172.23.121.136]_1644328140.06 2022-02-08 05:49:00,237 | infra | INFO | pool-3-thread-19 | [task:start_task:91] Thread 'Rebalance_task_IN=[172.23.121.128]_OUT=[172.23.121.136]_1644328140.06' started 2022-02-08 05:49:00,278 | test | DEBUG | pool-3-thread-19 | [rest_client:add_node:1199] Adding remote node 172.23.121.128:8091 to cluster 172.23.120.170:8091 2022-02-08 05:49:00,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/Hospital.py -> SystemTestMagma():L566 2022-02-08 05:49:00,296 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep for 60s for rebalance to start 2022-02-08 05:49:07,369 | test | DEBUG | pool-3-thread-19 | [task:start_rebalance:384] Removing node 172.23.121.136:8091 from cluster 2022-02-08 05:49:07,371 | test | DEBUG | pool-3-thread-19 | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.128,ns_1@172.23.121.115,ns_1@172.23.121.132,ns_1@172.23.121.136,ns_1@172.23.121.135,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.134,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123,ns_1@172.23.121.131,ns_1@172.23.121.130,ns_1@172.23.121.133', 'ejectedNodes': u'ns_1@172.23.121.136', 'user': 'Administrator'} 2022-02-08 05:49:07,931 | test | DEBUG | pool-3-thread-19 | [rest_client:rebalance:1459] Rebalance operation started 2022-02-08 05:49:07,983 | test | INFO | pool-3-thread-19 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.121.115 | kv | 7.1.0-2255-enterprise | 81.9460427055 | Cluster node | | 172.23.121.132 | index, n1ql | 7.1.0-2255-enterprise | 11.3925694986 | Cluster node | | 172.23.121.136 | kv | 7.1.0-2255-enterprise | 88.6953204476 | --- OUT ---> | | 172.23.121.135 | kv | 7.1.0-2255-enterprise | 88.008646999 | Cluster node | | 172.23.120.170 | kv | 7.1.0-2255-enterprise | 75.2695071869 | Cluster node | | 172.23.121.129 | kv | 7.1.0-2255-enterprise | 82.2935310662 | Cluster node | | 172.23.121.124 | kv | 7.1.0-2255-enterprise | 81.3174114022 | Cluster node | | 172.23.121.126 | kv | 7.1.0-2255-enterprise | 80.9706257982 | Cluster node | | 172.23.121.134 | index, n1ql | 7.1.0-2255-enterprise | 34.8252289436 | Cluster node | | 172.23.121.116 | kv | 7.1.0-2255-enterprise | 88.8832358179 | Cluster node | | 172.23.121.127 | kv | 7.1.0-2255-enterprise | 79.0161414297 | Cluster node | | 172.23.121.123 | kv | 7.1.0-2255-enterprise | 84.3157224698 | Cluster node | | 172.23.121.131 | index, n1ql | 7.1.0-2255-enterprise | 14.8530372306 | Cluster node | | 172.23.121.130 | index, n1ql | 7.1.0-2255-enterprise | 14.5274668055 | Cluster node | | 172.23.121.133 | index, n1ql | 7.1.0-2255-enterprise | 16.0339724617 | Cluster node | | 172.23.121.128 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2022-02-08 05:49:08,000 | infra | DEBUG | pool-3-thread-19 | [task:call:316] New rebalance status_id: 5fdffd4b4ef1b751df2c6850fcd101e6 2022-02-08 05:49:08,017 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 05:49:08,017 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 05:49:08,062 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:49:08,065 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:49:08,065 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:49:18,091 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 05:49:18,092 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 05:49:18,138 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:49:18,142 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:49:18,142 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:49:28,158 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 05:49:28,160 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 05:49:28,194 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:49:28,197 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:49:28,197 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:49:38,216 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.52 % 2022-02-08 05:49:38,216 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 0.52 2022-02-08 05:49:38,259 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:49:38,262 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:49:38,263 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:49:48,292 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.64 % 2022-02-08 05:49:48,292 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 0.64 2022-02-08 05:49:48,361 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:49:48,365 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:49:48,365 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 7956 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:49:58,404 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.16 % 2022-02-08 05:49:58,404 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 1.16 2022-02-08 05:49:58,463 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:49:58,467 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:49:58,467 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:50:00,305 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L766 2022-02-08 05:50:00,305 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 30 seconds. Reason: Let the rebalance begin! 2022-02-08 05:50:08,507 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.38 % 2022-02-08 05:50:08,507 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 2.38 2022-02-08 05:50:08,552 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:50:08,555 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:50:08,556 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:50:18,585 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.38 % 2022-02-08 05:50:18,585 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 2.38 2022-02-08 05:50:18,631 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:50:18,635 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:50:18,635 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:50:28,657 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.38 % 2022-02-08 05:50:28,657 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 2.38 2022-02-08 05:50:28,697 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:50:28,700 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:50:28,700 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:50:30,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.38 % 2022-02-08 05:50:30,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:50:30,359 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:50:38,724 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.54 % 2022-02-08 05:50:38,724 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 3.54 2022-02-08 05:50:38,762 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:50:38,765 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:50:38,765 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:50:40,391 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.54 % 2022-02-08 05:50:40,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:50:40,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:50:48,785 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.11 % 2022-02-08 05:50:48,786 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 4.11 2022-02-08 05:50:48,835 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:50:48,839 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:50:48,839 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 7985 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:50:50,424 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.11 % 2022-02-08 05:50:50,433 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:50:50,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:50:58,862 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.75 % 2022-02-08 05:50:58,864 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 4.75 2022-02-08 05:50:58,901 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:50:58,904 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:50:58,904 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:51:00,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.75 % 2022-02-08 05:51:00,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:51:00,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:51:08,921 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.79 % 2022-02-08 05:51:08,921 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 5.79 2022-02-08 05:51:08,953 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:51:08,956 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:51:08,956 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:51:10,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.79 % 2022-02-08 05:51:10,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:51:10,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:51:18,970 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.92 % 2022-02-08 05:51:18,970 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 5.92 2022-02-08 05:51:19,012 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:51:19,013 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:51:19,015 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:51:20,519 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.92 % 2022-02-08 05:51:20,526 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:51:20,526 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:51:29,115 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.13 % 2022-02-08 05:51:29,115 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 7.13 2022-02-08 05:51:29,171 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:51:29,174 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:51:29,174 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:51:30,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.65 % 2022-02-08 05:51:30,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:51:30,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:51:39,187 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.17 % 2022-02-08 05:51:39,187 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 8.17 2022-02-08 05:51:39,223 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:51:39,224 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:51:39,224 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:51:40,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.17 % 2022-02-08 05:51:40,575 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:51:40,575 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:51:49,252 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.17 % 2022-02-08 05:51:49,252 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 8.17 2022-02-08 05:51:49,296 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:51:49,299 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:51:49,301 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 8010 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:51:50,589 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.17 % 2022-02-08 05:51:50,599 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:51:50,599 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:51:59,313 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.33 % 2022-02-08 05:51:59,315 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 9.33 2022-02-08 05:51:59,346 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:51:59,349 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:51:59,349 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:52:00,614 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.90 % 2022-02-08 05:52:00,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:52:00,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:52:09,375 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.67 % 2022-02-08 05:52:09,375 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 10.67 2022-02-08 05:52:09,411 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:52:09,414 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:52:09,414 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:52:10,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.67 % 2022-02-08 05:52:10,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:52:10,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:52:19,427 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.67 % 2022-02-08 05:52:19,427 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 10.67 2022-02-08 05:52:19,460 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:52:19,463 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:52:19,463 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:52:20,670 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.19 % 2022-02-08 05:52:20,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:52:20,681 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:52:29,483 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.83 % 2022-02-08 05:52:29,483 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 11.83 2022-02-08 05:52:29,520 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:52:29,523 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:52:29,523 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:52:30,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.40 % 2022-02-08 05:52:30,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:52:30,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:52:39,540 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.52 % 2022-02-08 05:52:39,540 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 12.52 2022-02-08 05:52:39,575 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:52:39,578 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:52:39,578 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:52:40,742 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.52 % 2022-02-08 05:52:40,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:52:40,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 8038 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:52:49,599 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.17 % 2022-02-08 05:52:49,599 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 13.17 2022-02-08 05:52:49,634 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:52:49,638 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:52:49,638 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:52:50,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.17 % 2022-02-08 05:52:50,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:52:50,770 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:52:59,654 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.86 % 2022-02-08 05:52:59,654 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 13.86 2022-02-08 05:52:59,710 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:52:59,713 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:52:59,713 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:53:00,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.86 % 2022-02-08 05:53:00,813 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:53:00,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:53:09,746 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.50 % 2022-02-08 05:53:09,747 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 14.5 2022-02-08 05:53:09,776 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:53:09,779 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:53:09,779 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:53:10,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.50 % 2022-02-08 05:53:10,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:53:10,838 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:53:19,795 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.14 % 2022-02-08 05:53:19,795 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 15.14 2022-02-08 05:53:19,828 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:53:19,831 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:53:19,831 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:53:20,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.14 % 2022-02-08 05:53:20,871 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:53:20,871 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:53:29,851 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.83 % 2022-02-08 05:53:29,851 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 15.83 2022-02-08 05:53:29,881 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:53:29,884 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:53:29,884 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:53:30,884 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.83 % 2022-02-08 05:53:30,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:53:30,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:53:39,910 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.48 % 2022-02-08 05:53:39,910 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 16.48 2022-02-08 05:53:39,977 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:53:39,980 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:53:39,980 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:53:40,905 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.48 % 2022-02-08 05:53:40,913 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:53:40,913 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 8060 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:53:50,023 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.00 % 2022-02-08 05:53:50,023 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 17.0 2022-02-08 05:53:50,062 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:53:50,065 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:53:50,065 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:53:50,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.00 % 2022-02-08 05:53:50,931 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:53:50,933 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:54:00,082 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.33 % 2022-02-08 05:54:00,082 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 18.33 2022-02-08 05:54:00,111 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:54:00,112 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:54:00,114 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:54:00,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.33 % 2022-02-08 05:54:00,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:54:00,954 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:54:10,128 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.33 % 2022-02-08 05:54:10,128 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 18.33 2022-02-08 05:54:10,164 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:54:10,167 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:54:10,167 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:54:10,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.33 % 2022-02-08 05:54:10,990 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:54:10,990 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:54:20,198 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.54 % 2022-02-08 05:54:20,198 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 19.54 2022-02-08 05:54:20,278 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:54:20,280 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:54:20,280 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:54:21,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.54 % 2022-02-08 05:54:21,017 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:54:21,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:54:30,298 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.06 % 2022-02-08 05:54:30,299 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 20.06 2022-02-08 05:54:30,335 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:54:30,338 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:54:30,338 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:54:31,030 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.06 % 2022-02-08 05:54:31,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 05:54:31,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:54:40,351 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.19 % 2022-02-08 05:54:40,351 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 20.19 2022-02-08 05:54:40,384 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:54:40,388 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:54:40,388 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:54:41,040 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >20.06% in 250.718999863 seconds 2022-02-08 05:54:41,154 | test | DEBUG | MainThread | [rest_client:diag_eval:1476] /diag/eval status on 172.23.120.170:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2022-02-08 05:54:41,154 | test | INFO | MainThread | [opd:abort_rebalance:780] Abort rebalance 2022-02-08 05:54:41,167 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:54:42,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:54:42,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:54:42,703 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:54:42,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:54:42,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:54:42,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:54:42,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:54:42,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:54:42,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:54:42,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:54:42,835 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 05:54:42,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:54:42,894 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['114364 memcached\n'], []) 2022-02-08 05:54:42,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 05:54:42,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 05:54:42,964 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['114364 memcached\n'], []) 2022-02-08 05:54:42,966 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:54:42,967 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 05:54:43,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:54:43,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 05:54:43,338 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:54:43,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 05:54:43,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 05:54:43,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 05:54:43,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 05:54:43,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 05:54:43,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 05:54:43,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 05:54:43,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:54:43,453 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 05:54:43,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:54:43,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:54:47,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:54:47,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:54:48,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:54:48,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:54:48,503 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 05:54:48,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:54:48,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:54:48,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:54:48,582 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 05:54:48,584 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 05:54:48,585 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 05:54:49,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 8084 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:54:49,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 05:54:50,078 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:54:50,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 05:54:50,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 05:54:50,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 05:54:50,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 05:54:50,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 05:54:50,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 05:54:50,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 05:54:50,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:54:50,211 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 05:54:50,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:54:50,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:54:50,407 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.40 % 2022-02-08 05:54:50,407 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 21.4 2022-02-08 05:54:50,443 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:54:50,446 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:54:50,446 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:54:54,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:54:55,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:54:55,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:54:55,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:54:55,819 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 05:54:55,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:54:55,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:54:55,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:54:55,907 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 05:54:55,907 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 05:54:55,908 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 05:54:56,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:54:56,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 05:54:56,881 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:54:56,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 05:54:56,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 05:54:56,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 05:54:56,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 05:54:56,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 05:54:56,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 05:54:57,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 05:54:57,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:54:57,032 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 05:54:57,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:54:57,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:00,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:55:00,461 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.52 % 2022-02-08 05:55:00,463 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 21.52 2022-02-08 05:55:00,493 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:55:00,496 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:55:00,497 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:55:00,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:01,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:01,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:55:01,220 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 05:55:01,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:55:01,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:01,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:55:01,318 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 05:55:01,319 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 05:55:01,321 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 05:55:01,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:55:01,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 05:55:01,664 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:55:01,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 05:55:01,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 05:55:01,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 05:55:01,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 05:55:01,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 05:55:01,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 05:55:01,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 05:55:01,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:55:01,770 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 05:55:01,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:55:01,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:01,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:55:01,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:01,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:01,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:55:01,905 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 05:55:01,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:55:01,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:01,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:55:01,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:01,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:55:01,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:01,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:02,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:02,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:02,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:02,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:02,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:02,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:02,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:02,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:55:02,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:02,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:02,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:02,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:02,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:02,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:02,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:02,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:02,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:02,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:55:02,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:02,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:02,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:02,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:02,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:02,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:02,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:02,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:02,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:02,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:55:02,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:02,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:02,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:02,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:02,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:02,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:02,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:02,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:02,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:02,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:55:02,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:02,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:02,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:02,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:02,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:02,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:02,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:02,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:02,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:02,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 05:55:02,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:02,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:02,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:02,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:02,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:02,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:02,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:02,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:02,881 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 05:55:02,882 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 05:55:03,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:55:03,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 05:55:04,059 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:55:04,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 05:55:04,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 05:55:04,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 05:55:04,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 05:55:04,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 05:55:04,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 05:55:04,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 05:55:04,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:55:04,200 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 05:55:04,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:55:04,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:06,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:55:07,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:07,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:07,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:55:07,582 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 05:55:07,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:55:07,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:07,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:55:07,667 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 05:55:07,668 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 05:55:07,671 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 05:55:07,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:55:07,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 05:55:08,045 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:55:08,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 05:55:08,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 05:55:08,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 05:55:08,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 05:55:08,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 05:55:08,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 05:55:08,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 05:55:08,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:55:08,142 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 05:55:08,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:55:08,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:08,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:55:08,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:08,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:08,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:55:08,263 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 05:55:08,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:55:08,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:08,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:55:08,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:08,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:55:08,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:08,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:08,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:08,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:08,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:08,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:08,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:08,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:08,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:08,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:55:08,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:08,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:08,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:08,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:08,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:08,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:08,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:08,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:08,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:08,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:55:08,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:08,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:08,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:08,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:08,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:08,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:08,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:08,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:08,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:08,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:55:08,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:08,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:08,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:08,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:08,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:08,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:08,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:08,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:08,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:08,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:55:08,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:08,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:09,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:09,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:09,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:09,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:09,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:09,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:09,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:09,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 05:55:09,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:09,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:09,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:09,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:09,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:09,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:09,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:09,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:09,263 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 05:55:09,266 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 05:55:10,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:55:10,513 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.65 % 2022-02-08 05:55:10,513 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 23.65 2022-02-08 05:55:10,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 05:55:10,549 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:55:10,552 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:55:10,552 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:55:10,730 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:55:10,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 05:55:10,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 05:55:10,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 05:55:10,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 05:55:10,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 05:55:10,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 05:55:10,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 05:55:10,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:55:10,888 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 05:55:10,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:55:10,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:15,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:55:15,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:16,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:16,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:55:16,463 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 05:55:16,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:55:16,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:16,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:55:16,536 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 05:55:16,538 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 05:55:16,540 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 05:55:17,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:55:17,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 05:55:17,756 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:55:17,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 05:55:17,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 05:55:17,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 05:55:17,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 05:55:17,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 05:55:17,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 05:55:17,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 05:55:17,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:55:17,904 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 05:55:17,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:55:17,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:20,573 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 24.69 % 2022-02-08 05:55:20,575 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 24.69 2022-02-08 05:55:20,605 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:55:20,609 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:55:20,611 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:55:22,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:55:22,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:22,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:23,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:55:23,253 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 05:55:23,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:55:23,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:23,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:55:23,358 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 05:55:23,358 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 05:55:23,361 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 05:55:23,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:55:23,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 05:55:23,740 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:55:23,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 05:55:23,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 05:55:23,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 05:55:23,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 05:55:23,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 05:55:23,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 05:55:23,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 05:55:23,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:55:23,855 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 05:55:23,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:55:23,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:23,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:55:23,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:23,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:23,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:55:23,974 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 05:55:23,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:55:23,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:24,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:55:24,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:24,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:55:24,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:24,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:24,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:24,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:24,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:24,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:24,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:24,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:24,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:24,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:55:24,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:24,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:24,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:24,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:24,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:24,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:24,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:24,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:24,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:24,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:55:24,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:24,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:24,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:24,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:24,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:24,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:24,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:24,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:24,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:24,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:55:24,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:24,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:24,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:24,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:24,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:24,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:24,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:24,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:24,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:24,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:55:24,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:24,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:24,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:24,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:24,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:24,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:24,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:24,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:24,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:24,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 05:55:24,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:24,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:24,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:24,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:24,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:24,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:24,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:25,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:25,017 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 05:55:25,020 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 05:55:25,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:55:25,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 05:55:25,375 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:55:25,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 05:55:25,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 05:55:25,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 05:55:25,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 05:55:25,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 05:55:25,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 05:55:25,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 05:55:25,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:55:25,486 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 05:55:25,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:55:25,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:25,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:55:25,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:25,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:25,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:55:25,608 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 05:55:25,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:55:25,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:25,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:55:25,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:25,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:55:25,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:25,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:25,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:25,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:25,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:25,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:25,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:25,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:25,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:25,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:55:25,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:25,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:25,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:25,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:25,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:25,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:25,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:25,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:25,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:25,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:55:26,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:26,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:26,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:26,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:26,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:26,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:26,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:26,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:26,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:26,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:55:26,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:26,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:26,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:26,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:26,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:26,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:26,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:26,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:26,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:26,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:55:26,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:26,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:26,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:26,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:26,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:26,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:26,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:26,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:26,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:26,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 05:55:26,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:26,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:26,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:26,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:26,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:26,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:26,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:26,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:26,596 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 05:55:26,599 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 05:55:27,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:55:27,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 05:55:27,907 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:55:27,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 05:55:27,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 05:55:27,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 05:55:27,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 05:55:27,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 05:55:27,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 05:55:28,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 05:55:28,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:55:28,042 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 05:55:28,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:55:28,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:30,642 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 24.69 % 2022-02-08 05:55:30,642 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 24.69 2022-02-08 05:55:30,684 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:55:30,687 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:55:30,688 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:55:32,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:55:32,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:32,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:33,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:55:33,134 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 05:55:33,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:55:33,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:33,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:55:33,220 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 05:55:33,221 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 05:55:33,226 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 05:55:33,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:55:34,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 05:55:34,569 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:55:34,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 05:55:34,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 05:55:34,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 05:55:34,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 05:55:34,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 05:55:34,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 05:55:34,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 05:55:34,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:55:34,696 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 05:55:34,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:55:34,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:38,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:55:38,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:39,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:39,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:55:39,408 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 05:55:39,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:55:39,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:39,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:55:39,473 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 05:55:39,474 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 05:55:39,476 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 05:55:40,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:55:40,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 05:55:40,664 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:55:40,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 05:55:40,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 05:55:40,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 05:55:40,700 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.91 % 2022-02-08 05:55:40,700 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 25.91 2022-02-08 05:55:40,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 05:55:40,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 05:55:40,737 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:55:40,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 05:55:40,742 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:55:40,742 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:55:40,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 05:55:40,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:55:40,796 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 05:55:40,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:55:40,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:44,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:55:44,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:44,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:44,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:55:44,811 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 05:55:44,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:55:44,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:44,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:55:44,875 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 05:55:44,877 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 05:55:44,878 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 05:55:45,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:55:46,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 05:55:46,345 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:55:46,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 05:55:46,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 05:55:46,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 05:55:46,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 05:55:46,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 05:55:46,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 05:55:46,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 05:55:46,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:55:46,499 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 05:55:46,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:55:46,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 8111 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:55:50,759 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.84 % 2022-02-08 05:55:50,759 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 26.84 2022-02-08 05:55:50,792 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:55:50,795 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:55:50,795 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:55:51,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:55:52,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:52,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:52,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:55:52,655 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 05:55:52,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:55:52,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:52,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:55:52,723 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 05:55:52,723 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 05:55:52,726 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 05:55:52,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 05:55:53,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 05:55:53,085 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 05:55:53,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 05:55:53,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 05:55:53,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 05:55:53,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 05:55:53,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 05:55:53,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 05:55:53,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 05:55:53,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 05:55:53,197 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 05:55:53,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 05:55:53,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:53,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 05:55:53,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:53,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 05:55:53,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 05:55:53,312 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 05:55:53,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 05:55:53,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 05:55:53,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 05:55:53,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:53,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 05:55:53,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:53,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:53,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:53,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:53,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:53,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:53,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:53,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 05:55:53,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:53,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 05:55:53,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:53,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:53,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:53,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:53,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:53,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:53,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:53,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 05:55:53,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:53,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 05:55:53,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:53,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:53,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:53,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:53,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:53,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:53,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:53,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 05:55:53,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:53,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 05:55:53,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:53,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:53,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:53,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:53,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:53,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:53,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:54,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 05:55:54,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:54,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 05:55:54,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:54,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:54,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:54,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:54,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:54,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:54,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:54,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 05:55:54,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:54,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 05:55:54,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:54,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:54,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:54,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:54,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:54,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:54,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:54,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 05:55:54,341 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 05:55:54,354 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L789 2022-02-08 05:55:54,355 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2022-02-08 05:56:00,812 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.36 % 2022-02-08 05:56:00,812 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 27.36 2022-02-08 05:56:00,846 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:56:00,849 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:56:00,849 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:56:10,872 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.18 % 2022-02-08 05:56:10,872 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 28.18 2022-02-08 05:56:10,911 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:56:10,914 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:56:10,914 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:56:20,927 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.23 % 2022-02-08 05:56:20,927 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 29.23 2022-02-08 05:56:20,964 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:56:20,967 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:56:20,967 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:56:30,982 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.24 % 2022-02-08 05:56:30,982 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 30.24 2022-02-08 05:56:31,017 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:56:31,022 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:56:31,022 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:56:41,048 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.24 % 2022-02-08 05:56:41,049 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 30.24 2022-02-08 05:56:41,085 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:56:41,088 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:56:41,088 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 8136 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:56:51,101 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 31.63 % 2022-02-08 05:56:51,101 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 31.63 2022-02-08 05:56:51,137 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:56:51,140 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:56:51,140 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:56:54,371 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.120.170:8091 2022-02-08 05:56:54,371 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 05:56:54,391 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:56:54,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:56:54,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:56:56,421 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:56:56,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:56:56,436 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:56:58,451 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:56:58,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:56:58,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:00,479 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:00,487 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:00,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:01,154 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.12 % 2022-02-08 05:57:01,154 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 32.12 2022-02-08 05:57:01,181 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:57:01,184 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:57:01,184 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:57:02,506 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:02,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:02,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:04,529 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:04,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:04,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:06,553 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:06,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:06,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:08,578 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:08,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:08,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:10,601 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:10,609 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:10,609 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:11,203 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.00 % 2022-02-08 05:57:11,203 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 33.0 2022-02-08 05:57:11,234 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:57:11,236 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:57:11,237 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:57:12,667 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:12,677 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:12,677 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:14,696 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:14,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:14,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:16,720 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:16,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:16,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:18,766 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:18,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:18,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:20,795 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:20,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:20,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:21,253 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.63 % 2022-02-08 05:57:21,253 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 34.63 2022-02-08 05:57:21,286 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:57:21,289 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:57:21,289 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:57:22,819 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:22,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:22,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:24,846 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:24,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:24,857 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:26,872 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:26,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:26,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:28,895 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:28,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:28,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:30,917 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:30,926 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:30,927 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:31,302 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.24 % 2022-02-08 05:57:31,302 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 35.24 2022-02-08 05:57:31,334 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:57:31,336 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:57:31,338 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:57:32,944 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:32,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:32,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:34,967 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:34,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:34,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:36,993 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:37,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:37,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:39,029 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:39,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:39,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:41,055 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:41,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:41,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:41,351 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.37 % 2022-02-08 05:57:41,351 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 35.37 2022-02-08 05:57:41,381 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:57:41,384 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:57:41,384 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:57:43,078 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:43,086 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:43,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:45,101 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:45,109 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:45,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:47,127 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:47,135 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:47,135 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:49,154 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:49,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:49,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 5 queries submitted, 0 failed, 8158 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:57:51,180 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:51,187 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:51,188 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:51,394 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.37 % 2022-02-08 05:57:51,394 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 35.37 2022-02-08 05:57:51,428 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:57:51,430 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:57:51,431 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:57:53,207 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:53,217 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:53,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:55,233 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:55,242 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:55,242 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:57,257 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:57,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:57,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:57:59,283 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:57:59,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:57:59,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:01,315 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:01,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:01,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:01,446 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.98 % 2022-02-08 05:58:01,446 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 35.98 2022-02-08 05:58:01,480 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:58:01,483 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:58:01,483 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:58:03,342 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:03,351 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:03,351 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:05,365 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:05,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:05,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:07,394 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:07,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:07,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:09,420 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:09,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:09,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:11,444 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:11,451 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:11,451 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:11,496 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.92 % 2022-02-08 05:58:11,497 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 36.92 2022-02-08 05:58:11,536 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:58:11,539 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:58:11,539 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:58:13,466 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:13,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:13,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:15,493 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:15,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:15,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:17,519 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:17,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:17,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:19,542 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:19,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:19,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:21,558 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.85 % 2022-02-08 05:58:21,558 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 37.85 2022-02-08 05:58:21,572 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:21,591 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:58:21,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:21,592 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:21,592 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:58:21,594 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:58:23,611 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:23,627 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:23,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:25,663 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:25,673 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:25,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:27,694 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:27,701 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:27,703 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:29,716 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:29,724 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:29,724 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:31,628 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.31 % 2022-02-08 05:58:31,628 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 39.31 2022-02-08 05:58:31,673 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:58:31,676 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:58:31,677 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:58:31,740 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:31,747 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:31,747 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:33,769 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:33,778 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:33,778 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:35,790 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:35,799 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:35,799 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:37,812 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:37,821 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:37,821 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:39,836 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:39,846 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:39,846 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:41,694 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.43 % 2022-02-08 05:58:41,696 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 39.43 2022-02-08 05:58:41,733 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:58:41,736 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:58:41,736 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:58:41,869 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:41,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:41,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:43,898 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:43,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:43,908 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:45,921 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:45,928 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:45,928 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:47,946 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:47,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:47,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 5 queries submitted, 0 failed, 8175 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:58:49,969 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:49,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:49,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:51,756 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.80 % 2022-02-08 05:58:51,756 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 39.8 2022-02-08 05:58:51,793 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:58:51,796 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:58:51,796 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:58:51,989 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:51,997 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:51,997 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:54,009 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:54,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:54,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:56,036 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:56,045 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:56,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:58:58,058 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:58:58,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:58:58,066 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:00,078 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:00,086 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:00,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:01,812 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.86 % 2022-02-08 05:59:01,813 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 40.86 2022-02-08 05:59:01,846 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:59:01,849 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:59:01,849 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:59:02,099 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:02,107 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:02,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:04,121 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:04,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:04,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:06,148 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:06,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:06,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:08,177 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:08,194 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:08,194 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:10,209 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:10,217 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:10,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:11,861 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 43.77 % 2022-02-08 05:59:11,862 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 43.77 2022-02-08 05:59:11,892 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:59:11,895 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:59:11,895 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:59:12,232 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:12,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:12,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:14,253 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:14,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:14,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:16,282 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:16,298 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:16,298 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:18,312 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:18,321 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:18,321 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:20,332 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:20,341 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:20,341 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:21,920 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 46.61 % 2022-02-08 05:59:21,920 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 46.61 2022-02-08 05:59:21,963 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:59:21,964 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:59:21,966 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:59:22,352 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:22,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:22,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:24,375 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:24,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:24,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:26,397 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:26,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:26,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:28,423 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:28,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:28,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:30,450 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:30,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:30,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:31,986 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.18 % 2022-02-08 05:59:31,986 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 47.18 2022-02-08 05:59:32,028 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:59:32,030 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:59:32,032 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:59:32,489 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:32,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:32,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:34,525 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:34,533 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:34,533 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:36,555 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:36,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:36,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:38,582 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:38,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:38,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:40,611 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:40,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:40,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:42,045 | test | DEBUG | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 48.89 % 2022-02-08 05:59:42,045 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance - status: running, progress: 48.89 2022-02-08 05:59:42,075 | infra | DEBUG | pool-3-thread-19 | [task:check:441] Wait before next rebalance progress check 2022-02-08 05:59:42,078 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 05:59:42,078 | infra | DEBUG | pool-3-thread-19 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 05:59:42,638 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:42,645 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:42,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:44,661 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:44,670 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:44,670 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:46,686 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:46,696 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:46,696 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:48,710 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:48,719 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:48,720 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 5 queries submitted, 0 failed, 8202 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 05:59:50,734 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:50,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:50,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:52,088 | test | ERROR | pool-3-thread-19 | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'd26ff90bd5fa1657f4a38236decbc3d3', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=03ebb229ae601638fc5b2af0610de8db', u'status': u'notRunning'} - rebalance failed 2022-02-08 05:59:52,115 | test | INFO | pool-3-thread-19 | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.120.170: 2022-02-08 05:59:52,115 | test | ERROR | pool-3-thread-19 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644328788078L, u'shortText': u'message', u'serverTime': u'2022-02-08T05:59:48.078Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.18425.105>,\n {{bulk_set_vbucket_state_failed,\n [{\'ns_1@172.23.120.170\',\n {\'EXIT\',\n {{{{badmatch,{error,timeout}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,151}]},\n {mc_client_binary,create_bucket,5,\n [{file,"src/mc_client_binary.erl"},\n {line,209}]},\n {ns_memcached,do_ensure_bucket,4,\n [{file,"src/ns_memcached.erl"},\n {line,1355}]},\n {ns_memcached,ensure_bucket_inner,3,\n [{file,"src/ns_memcached.erl"},\n {line,1319}]},\n {ns_memcached,handle_info,2,\n [{file,"src/ns_memcached.erl"},\n {line,695}]},\n {gen_server,try_dispatch,4,\n [{file,"gen_server.erl"},\n {line,695}]},\n {gen_server,handle_msg,6,\n [{file,"gen_server.erl"},\n {line,771}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},\n {line,226}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {set_vbucket,808,replica,undefined},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.120.170\'},\n {if_rebalance,<0.19347.103>,\n {update_vbucket_state,808,replica,\n undefined,\'ns_1@172.23.121.136\'}},\n infinity]}}}}]},\n [{janitor_agent,bulk_set_vbucket_state,4,\n [{file,"src/janitor_agent.erl"},\n {line,372}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,211}]}]}}}}.\nRebalance Operation Id = 037ae53d3d6e777ca9d6c3ee220c77be'} 2022-02-08 05:59:52,115 | test | ERROR | pool-3-thread-19 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644328788054L, u'shortText': u'message', u'serverTime': u'2022-02-08T05:59:48.054Z', u'text': u'Worker <0.18659.105> (for action {move,\n {808,\n [\'ns_1@172.23.121.136\',\n \'ns_1@172.23.120.170\'],\n [\'ns_1@172.23.121.128\',\n \'ns_1@172.23.120.170\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.18425.105>,\n {{bulk_set_vbucket_state_failed,\n [{\'ns_1@172.23.120.170\',\n {\'EXIT\',\n {{{{badmatch,\n {error,\n timeout}},\n [{mc_client_binary,\n cmd_vocal_recv,\n 5,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 151}]},\n {mc_client_binary,\n create_bucket,\n 5,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 209}]},\n {ns_memcached,\n do_ensure_bucket,\n 4,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 1355}]},\n {ns_memcached,\n ensure_bucket_inner,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 1319}]},\n {ns_memcached,\n handle_info,\n 2,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 695}]},\n {gen_server,\n try_dispatch,\n 4,\n [{file,\n "gen_server.erl"},\n {line,\n 695}]},\n {gen_server,\n handle_msg,\n 6,\n [{file,\n "gen_server.erl"},\n {line,\n 771}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 226}]}]},\n {gen_server,\n call,\n [\'ns_memcached-GleamBookUsers0\',\n {set_vbucket,\n 808,\n replica,\n undefined},\n 300000]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.120.170\'},\n {if_rebalance,\n <0.19347.103>,\n {update_vbucket_state,\n 808,\n replica,\n undefined,\n \'ns_1@172.23.121.136\'}},\n infinity]}}}}]},\n [{janitor_agent,\n bulk_set_vbucket_state,\n 4,\n [{file,\n "src/janitor_agent.erl"},\n {line,\n 372}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 211}]}]}}}'} 2022-02-08 05:59:52,117 | test | ERROR | pool-3-thread-19 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644328484030L, u'shortText': u'message', u'serverTime': u'2022-02-08T05:54:44.030Z', u'text': u"Control connection to memcached on 'ns_1@172.23.120.170' disconnected. Check logs for details."} 2022-02-08 05:59:52,117 | test | ERROR | pool-3-thread-19 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644328483838L, u'shortText': u'message', u'serverTime': u'2022-02-08T05:54:43.838Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW0208 05:43:04.785943 114420 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2022-02-08 05:59:52,118 | test | ERROR | pool-3-thread-19 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644328150258L, u'shortText': u'message', u'serverTime': u'2022-02-08T05:49:10.258Z', u'text': u'Bucket "GleamBookUsers0" rebalance appears to be swap rebalance'} 2022-02-08 05:59:52,118 | test | ERROR | pool-3-thread-19 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.128', u'tstamp': 1644328148896L, u'shortText': u'message', u'serverTime': u'2022-02-08T05:49:08.896Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.121.128\' in 0 seconds.'} 2022-02-08 05:59:52,118 | test | ERROR | pool-3-thread-19 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644328148410L, u'shortText': u'message', u'serverTime': u'2022-02-08T05:49:08.410Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2022-02-08 05:59:52,118 | test | ERROR | pool-3-thread-19 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_storage_conf', u'type': u'info', u'node': u'ns_1@172.23.121.128', u'tstamp': 1644328148295L, u'shortText': u'message', u'serverTime': u'2022-02-08T05:49:08.295Z', u'text': u'Deleting old data files of bucket "GleamBookUsers0"'} 2022-02-08 05:59:52,118 | test | ERROR | pool-3-thread-19 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644328147377L, u'shortText': u'message', u'serverTime': u'2022-02-08T05:49:07.377Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.121.128','ns_1@172.23.121.115',\n 'ns_1@172.23.121.132','ns_1@172.23.121.135',\n 'ns_1@172.23.120.170','ns_1@172.23.121.129',\n 'ns_1@172.23.121.124','ns_1@172.23.121.126',\n 'ns_1@172.23.121.134','ns_1@172.23.121.116',\n 'ns_1@172.23.121.127','ns_1@172.23.121.123',\n 'ns_1@172.23.121.131','ns_1@172.23.121.130',\n 'ns_1@172.23.121.133'], EjectNodes = ['ns_1@172.23.121.136'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 037ae53d3d6e777ca9d6c3ee220c77be"} 2022-02-08 05:59:52,118 | test | ERROR | pool-3-thread-19 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.121.128', u'tstamp': 1644328147327L, u'shortText': u'message', u'serverTime': u'2022-02-08T05:49:07.327Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_storage_threads">>,\n <<"num_writer_threads">>,\n <<"scramsha_fallback_salt">>]'} 2022-02-08 05:59:52,118 | test | ERROR | pool-3-thread-19 | [task:call:325] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'd26ff90bd5fa1657f4a38236decbc3d3', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=03ebb229ae601638fc5b2af0610de8db', u'status': u'notRunning'} - rebalance failed 2022-02-08 05:59:52,760 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:52,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:52,769 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:54,788 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:54,801 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:54,801 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:56,815 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:56,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:56,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 05:59:58,845 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 05:59:58,854 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 05:59:58,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:00,871 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:00,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:00,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:02,894 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:02,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:02,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:04,920 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:04,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:04,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:06,947 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:06,956 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:06,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:08,977 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:08,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:08,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:11,003 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:11,012 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:11,012 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:13,035 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:13,043 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:13,045 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:15,059 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:15,068 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:15,068 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:17,081 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:17,089 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:17,089 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:19,107 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:19,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:19,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:21,135 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:21,147 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:21,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:23,163 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:23,173 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:23,173 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:25,190 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:25,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:25,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:27,213 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:27,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:27,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:29,236 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:29,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:29,244 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:31,262 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:31,270 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:31,272 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:33,290 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:33,301 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:33,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:35,315 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:35,323 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:35,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:37,346 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:37,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:37,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:39,375 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:39,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:39,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:41,398 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:41,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:41,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:43,421 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:43,428 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:43,428 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:45,444 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.120.170:8091 status is warmup 2022-02-08 06:00:45,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2022-02-08 06:00:45,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2022-02-08 06:00:47,467 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.120.170:8091 is running 2022-02-08 06:00:47,480 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.126:8091 2022-02-08 06:00:47,480 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 06:00:47,493 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.126:8091 is running 2022-02-08 06:00:47,509 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.128:8091 2022-02-08 06:00:47,509 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 06:00:47,526 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.128:8091 is running 2022-02-08 06:00:47,581 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.133:8091 2022-02-08 06:00:47,581 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 06:00:47,595 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.133:8091 is running 2022-02-08 06:00:47,612 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.135:8091 2022-02-08 06:00:47,612 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 06:00:47,628 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.135:8091 is running 2022-02-08 06:00:47,642 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.134:8091 2022-02-08 06:00:47,644 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 06:00:47,658 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.134:8091 is running 2022-02-08 06:00:47,674 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.127:8091 2022-02-08 06:00:47,674 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 06:00:47,687 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.127:8091 is running 2022-02-08 06:00:47,701 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.124:8091 2022-02-08 06:00:47,701 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 06:00:47,716 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.124:8091 is running 2022-02-08 06:00:47,730 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.130:8091 2022-02-08 06:00:47,730 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 06:00:47,746 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.130:8091 is running 2022-02-08 06:00:47,760 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.132:8091 2022-02-08 06:00:47,760 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 06:00:47,776 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.132:8091 is running 2022-02-08 06:00:47,792 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.123:8091 2022-02-08 06:00:47,792 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 06:00:47,808 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.123:8091 is running 2022-02-08 06:00:47,822 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.115:8091 2022-02-08 06:00:47,823 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 06:00:47,835 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.115:8091 is running 2022-02-08 06:00:47,851 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.116:8091 2022-02-08 06:00:47,852 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 06:00:47,867 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.116:8091 is running 2022-02-08 06:00:47,882 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.129:8091 2022-02-08 06:00:47,882 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 06:00:47,895 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.129:8091 is running 2022-02-08 06:00:47,911 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.121.131:8091 2022-02-08 06:00:47,911 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2022-02-08 06:00:47,923 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.121.131:8091 is running 2022-02-08 06:00:47,926 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:00:48,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:00:48,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:00:49,023 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:00:49,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:00:49,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:00:49,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:00:49,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:00:49,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:00:49,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:00:49,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:00:49,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:00:49,164 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 06:00:49,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:00:49,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 8224 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:00:49,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:00:50,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:00:50,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:00:50,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:00:50,769 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 06:00:50,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:00:50,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:00:50,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:00:50,848 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 06:00:50,848 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:00:50,851 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:00:50,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:00:51,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:00:51,203 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:00:51,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:00:51,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:00:51,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:00:51,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:00:51,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:00:51,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:00:51,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:00:51,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:00:51,319 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 06:00:51,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:00:51,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:00:51,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:00:52,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:00:52,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:00:52,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:00:52,835 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 06:00:52,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:00:52,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:00:52,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:00:52,878 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 06:00:52,878 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:00:52,881 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:00:53,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:00:53,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:00:53,269 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:00:53,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:00:53,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:00:53,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:00:53,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:00:53,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:00:53,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:00:53,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:00:53,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:00:53,394 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 06:00:53,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:00:53,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:00:53,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:00:54,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:00:54,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:00:54,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:00:54,523 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 06:00:54,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:00:54,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:00:54,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:00:54,565 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 06:00:54,565 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:00:54,566 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 06:00:54,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:00:54,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 06:00:54,993 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:00:54,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 06:00:55,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 06:00:55,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 06:00:55,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 06:00:55,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 06:00:55,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 06:00:55,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 06:00:55,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:00:55,117 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 06:00:55,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:00:55,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:00:55,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:00:55,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:00:55,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:00:55,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:00:55,242 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 06:00:55,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:00:55,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:00:55,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:00:55,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:55,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:00:55,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:55,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:55,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:55,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:55,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:55,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:55,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:55,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:55,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:55,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:00:55,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:55,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:55,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:55,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:55,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:55,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:55,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:55,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:55,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:55,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:00:55,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:55,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:55,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:55,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:55,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:55,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:55,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:55,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:55,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:00:55,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:00:55,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:00:55,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:00:55,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:00:55,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:00:55,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:00:55,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:00:55,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:00:56,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:00:56,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:00:56,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:00:56,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:00:56,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:00:56,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:00:56,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:00:56,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:00:56,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:00:56,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:00:56,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:00:56,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:00:56,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:00:56,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:00:56,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:00:56,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:00:56,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:00:56,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:00:56,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:00:56,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:00:56,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:00:56,388 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 06:00:56,391 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:00:56,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:00:57,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:00:57,095 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:00:57,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:00:57,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:00:57,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:00:57,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:00:57,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:00:57,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:00:57,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:00:57,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:00:57,243 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 06:00:57,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:00:57,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:00:57,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:00:57,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:00:58,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:00:58,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:00:58,526 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 06:00:58,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:00:58,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:00:58,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:00:58,647 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 06:00:58,647 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:00:58,650 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 06:00:58,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:00:58,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 06:00:59,033 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:00:59,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 06:00:59,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 06:00:59,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 06:00:59,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 06:00:59,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 06:00:59,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 06:00:59,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 06:00:59,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:00:59,153 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 06:00:59,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:00:59,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:00:59,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:00:59,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:00:59,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:00:59,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:00:59,266 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 06:00:59,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:00:59,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:00:59,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:00:59,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:59,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:00:59,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:59,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:59,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:59,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:59,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:59,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:59,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:59,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:00:59,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:59,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:00:59,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:59,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:59,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:59,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:59,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:59,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:59,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:59,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:00:59,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:59,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:00:59,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:59,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:59,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:59,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:59,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:59,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:59,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:59,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:00:59,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:00:59,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:00:59,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:00:59,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:00:59,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:00:59,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:00:59,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:00,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:00,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:00,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:00,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:00,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:01:00,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:00,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:00,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:00,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:00,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:00,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:00,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:00,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:00,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:00,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:01:00,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:00,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:00,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:00,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:00,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:00,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:00,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:00,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:00,407 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 06:01:00,408 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:01:00,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:01:00,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:01:00,746 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:01:00,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:01:00,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:01:00,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:01:00,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:01:00,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:01:00,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:01:00,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:01:00,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:01:00,868 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 06:01:00,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:01:00,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:01,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:01:01,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:02,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:02,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:01:02,497 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 06:01:02,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:01:02,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:02,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:01:02,542 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 06:01:02,542 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:01:02,545 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:01:02,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:01:02,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:01:02,941 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:01:02,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:01:02,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:01:02,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:01:02,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:01:03,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:01:03,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:01:03,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:01:03,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:01:03,084 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 06:01:03,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:01:03,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:03,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:01:04,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:04,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:04,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:01:04,545 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 06:01:04,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:01:04,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:04,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:01:04,591 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 06:01:04,591 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:01:04,594 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 06:01:04,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:01:04,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 06:01:04,977 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:01:04,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 06:01:04,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 06:01:05,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 06:01:05,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 06:01:05,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 06:01:05,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 06:01:05,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 06:01:05,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:01:05,088 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 06:01:05,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:01:05,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:05,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:01:05,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:05,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:05,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:01:05,194 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 06:01:05,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:01:05,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:05,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:01:05,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:05,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:01:05,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:05,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:05,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:05,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:05,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:05,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:05,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:05,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:05,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:05,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:01:05,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:05,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:05,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:05,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:05,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:05,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:05,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:05,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:05,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:05,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:01:05,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:05,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:05,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:05,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:05,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:05,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:05,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:05,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:05,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:05,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:01:05,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:05,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:05,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:05,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:05,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:05,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:05,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:05,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:05,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:05,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:01:05,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:05,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:05,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:06,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:06,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:06,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:06,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:06,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:06,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:06,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:01:06,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:06,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:06,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:06,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:06,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:06,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:06,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:06,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:06,260 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 06:01:06,263 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 06:01:06,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:01:06,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 06:01:06,644 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:01:06,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 06:01:06,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 06:01:06,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 06:01:06,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 06:01:06,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 06:01:06,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 06:01:06,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 06:01:06,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:01:06,766 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 06:01:06,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:01:06,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:06,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:01:06,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:06,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:06,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:01:06,887 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 06:01:06,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:01:06,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:06,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:01:06,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:06,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:01:06,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:06,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:07,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:07,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:07,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:07,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:07,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:07,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:07,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:07,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:01:07,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:07,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:07,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:07,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:07,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:07,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:07,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:07,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:07,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:07,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:01:07,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:07,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:07,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:07,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:07,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:07,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:07,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:07,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:07,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:07,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:01:07,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:07,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:07,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:07,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:07,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:07,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:07,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:07,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:07,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:07,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:01:07,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:07,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:07,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:07,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:07,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:07,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:07,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:07,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:07,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:07,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:01:07,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:07,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:07,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:07,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:07,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:07,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:07,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:08,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:08,033 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 06:01:08,036 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:01:08,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:01:08,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:01:08,482 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:01:08,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:01:08,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:01:08,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:01:08,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:01:08,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:01:08,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:01:08,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:01:08,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:01:08,640 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 06:01:08,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:01:08,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:09,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:01:09,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:09,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:10,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:01:10,230 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 06:01:10,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:01:10,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:10,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:01:10,332 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 06:01:10,332 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:01:10,335 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:01:10,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:01:10,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:01:10,782 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:01:10,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:01:10,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:01:10,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:01:10,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:01:10,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:01:10,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:01:11,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:01:11,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:01:11,138 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 06:01:11,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:01:11,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:11,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:01:12,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:12,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:12,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:01:12,522 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 06:01:12,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:01:12,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:12,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:01:12,569 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 06:01:12,571 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:01:12,572 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:01:13,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:01:13,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:01:13,561 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:01:13,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:01:13,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:01:13,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:01:13,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:01:13,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:01:13,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:01:13,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:01:13,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:01:13,720 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 06:01:13,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:01:13,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:14,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:01:14,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:14,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:14,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:01:15,023 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 06:01:15,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:01:15,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:15,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:01:15,165 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 06:01:15,165 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:01:15,168 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:01:15,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:01:15,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:01:15,717 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:01:15,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:01:15,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:01:15,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:01:15,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:01:15,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:01:15,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:01:15,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:01:15,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:01:15,999 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 06:01:16,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:01:16,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:16,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:01:17,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:17,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:18,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:01:18,079 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 06:01:18,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:01:18,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:18,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:01:18,171 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 06:01:18,171 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:01:18,174 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 06:01:18,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:01:18,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 06:01:18,572 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:01:18,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 06:01:18,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 06:01:18,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 06:01:18,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 06:01:18,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 06:01:18,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 06:01:18,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 06:01:18,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:01:18,716 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 06:01:18,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:01:18,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:18,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:01:18,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:18,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:01:18,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:01:18,865 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 06:01:18,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:01:18,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:01:18,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:01:18,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:18,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:01:18,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:18,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:19,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:19,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:19,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:19,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:19,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:19,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:01:19,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:19,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:01:19,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:19,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:19,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:19,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:19,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:19,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:19,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:19,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:01:19,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:19,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:01:19,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:19,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:19,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:19,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:19,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:19,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:19,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:19,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:01:19,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:19,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:01:19,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:19,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:19,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:19,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:19,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:19,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:19,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:19,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:01:19,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:19,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:01:19,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:19,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:19,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:19,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:19,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:19,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:19,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:19,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:01:19,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:19,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:01:19,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:19,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:19,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:20,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:20,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:20,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:20,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:20,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:01:20,101 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 06:01:20,101 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[172.23.121.128]_OUT=[172.23.121.136]_1644328140.06 2022-02-08 06:01:20,101 | test | INFO | MainThread | [opd:abort_rebalance:808] Restarting Rebalance after killing at 20 2022-02-08 06:01:20,168 | infra | DEBUG | MainThread | [task:__init__:249] Last known rebalance status_id: d26ff90bd5fa1657f4a38236decbc3d3 2022-02-08 06:01:20,170 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.121.136]_1644328880.1 2022-02-08 06:01:20,170 | infra | INFO | pool-3-thread-12 | [task:start_task:91] Thread 'Rebalance_task_IN=[]_OUT=[172.23.121.136]_1644328880.1' started 2022-02-08 06:01:20,178 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L813 2022-02-08 06:01:20,180 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2022-02-08 06:01:20,289 | test | DEBUG | pool-3-thread-12 | [task:start_rebalance:384] Removing node 172.23.121.136:8091 from cluster 2022-02-08 06:01:20,289 | test | DEBUG | pool-3-thread-12 | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.128,ns_1@172.23.121.115,ns_1@172.23.121.132,ns_1@172.23.121.136,ns_1@172.23.121.135,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.134,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123,ns_1@172.23.121.131,ns_1@172.23.121.130,ns_1@172.23.121.133', 'ejectedNodes': u'ns_1@172.23.121.136', 'user': 'Administrator'} 2022-02-08 06:01:20,543 | test | DEBUG | pool-3-thread-12 | [rest_client:rebalance:1459] Rebalance operation started 2022-02-08 06:01:20,553 | test | INFO | pool-3-thread-12 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.121.128 | kv | 7.1.0-2255-enterprise | 43.8276299113 | Cluster node | | 172.23.121.115 | kv | 7.1.0-2255-enterprise | 54.1308741789 | Cluster node | | 172.23.121.132 | index, n1ql | 7.1.0-2255-enterprise | 15.208928108 | Cluster node | | 172.23.121.136 | kv | 7.1.0-2255-enterprise | 74.6785984371 | --- OUT ---> | | 172.23.121.135 | kv | 7.1.0-2255-enterprise | 73.6616161616 | Cluster node | | 172.23.120.170 | kv | 7.1.0-2255-enterprise | 11.4948387919 | Cluster node | | 172.23.121.129 | kv | 7.1.0-2255-enterprise | 35.3567355215 | Cluster node | | 172.23.121.124 | kv | 7.1.0-2255-enterprise | 30.4023279352 | Cluster node | | 172.23.121.126 | kv | 7.1.0-2255-enterprise | 27.0253164557 | Cluster node | | 172.23.121.134 | index, n1ql | 7.1.0-2255-enterprise | 10.3648317649 | Cluster node | | 172.23.121.116 | kv | 7.1.0-2255-enterprise | 75.7613893783 | Cluster node | | 172.23.121.127 | kv | 7.1.0-2255-enterprise | 30.0227732794 | Cluster node | | 172.23.121.123 | kv | 7.1.0-2255-enterprise | 38.7259858443 | Cluster node | | 172.23.121.131 | index, n1ql | 7.1.0-2255-enterprise | 13.5344160866 | Cluster node | | 172.23.121.130 | index, n1ql | 7.1.0-2255-enterprise | 17.8850156087 | Cluster node | | 172.23.121.133 | index, n1ql | 7.1.0-2255-enterprise | 12.1621621622 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2022-02-08 06:01:20,575 | infra | DEBUG | pool-3-thread-12 | [task:call:316] New rebalance status_id: 2b0983d6f08c9f2ee4b3718dc741948e 2022-02-08 06:01:20,588 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 06:01:20,588 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 06:01:20,654 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:01:20,657 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:01:20,657 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:01:30,671 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 06:01:30,673 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 06:01:30,711 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:01:30,716 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:01:30,716 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:01:40,727 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.13 % 2022-02-08 06:01:40,727 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 2.13 2022-02-08 06:01:40,767 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:01:40,770 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:01:40,770 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 8248 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:01:50,785 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.69 % 2022-02-08 06:01:50,786 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 3.69 2022-02-08 06:01:50,822 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:01:50,825 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:01:50,825 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:02:00,845 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.93 % 2022-02-08 06:02:00,845 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 7.93 2022-02-08 06:02:00,878 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:02:00,881 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:02:00,881 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:02:10,894 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.50 % 2022-02-08 06:02:10,894 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 8.5 2022-02-08 06:02:10,937 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:02:10,938 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:02:10,938 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:02:20,953 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.17 % 2022-02-08 06:02:20,953 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 12.17 2022-02-08 06:02:20,996 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:02:20,997 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:02:20,997 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:02:31,016 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.73 % 2022-02-08 06:02:31,016 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 13.73 2022-02-08 06:02:31,062 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:02:31,065 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:02:31,065 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:02:41,082 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.73 % 2022-02-08 06:02:41,084 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 13.73 2022-02-08 06:02:41,121 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:02:41,124 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:02:41,124 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 8271 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:02:51,138 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.73 % 2022-02-08 06:02:51,138 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 13.73 2022-02-08 06:02:51,187 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:02:51,190 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:02:51,190 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:03:01,211 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.73 % 2022-02-08 06:03:01,213 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 13.73 2022-02-08 06:03:01,259 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:03:01,262 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:03:01,262 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:03:11,278 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.30 % 2022-02-08 06:03:11,278 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 14.3 2022-02-08 06:03:11,313 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:03:11,316 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:03:11,316 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:03:20,198 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.30 % 2022-02-08 06:03:20,198 | test | INFO | MainThread | [opd:abort_rebalance:814] Rebalance % = 14.3 2022-02-08 06:03:20,223 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.30 % 2022-02-08 06:03:20,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:03:20,232 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:03:21,338 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.30 % 2022-02-08 06:03:21,338 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 14.3 2022-02-08 06:03:21,384 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:03:21,388 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:03:21,388 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:03:30,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.84 % 2022-02-08 06:03:30,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:03:30,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:03:31,405 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.84 % 2022-02-08 06:03:31,405 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 14.84 2022-02-08 06:03:31,460 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:03:31,463 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:03:31,463 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:03:40,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.84 % 2022-02-08 06:03:40,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:03:40,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:03:41,476 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.84 % 2022-02-08 06:03:41,476 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 14.84 2022-02-08 06:03:41,528 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:03:41,530 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:03:41,532 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 8294 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:03:50,308 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.38 % 2022-02-08 06:03:50,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:03:50,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:03:51,551 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.38 % 2022-02-08 06:03:51,552 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 15.38 2022-02-08 06:03:51,595 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:03:51,598 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:03:51,598 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:04:00,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.38 % 2022-02-08 06:04:00,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:04:00,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:04:01,611 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.38 % 2022-02-08 06:04:01,612 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 15.38 2022-02-08 06:04:01,654 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:04:01,657 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:04:01,657 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:04:10,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.38 % 2022-02-08 06:04:10,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:04:10,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:04:11,671 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.38 % 2022-02-08 06:04:11,671 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 15.38 2022-02-08 06:04:11,717 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:04:11,720 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:04:11,720 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:04:20,392 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.38 % 2022-02-08 06:04:20,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:04:20,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:04:21,750 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.38 % 2022-02-08 06:04:21,750 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 15.38 2022-02-08 06:04:21,802 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:04:21,806 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:04:21,806 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:04:30,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.95 % 2022-02-08 06:04:30,436 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:04:30,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:04:31,848 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.95 % 2022-02-08 06:04:31,848 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 16.95 2022-02-08 06:04:31,905 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:04:31,908 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:04:31,908 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:04:40,466 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.17 % 2022-02-08 06:04:40,474 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:04:40,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:04:41,931 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.17 % 2022-02-08 06:04:41,931 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 19.17 2022-02-08 06:04:41,994 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:04:41,996 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:04:41,997 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 8317 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:04:50,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.17 % 2022-02-08 06:04:50,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:04:50,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:04:52,023 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.17 % 2022-02-08 06:04:52,023 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 19.17 2022-02-08 06:04:52,072 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:04:52,078 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:04:52,078 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:05:00,519 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.17 % 2022-02-08 06:05:00,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:05:00,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:05:02,098 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.17 % 2022-02-08 06:05:02,099 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 19.17 2022-02-08 06:05:02,148 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:05:02,151 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:05:02,151 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:05:10,548 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.71 % 2022-02-08 06:05:10,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:05:10,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:05:12,170 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.26 % 2022-02-08 06:05:12,170 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 20.26 2022-02-08 06:05:12,214 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:05:12,217 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:05:12,219 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:05:20,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.26 % 2022-02-08 06:05:20,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:05:20,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:05:22,237 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.26 % 2022-02-08 06:05:22,237 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 20.26 2022-02-08 06:05:22,288 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:05:22,290 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:05:22,290 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:05:30,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.37 % 2022-02-08 06:05:30,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:05:30,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:05:32,309 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.37 % 2022-02-08 06:05:32,309 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 21.37 2022-02-08 06:05:32,388 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:05:32,394 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:05:32,394 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:05:40,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.91 % 2022-02-08 06:05:40,686 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:05:40,687 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:05:42,408 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.91 % 2022-02-08 06:05:42,408 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 21.91 2022-02-08 06:05:42,451 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:05:42,453 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:05:42,453 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 8343 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:05:50,717 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.91 % 2022-02-08 06:05:50,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:05:50,727 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:05:52,470 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.91 % 2022-02-08 06:05:52,470 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 21.91 2022-02-08 06:05:52,520 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:05:52,523 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:05:52,523 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:06:00,747 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.91 % 2022-02-08 06:06:00,755 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:06:00,755 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:06:02,536 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.91 % 2022-02-08 06:06:02,536 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 21.91 2022-02-08 06:06:02,573 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:06:02,576 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:06:02,576 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:06:10,780 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.48 % 2022-02-08 06:06:10,792 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:06:10,792 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:06:12,611 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.48 % 2022-02-08 06:06:12,611 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 22.48 2022-02-08 06:06:12,686 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:06:12,688 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:06:12,688 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:06:20,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.02 % 2022-02-08 06:06:20,857 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:06:20,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:06:22,710 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.02 % 2022-02-08 06:06:22,720 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 23.02 2022-02-08 06:06:22,770 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:06:22,773 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:06:22,773 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:06:30,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.02 % 2022-02-08 06:06:30,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:06:30,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:06:32,792 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.02 % 2022-02-08 06:06:32,792 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 23.02 2022-02-08 06:06:32,845 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:06:32,849 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:06:32,849 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:06:40,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.02 % 2022-02-08 06:06:40,933 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:06:40,933 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:06:42,869 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.02 % 2022-02-08 06:06:42,869 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 23.02 2022-02-08 06:06:42,917 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:06:42,918 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:06:42,920 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 8363 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:06:50,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.57 % 2022-02-08 06:06:50,966 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:06:50,966 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:06:52,944 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.57 % 2022-02-08 06:06:52,946 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 23.57 2022-02-08 06:06:53,000 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:06:53,003 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:06:53,003 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:07:00,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 24.70 % 2022-02-08 06:07:00,993 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:07:00,994 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:07:03,025 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.27 % 2022-02-08 06:07:03,026 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 25.27 2022-02-08 06:07:03,075 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:07:03,078 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:07:03,078 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:07:11,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.82 % 2022-02-08 06:07:11,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:07:11,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:07:13,095 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.82 % 2022-02-08 06:07:13,095 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 25.82 2022-02-08 06:07:13,155 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:07:13,174 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:07:13,174 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:07:21,045 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.82 % 2022-02-08 06:07:21,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:07:21,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:07:23,194 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.36 % 2022-02-08 06:07:23,194 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 26.36 2022-02-08 06:07:23,240 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:07:23,244 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:07:23,244 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:07:31,081 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.90 % 2022-02-08 06:07:31,089 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:07:31,089 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:07:33,259 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.90 % 2022-02-08 06:07:33,259 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 26.9 2022-02-08 06:07:33,308 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:07:33,312 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:07:33,312 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:07:41,117 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.90 % 2022-02-08 06:07:41,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:07:41,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:07:43,323 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.45 % 2022-02-08 06:07:43,323 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 27.45 2022-02-08 06:07:43,354 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:07:43,355 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:07:43,355 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 8387 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:07:51,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.45 % 2022-02-08 06:07:51,158 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:07:51,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:07:53,369 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.45 % 2022-02-08 06:07:53,371 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 27.45 2022-02-08 06:07:53,430 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:07:53,434 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:07:53,434 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:08:01,173 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.01 % 2022-02-08 06:08:01,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:08:01,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:08:03,461 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.01 % 2022-02-08 06:08:03,461 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 28.01 2022-02-08 06:08:03,494 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:08:03,497 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:08:03,497 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:08:11,198 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.56 % 2022-02-08 06:08:11,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:08:11,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:08:13,519 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.10 % 2022-02-08 06:08:13,519 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 29.1 2022-02-08 06:08:13,568 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:08:13,571 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:08:13,571 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:08:21,227 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.10 % 2022-02-08 06:08:21,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:08:21,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:08:23,585 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.10 % 2022-02-08 06:08:23,585 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 29.1 2022-02-08 06:08:23,628 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:08:23,632 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:08:23,632 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:08:31,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.21 % 2022-02-08 06:08:31,267 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:08:31,267 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:08:33,651 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.21 % 2022-02-08 06:08:33,653 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 30.21 2022-02-08 06:08:33,694 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:08:33,697 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:08:33,697 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:08:41,288 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.21 % 2022-02-08 06:08:41,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:08:41,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:08:43,733 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.76 % 2022-02-08 06:08:43,733 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 30.76 2022-02-08 06:08:43,773 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:08:43,776 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:08:43,776 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 8410 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:08:51,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.76 % 2022-02-08 06:08:51,411 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:08:51,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:08:53,806 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.76 % 2022-02-08 06:08:53,808 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 30.76 2022-02-08 06:08:53,849 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:08:53,852 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:08:53,854 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:09:01,426 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 31.30 % 2022-02-08 06:09:01,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:09:01,436 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:09:03,881 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 31.30 % 2022-02-08 06:09:03,881 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 31.3 2022-02-08 06:09:03,937 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:09:03,938 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:09:03,940 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:09:11,461 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.41 % 2022-02-08 06:09:11,526 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:09:11,526 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:09:13,957 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.41 % 2022-02-08 06:09:13,957 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 32.41 2022-02-08 06:09:14,006 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:09:14,009 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:09:14,009 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:09:21,546 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.41 % 2022-02-08 06:09:21,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:09:21,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:09:24,029 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.41 % 2022-02-08 06:09:24,029 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 32.41 2022-02-08 06:09:24,101 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:09:24,102 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:09:24,102 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:09:31,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.41 % 2022-02-08 06:09:31,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:09:31,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:09:34,119 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.41 % 2022-02-08 06:09:34,121 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 32.41 2022-02-08 06:09:34,170 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:09:34,173 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:09:34,173 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:09:41,619 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.41 % 2022-02-08 06:09:41,627 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:09:41,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:09:44,197 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.41 % 2022-02-08 06:09:44,197 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 32.41 2022-02-08 06:09:44,255 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:09:44,257 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:09:44,257 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 8433 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:09:51,688 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.09 % 2022-02-08 06:09:51,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:09:51,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:09:54,301 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.09 % 2022-02-08 06:09:54,301 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 34.09 2022-02-08 06:09:54,355 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:09:54,358 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:09:54,358 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:10:01,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.09 % 2022-02-08 06:10:01,720 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:10:01,720 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:10:04,388 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.09 % 2022-02-08 06:10:04,388 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 34.09 2022-02-08 06:10:04,437 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:10:04,438 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:10:04,440 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:10:11,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.09 % 2022-02-08 06:10:11,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:10:11,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:10:14,456 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.09 % 2022-02-08 06:10:14,457 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 34.09 2022-02-08 06:10:14,509 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:10:14,512 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:10:14,513 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:10:21,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.09 % 2022-02-08 06:10:21,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:10:21,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:10:24,552 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.09 % 2022-02-08 06:10:24,552 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 34.09 2022-02-08 06:10:24,599 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:10:24,602 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:10:24,602 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:10:31,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.09 % 2022-02-08 06:10:31,783 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:10:31,785 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:10:34,630 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.09 % 2022-02-08 06:10:34,630 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 34.09 2022-02-08 06:10:34,683 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:10:34,687 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:10:34,687 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:10:41,823 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.66 % 2022-02-08 06:10:41,835 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:10:41,835 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:10:44,704 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.66 % 2022-02-08 06:10:44,704 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 34.66 2022-02-08 06:10:44,743 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:10:44,746 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:10:44,746 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 8454 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:10:51,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.23 % 2022-02-08 06:10:51,859 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:10:51,859 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:10:54,762 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.23 % 2022-02-08 06:10:54,762 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 35.23 2022-02-08 06:10:54,809 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:10:54,812 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:10:54,812 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:11:01,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.80 % 2022-02-08 06:11:01,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:11:01,890 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:11:04,825 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.36 % 2022-02-08 06:11:04,825 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 36.36 2022-02-08 06:11:04,861 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:11:04,865 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:11:04,865 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:11:11,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.36 % 2022-02-08 06:11:11,908 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:11:11,908 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:11:14,878 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.36 % 2022-02-08 06:11:14,880 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 36.36 2022-02-08 06:11:14,910 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:11:14,913 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:11:14,913 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:11:21,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.36 % 2022-02-08 06:11:21,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:11:21,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:11:24,951 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.36 % 2022-02-08 06:11:24,953 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 36.36 2022-02-08 06:11:25,005 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:11:25,007 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:11:25,007 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:11:31,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.36 % 2022-02-08 06:11:31,973 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:11:31,973 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:11:35,023 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.93 % 2022-02-08 06:11:35,023 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 36.93 2022-02-08 06:11:35,055 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:11:35,058 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:11:35,058 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:11:41,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.50 % 2022-02-08 06:11:42,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:11:42,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:11:45,072 | test | DEBUG | pool-3-thread-12 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.50 % 2022-02-08 06:11:45,072 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: running, progress: 37.5 2022-02-08 06:11:45,104 | infra | DEBUG | pool-3-thread-12 | [task:check:441] Wait before next rebalance progress check 2022-02-08 06:11:45,107 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 06:11:45,107 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 8479 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:11:52,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 68.75 % 2022-02-08 06:11:52,038 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:11:52,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:11:55,121 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-08 06:11:55,121 | infra | DEBUG | pool-3-thread-12 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 06:12:00,132 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance - status: none, progress: 100 2022-02-08 06:12:00,250 | test | INFO | pool-3-thread-12 | [task:check:473] Rebalance completed with progress: 100% in 639.705999851 sec 2022-02-08 06:12:00,250 | infra | INFO | pool-3-thread-12 | [task:complete_task:106] Thread 'Rebalance_task_IN=[]_OUT=[172.23.121.136]_1644328880.1' completed 2022-02-08 06:12:02,039 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >68.75% in 521.840000153 seconds 2022-02-08 06:12:02,055 | test | DEBUG | MainThread | [rest_client:diag_eval:1476] /diag/eval status on 172.23.120.170:8091: 1 content: false command: ns_orchestrator:needs_rebalance() 2022-02-08 06:12:02,092 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-08 06:12:02,092 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 06:12:07,117 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:12:07,118 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:12:17,134 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >100% in 15.0780000687 seconds 2022-02-08 06:12:17,177 | test | DEBUG | MainThread | [rest_client:diag_eval:1476] /diag/eval status on 172.23.120.170:8091: 1 content: false command: ns_orchestrator:needs_rebalance() 2022-02-08 06:12:17,213 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-08 06:12:17,214 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 06:12:22,265 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2022-02-08 06:12:22,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 06:12:32,266 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >100% in 15.0889999866 seconds 2022-02-08 06:12:32,290 | test | DEBUG | MainThread | [rest_client:diag_eval:1476] /diag/eval status on 172.23.120.170:8091: 1 content: false command: ns_orchestrator:needs_rebalance() 2022-02-08 06:12:32,292 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.121.136]_1644328880.1 2022-02-08 06:12:32,378 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3970] Stats: {u'opsPerSec': 18463.7, u'diskFetches': 4, u'dataUsed': 111393399166L, u'itemCount': 99458443, u'quotaPercentUsed': 74.50805077826149, u'storageTotals': {u'hdd': {u'usedByData': 187425134418L, u'used': 301266066918L, u'quotaTotal': 1298666618880L, u'total': 1298666618880L, u'free': 849572549850L}, u'ram': {u'usedByData': 64462879664L, u'used': 116896686080L, u'quotaTotal': 87566581760L, u'total': 122941186048L, u'quotaUsed': 86518005760L, u'quotaUsedPerNode': 8651800576L, u'quotaTotalPerNode': 8756658176L}}, u'diskUsed': 187425134418L, u'vbActiveNumNonResident': 65605840, u'memUsed': 64462879664L} 2022-02-08 06:12:33,252 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+------------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 99458443 | 80.58 GiB | 60.04 GiB | 174.55 GiB | 34.0369323899 | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+------------+---------------+ 2022-02-08 06:12:33,252 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+------------------+------------------+-----------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+------------------+------------------+-----------------+---------------+ | 40000000 | 380000000 | 2000000-20000000 | 2000000-20000000 | 1000000-2000000 | 0-0 | +---------------+---------------+------------------+------------------+-----------------+---------------+ 2022-02-08 06:12:33,588 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 34.0956358877 2022-02-08 06:12:33,611 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:12:34,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:12:34,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:12:35,023 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:12:35,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:12:35,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:12:35,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:12:35,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:12:35,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:12:35,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:12:35,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:12:35,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 06:12:35,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 06:12:35,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 06:12:35,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 06:12:35,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 06:12:36,086 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:12:36,089 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:12:36,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:12:37,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:12:37,670 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:12:37,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:12:37,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:12:37,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:12:37,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:12:37,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:12:37,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:12:37,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:12:37,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 06:12:38,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 06:12:38,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 06:12:38,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 06:12:38,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 06:12:38,801 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:12:38,803 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:12:39,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:12:40,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:12:40,193 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:12:40,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:12:40,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:12:40,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:12:40,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:12:40,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:12:40,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:12:40,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:12:40,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 06:12:40,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 06:12:40,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 06:12:40,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 06:12:40,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 06:12:41,072 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:12:41,075 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 06:12:41,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:12:41,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 06:12:41,476 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:12:41,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 06:12:41,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 06:12:41,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 06:12:41,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 06:12:41,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 06:12:41,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 06:12:41,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 06:12:41,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 06:12:41,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 06:12:41,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 06:12:41,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 06:12:41,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 06:12:41,720 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 06:12:41,723 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:12:42,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:12:42,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:12:43,006 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:12:43,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:12:43,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:12:43,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:12:43,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:12:43,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:12:43,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:12:43,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:12:43,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 06:12:43,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 06:12:43,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 06:12:43,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 06:12:43,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 06:12:43,993 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:12:43,996 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 06:12:44,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:12:44,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 06:12:44,374 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:12:44,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 06:12:44,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 06:12:44,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 06:12:44,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 06:12:44,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 06:12:44,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 06:12:44,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 06:12:44,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 06:12:44,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 06:12:44,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 06:12:44,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 06:12:44,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 06:12:44,673 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 06:12:44,674 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:12:45,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:12:46,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:12:46,233 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:12:46,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:12:46,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:12:46,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:12:46,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:12:46,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:12:46,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:12:46,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:12:46,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 06:12:46,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 06:12:46,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 06:12:46,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 06:12:46,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 06:12:47,303 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:12:47,306 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:12:47,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:12:48,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:12:48,556 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:12:48,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:12:48,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:12:48,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:12:48,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:12:48,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:12:48,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:12:48,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:12:48,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 06:12:49,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 06:12:49,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 06:12:49,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 06:12:49,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 5 queries submitted, 0 failed, 8501 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:12:49,730 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:12:49,733 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 06:12:49,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:12:50,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 06:12:50,313 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:12:50,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 06:12:50,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 06:12:50,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 06:12:50,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 06:12:50,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 06:12:50,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 06:12:50,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 06:12:50,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 06:12:50,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 06:12:50,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 06:12:50,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 06:12:50,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 06:12:50,642 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 06:12:50,644 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 06:12:50,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:12:50,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 06:12:51,032 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:12:51,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 06:12:51,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 06:12:51,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 06:12:51,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 06:12:51,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 06:12:51,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 06:12:51,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 06:12:51,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 06:12:51,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 06:12:51,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 06:12:51,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 06:12:51,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 06:12:51,272 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 06:12:51,276 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:12:52,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:12:52,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:12:52,963 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:12:52,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:12:53,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:12:53,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:12:53,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:12:53,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:12:53,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:12:53,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:12:53,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 06:12:53,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 06:12:53,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 06:12:53,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 06:12:53,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 06:12:54,088 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:12:54,091 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:12:54,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:12:55,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:12:55,621 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:12:55,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:12:55,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:12:55,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:12:55,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:12:55,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:12:55,711 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:12:55,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:12:55,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 06:12:55,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 06:12:55,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 06:12:55,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 06:12:56,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 06:12:56,441 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:12:56,444 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:12:57,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:12:57,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:12:57,759 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:12:57,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:12:57,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:12:57,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:12:57,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:12:57,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:12:57,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:12:57,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:12:57,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 06:12:58,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 06:12:58,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 06:12:58,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 06:12:58,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 06:12:58,723 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:12:58,727 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:12:59,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:13:00,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:13:00,282 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:13:00,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:13:00,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:13:00,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:13:00,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:13:00,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:13:00,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:13:00,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:13:00,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 06:13:00,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 06:13:00,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 06:13:00,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 06:13:00,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 06:13:01,262 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:13:01,265 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 06:13:01,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:13:01,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 06:13:01,714 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:13:01,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 06:13:01,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 06:13:01,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 06:13:01,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 06:13:01,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 06:13:01,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 06:13:01,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 06:13:01,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 06:13:01,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 06:13:01,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 06:13:01,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 06:13:01,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 06:13:01,943 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 06:13:01,944 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for kvstore is 175752MB 2022-02-08 06:13:01,944 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for wal is 5349MB 2022-02-08 06:13:01,944 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:642] Total Disk usage for keyTree is 11650MB 2022-02-08 06:13:01,944 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:643] Total Disk usage for seqTree is 163791MB 2022-02-08 06:13:01,947 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:13:02,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:13:02,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:13:02,434 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:13:02,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:13:02,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:13:02,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:13:02,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:13:02,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:13:02,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:13:02,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:13:02,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 06:13:02,595 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.116 - core(s): 8 2022-02-08 06:13:02,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 06:13:03,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 06:13:03,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 06:13:03,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 06:13:03,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 06:13:03,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 06:13:03,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 06:13:04,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 06:13:04,260 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:13:04,265 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:13:04,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:13:04,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:13:04,721 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:13:04,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:13:04,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:13:04,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:13:04,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:13:04,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:13:04,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:13:04,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:13:04,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 06:13:04,907 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.123 - core(s): 8 2022-02-08 06:13:04,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 06:13:05,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 06:13:05,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 06:13:06,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 06:13:06,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 06:13:06,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 06:13:06,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 06:13:06,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 06:13:06,859 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:13:06,868 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:13:06,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:13:07,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:13:07,365 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:13:07,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:13:07,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:13:07,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:13:07,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:13:07,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:13:07,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:13:07,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:13:07,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 06:13:07,693 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.124 - core(s): 8 2022-02-08 06:13:07,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 06:13:08,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 06:13:08,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 06:13:08,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 06:13:08,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 06:13:09,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 06:13:09,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 06:13:09,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 06:13:09,861 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:13:09,864 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:13:10,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:13:10,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:13:10,454 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:13:10,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:13:10,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:13:10,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:13:10,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:13:10,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:13:10,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:13:10,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:13:10,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 06:13:10,753 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.126 - core(s): 8 2022-02-08 06:13:10,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 06:13:11,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 06:13:11,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 06:13:12,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 06:13:12,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 06:13:12,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 06:13:12,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 06:13:12,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 06:13:13,098 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:13:13,101 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:13:13,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:13:13,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:13:13,678 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:13:13,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:13:13,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:13:13,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:13:13,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:13:13,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:13:13,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:13:13,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:13:13,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 06:13:13,868 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.127 - core(s): 8 2022-02-08 06:13:13,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 06:13:14,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 06:13:14,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 06:13:14,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 06:13:14,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 06:13:15,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 06:13:15,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 06:13:15,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 06:13:15,878 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:13:15,881 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:13:15,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:13:16,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:13:16,293 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:13:16,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:13:16,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:13:16,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:13:16,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:13:16,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:13:16,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:13:16,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:13:16,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 06:13:16,493 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.129 - core(s): 8 2022-02-08 06:13:16,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 06:13:17,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 06:13:17,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 06:13:17,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 06:13:17,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 06:13:17,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 06:13:18,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 06:13:18,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 06:13:18,367 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:13:18,371 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:13:18,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:13:18,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:13:18,772 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:13:18,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:13:18,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:13:18,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:13:18,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:13:18,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:13:18,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:13:18,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:13:18,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 06:13:18,976 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.135 - core(s): 8 2022-02-08 06:13:18,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 06:13:19,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 06:13:19,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 06:13:19,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 06:13:19,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 06:13:20,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 06:13:20,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 06:13:20,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 06:13:20,668 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:13:20,671 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:13:20,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:13:21,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:13:21,246 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:13:21,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:13:21,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:13:21,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:13:21,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:13:21,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:13:21,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:13:21,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:13:21,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 06:13:21,420 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.115 - core(s): 8 2022-02-08 06:13:21,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 06:13:22,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 06:13:22,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 06:13:22,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 06:13:22,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 06:13:22,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 06:13:22,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 06:13:23,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 06:13:23,211 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:13:23,213 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:13:23,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:13:23,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:13:23,693 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:13:23,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:13:23,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:13:23,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:13:23,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:13:23,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:13:23,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:13:23,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:13:23,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 06:13:23,974 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.128 - core(s): 8 2022-02-08 06:13:23,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 06:13:24,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 06:13:24,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 06:13:24,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 06:13:25,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 06:13:25,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 06:13:25,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 06:13:25,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 06:13:25,880 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:13:25,881 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:13:25,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:13:26,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:13:26,345 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:13:26,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:13:26,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:13:26,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:13:26,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:13:26,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:13:26,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:13:26,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:13:26,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 06:13:26,517 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.120.170 - core(s): 8 2022-02-08 06:13:26,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 06:13:27,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 06:13:27,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 06:13:27,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 06:13:27,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 06:13:28,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 06:13:28,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 06:13:28,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 06:13:28,733 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:13:28,733 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:723] {'172.23.121.116': {'rw_0:magma': {u'NReadBytesGetPerSec': 363652.15625, u'ReplayWALOffset': u'1191:4448256', u'NTablesCreated': 9695, u'RecentReadIOAmpSet': 0.023920053616166115, u'ReadIOAmp': 2.1882588863372803, u'BlockCacheMisses': 190583, u'NFlushes': 2386, u'RecentWriteAmp': 2.1239752769470215, u'NumSyncFlushes': 104, u'TotalMemUsed': 48675224, u'NBloomFilterHits': 4682477, u'Fragmentation': 0.3946976065635681, u'NumNormalFlushes': 1142, u'RecentBlockCacheHitRatio': 77.0920181274414, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 4837, u'ReadIOAmp': 110.25, u'MemtablesMemUsed': 7460176, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 17767416, u'CheckpointOverhead': 15220736, u'NItemsGet': 32, u'NWriteIOs': 6374, u'NReadBytes': 161488892, u'LevelStats': [{u'TargetSize': 109051904, u'LogicalSize': 18486347, u'SizeRatio': 0.1695187917122474, u'NWriteBytes': 21372928, u'BloomFilterMemUsed': 0, u'NumTables': 214, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2088960}, {u'TargetSize': 109051904, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 109051904, u'LogicalSize': 79215927, u'SizeRatio': 0.72640572144435, u'NWriteBytes': 10018816, u'BloomFilterMemUsed': 0, u'NumTables': 52, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 1347584}, {u'TargetSize': 1085454079, u'LogicalSize': 883688931, u'SizeRatio': 0.814119130506303, u'NWriteBytes': 3448832, u'BloomFilterMemUsed': 0, u'NumTables': 399, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 14512128}, {u'TargetSize': 4236192986L, u'LogicalSize': 2170908168L, u'SizeRatio': 0.512466777404744, u'NWriteBytes': 391536640, u'BloomFilterMemUsed': 0, u'NumTables': 932, u'WastedSpace': 52811675, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 2170847232L}], u'NItemsGarbageCollected': 437567, u'NInternalFragmentationCompacts': 10, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 15307309, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 195072, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 112707, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 573024, u'CompactWriter': {u'NDataBlockWrites': 34862, u'NWriteBytes': 105021440, u'NIndexBlockWrites': 612, u'NBytesIncoming': 160763859, u'NSyncs': 0, u'NInserts': 849452, u'NWriteIOs': 560}, u'NonResidentBloomFilterSize': 0, u'IteratorMemUsed': 0, u'NWriteBytes': 434034642, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 3528, u'NReadBytes': 525011, u'NDataBlockReads': 32, u'NBytesOutgoing': 0, u'NIndexBlockReads': 50}, u'NBloomFilterMisses': 0, u'NItemsMerged': 97248, u'LogicalDiskSize': 2170908168L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.5208794914687379, u'BufferMemUsed': 1116160, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 1287219237, u'NTableFiles': 1907, u'NFileCountCompacts': 127, u'TableObjectMemUsed': 560160, u'TreeSnapshotMemoryUsed': 271184, u'TotalDiskUsage': 2516115456L, u'BytesIncoming': 833272665, u'NItemsDel': 0, u'PendingCheckpointBytes': 54472704, u'NCompacts': 248, u'NItemsPut': 1274152, u'NSyncs': 1250, u'CompactReader': {u'NReadIOs': 59963, u'NReadBytes': 143544077, u'NDataBlockReads': 57042, u'NBytesOutgoing': 0, u'NIndexBlockReads': 2921}, u'ActiveDiskUsage': 2188795904L, u'NTablesDeleted': 2538, u'IterateReader': {u'NReadIOs': 49216, u'NReadBytes': 17419804, u'NDataBlockReads': 24783, u'NBytesOutgoing': 0, u'NIndexBlockReads': 24433}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 112049, u'NWriteBytes': 329013202, u'NIndexBlockWrites': 7295, u'NBytesIncoming': 513577810, u'NSyncs': 1250, u'NInserts': 776559, u'NWriteIOs': 5814}}, u'NReadBytes': 1058192939, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 89.69999694824219, u'NInternalFragmentationCompacts': 10, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 16927154, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 28672, u'NTxns': 153154, u'NWriteBytes': 1336700928, u'NSegments': 110, u'NSyncs': 153153, u'NLoggedBytes': 1336700928, u'NActiveSegments': 6, u'NWrites': 153484, u'PreallocateSupported': True, u'NInserts': 1811864, u'DiskUsed': 77307904}, u'NGets': 101217, u'NReadBytesSet': 59969516, u'NReadBytesCompact': 353992062, u'IndexResidentRatio': 0.3026030659675598, u'CompactQueueSize': 0, u'NSyncsPerSec': 3.700000047683716, u'BlockCacheHitRatio': 58.432464599609375, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 686781073, u'BloomFilterMemUsedRatio': 0.0250190868973732, u'NonResidentBloomFilterSize': 591528, u'NWriteBytes': 2085504655, u'BytesOutgoing': 28700, u'RecentReadIOAmp': 1.0527197122573853, u'WriteAmp': 2.01444673538208, u'NBlocksCached': 9276, u'LogicalDataSize': 1372244826, u'ActiveBloomFilterMemUsed': 2675952, u'NFileCountCompacts': 389, u'NTableFiles': 3032, u'FlushQueueSize': 8, u'TableObjectMemUsed': 830160, u'BlockCacheQuota': 82673484, u'TotalDiskUsage': 2678743040L, u'NPurgerRuns': 0, u'ReadIOAmpSet': 0.0, u'NSyncs': 3750, u'NReadIOsPerSec': 342.70001220703125, u'NReadIOsSetPerSec': 22.5, u'NTablesDeleted': 7303, u'BloomFilterFPR': 0.01630019024014473, u'NBloomFilterHitsPerSec': 5080.83349609375, u'WriteCacheMemUsed': 10485744, u'RecentBloomFilterFPR': 0.016232090070843697, u'BytesPerRead': 6192.75, u'BytesIncomingPerSec': 1169539.25, u'NReadIOsGetPerSec': 125.80000305175781, u'NumKVStores': 26, u'NGetStats': 1656, u'TotalIndexBlocksSize': 19761429, u'NGetStatsPerSec': 0.8333333134651184, u'NWriteBytesCompact': 348893184, u'NReadIOsSet': 20860, u'ActiveDataSize': 1372244826, u'NWriteBytesCompactPerSec': 515822.9375, u'NBloomFilterMissesPerSec': 83.83333587646484, u'NCommitBatches': 152300, u'NGetStatsComputed': 1656, u'NWriteIOs': 167615, u'NReadIOsGet': 242349, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 20668372, u'LSMTreeObjectMemUsed': 448448, u'NSetsPerSec': 940.6333618164062, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 2.4666666984558105, u'ManagedFileObjectMemUsed': 303072, u'ReadAmpGet': 21840.125, u'BlockCacheMemUsed': 31021952, u'NWriterCompacts': 0, u'NReadIOs': 429287, u'WALBufferMemUsed': 81920, u'ReadAmp': 34781.3046875, u'keyStats': {u'NItemsTransformed': 5242472, u'NTablesCreated': 2447, u'ReadIOAmp': 0.26934798559800693, u'MemtablesMemUsed': 906576, u'NBloomFilterHits': 4681978, u'TotalIndexBlocksSize': 1969275, u'CheckpointOverhead': 29519872, u'NItemsGet': 879045, u'NWriteIOs': 4096, u'NReadBytes': 892147102, u'LevelStats': [{u'TargetSize': 109051904, u'LogicalSize': 4536918, u'SizeRatio': 0.04160329011770395, u'NWriteBytes': 49430528, u'BloomFilterMemUsed': 56240, u'NumTables': 198, u'WastedSpace': 276, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 4132864}, {u'TargetSize': 109051904, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 109051904, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 0}, {u'TargetSize': 109051904, u'LogicalSize': 27962031, u'SizeRatio': 0.25641029614668626, u'NWriteBytes': 113709056, u'BloomFilterMemUsed': 763392, u'NumTables': 28, u'WastedSpace': 54416, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 28004352}, {u'TargetSize': 111982766, u'LogicalSize': 95255092, u'SizeRatio': 0.8506227824377904, u'NWriteBytes': 121479168, u'BloomFilterMemUsed': 1851392, u'NumTables': 42, u'WastedSpace': 263628, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 95518720}], u'NItemsGarbageCollected': 473203, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 1605929, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 59232, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 310144, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 196209, u'NFailedCompacts': 40, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 188268, u'CompactWriter': {u'NDataBlockWrites': 83599, u'NWriteBytes': 242782208, u'NIndexBlockWrites': 1190, u'NBytesIncoming': 315907809, u'NSyncs': 0, u'NInserts': 5642575, u'NWriteIOs': 572}, u'NonResidentBloomFilterSize': 588638, u'IteratorMemUsed': 0, u'NWriteBytes': 294399346, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 236769, u'NReadBytes': 685614042, u'NDataBlockReads': 924632, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1180609}, u'NBloomFilterMisses': 77385, u'NItemsMerged': 0, u'LogicalDiskSize': 95856028, u'TotalBloomFilterMemUsed': 2700528, u'WriteAmp': 6.647399086432186, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 2671024, u'LogicalDataSize': 84749563, u'NTableFiles': 617, u'NFileCountCompacts': 129, u'TableObjectMemUsed': 148080, u'TreeSnapshotMemoryUsed': 146384, u'TotalDiskUsage': 158466048, u'BytesIncoming': 44287900, u'NItemsDel': 0, u'PendingCheckpointBytes': 2387968, u'NCompacts': 132, u'NItemsPut': 887446, u'NSyncs': 1250, u'CompactReader': {u'NReadIOs': 73375, u'NReadBytes': 206533060, u'NDataBlockReads': 87723, u'NBytesOutgoing': 0, u'NIndexBlockReads': 3325}, u'ActiveDiskUsage': 127655936, u'NTablesDeleted': 2375, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.016259528848713577, u'Writer': {u'NDataBlockWrites': 9631, u'NWriteBytes': 51617138, u'NIndexBlockWrites': 2283, u'NBytesIncoming': 31777614, u'NSyncs': 1250, u'NInserts': 540036, u'NWriteIOs': 3524}}, u'NReadBytesCompactPerSec': 557042.9375, u'TableMetaMemUsed': 907048, u'NDeletes': 0, u'BlockCacheHitsPerSec': 301.8666687011719, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 77590, u'LogicalDiskSize': 2267040222L, u'NSets': 793330, u'TotalBloomFilterMemUsed': 2705752, u'BufferMemUsed': 1116160, u'RecentReadAmp': 0.0, u'NGetsPerSec': 119.5, u'NInserts': 94116, u'NCommitBatchesPerSec': 211.56666564941406, u'FragmentationWithOverhead': 0.4877280592918396, u'NCompactsPerSec': 0.6000000238418579, u'WALMemUsed': 147536, u'MemoryQuota': 108147512, u'NReadBytesPerSec': 920695.0625, u'TreeSnapshotMemoryUsed': 525648, u'BytesIncoming': 1035274149, u'NCompacts': 513, u'NumPersistentFlushes': 1140, u'NWriteIOsPerSec': 227.36666870117188, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 3043.1142578125, u'ActiveDiskUsage': 2318786560L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 0.8333333134651184, u'NReadBytesSetPerSec': 65231.16796875, u'BlockCacheHits': 267907, u'NWriteBytesPerSec': 2484072.25, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 2411, u'ReadIOAmp': 1.3987730061349692, u'MemtablesMemUsed': 2118992, u'NBloomFilterHits': 499, u'TotalIndexBlocksSize': 24738, u'CheckpointOverhead': 1679360, u'NItemsGet': 1467, u'NWriteIOs': 3661, u'NReadBytes': 4556945, u'LevelStats': [{u'TargetSize': 109051904, u'LogicalSize': 2642325, u'SizeRatio': 0.024229975847097542, u'NWriteBytes': 18096128, u'BloomFilterMemUsed': 4096, u'NumTables': 259, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2121728}, {u'TargetSize': 109051904, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 109051904, u'LogicalSize': 276026, u'SizeRatio': 0.002531143335195688, u'NWriteBytes': 1048576, u'BloomFilterMemUsed': 832, u'NumTables': 26, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 212992}], u'NItemsGarbageCollected': 481736, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 13916, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 48768, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 6436, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 145756, u'CompactWriter': {u'NDataBlockWrites': 133, u'NWriteBytes': 1089536, u'NIndexBlockWrites': 133, u'NBytesIncoming': 411796, u'NSyncs': 0, u'NInserts': 3059, u'NWriteIOs': 133}, u'NonResidentBloomFilterSize': 2890, u'IteratorMemUsed': 0, u'NWriteBytes': 20369739, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 2052, u'NReadBytes': 642020, u'NDataBlockReads': 1095, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1095}, u'NBloomFilterMisses': 205, u'NItemsMerged': 0, u'LogicalDiskSize': 276026, u'TotalBloomFilterMemUsed': 5224, u'WriteAmp': 0.09357738352023003, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 4928, u'LogicalDataSize': 276026, u'NTableFiles': 508, u'NFileCountCompacts': 133, u'TableObjectMemUsed': 121920, u'TreeSnapshotMemoryUsed': 108080, u'TotalDiskUsage': 4161536, u'BytesIncoming': 217678014, u'NItemsDel': 0, u'PendingCheckpointBytes': 229376, u'NCompacts': 133, u'NItemsPut': 486135, u'NSyncs': 1250, u'CompactReader': {u'NReadIOs': 4384, u'NReadBytes': 3914925, u'NDataBlockReads': 2390, u'NBytesOutgoing': 0, u'NIndexBlockReads': 2390}, u'ActiveDiskUsage': 2334720, u'NTablesDeleted': 2390, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.2911931818181818, u'Writer': {u'NDataBlockWrites': 2278, u'NWriteBytes': 19280203, u'NIndexBlockWrites': 2278, u'NBytesIncoming': 3714048, u'NSyncs': 1250, u'NInserts': 23577, u'NWriteIOs': 3528}}}}} 2022-02-08 06:13:28,740 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:728] magma stats fragmentation result {'172.23.121.123': [0.3834376931190491, 0.41270267963409424, 0.37972304224967957, 0.4004616439342499, 0.3755562901496887, 0.41380465030670166, 0.4130050539970398, 0.42327430844306946], '172.23.121.124': [0.3943871557712555, 0.3900085389614105, 0.38294076919555664, 0.39466604590415955, 0.3764042854309082, 0.36467859148979187, 0.3865298628807068, 0.38982605934143066], '172.23.121.135': [0.3443151116371155, 0.3537261486053467, 0.3527816832065582, 0.38370394706726074, 0.36464935541152954, 0.3825591206550598, 0.3695864677429199, 0.3962697684764862], '172.23.121.126': [0.3975524604320526, 0.394035279750824, 0.3816791772842407, 0.3998216390609741, 0.3898840844631195, 0.4073539972305298, 0.39903610944747925, 0.40075090527534485], '172.23.121.115': [0.2902943789958954, 0.27181413769721985, 0.28248098492622375, 0.3118097186088562, 0.2951269745826721, 0.3059336543083191, 0.28908881545066833, 0.2912891209125519], '172.23.121.116': [0.3946976065635681, 0.3793657124042511, 0.3601166009902954, 0.3702412545681, 0.40680089592933655, 0.401170939207077, 0.3812576234340668, 0.3742757737636566], '172.23.121.127': [0.40946677327156067, 0.3801705837249756, 0.3878418207168579, 0.3897183835506439, 0.40176135301589966, 0.393619179725647, 0.390080064535141, 0.3652205467224121], '172.23.121.128': [0.00551195815205574, 0.013907562009990215, 0.010706113651394844, 0.0025135662872344255, 0.002558869542554021, 0.0025999147910624743, 0.004474509507417679, 0.0015852105570957065], '172.23.121.129': [0.3532200753688812, 0.3746830224990845, 0.34592926502227783, 0.3892871141433716, 0.40173280239105225, 0.4156316816806793, 0.372718870639801, 0.3936421275138855], '172.23.120.170': [0.3647393584251404, 0.3885573446750641, 0.4017706513404846, 0.4120003581047058, 0.36227256059646606, 0.35867512226104736, 0.34180375933647156, 0.3950212001800537]} 2022-02-08 06:13:29,101 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.116 is 36.0 2022-02-08 06:13:29,585 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.123 is 36.0 2022-02-08 06:13:29,913 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.124 is 36.0 2022-02-08 06:13:30,263 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.126 is 36.0 2022-02-08 06:13:30,562 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.127 is 36.0 2022-02-08 06:13:30,917 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.129 is 36.0 2022-02-08 06:13:31,345 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.135 is 36.0 2022-02-08 06:13:31,775 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.115 is 36.0 2022-02-08 06:13:32,148 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.128 is 36.0 2022-02-08 06:13:32,460 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.120.170 is 36.0 2022-02-08 06:13:32,460 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:677] KV stats fragmentation values {'172.23.121.123': 36.0, '172.23.121.124': 36.0, '172.23.121.135': 36.0, '172.23.121.126': 36.0, '172.23.121.115': 36.0, '172.23.121.116': 36.0, '172.23.121.127': 36.0, '172.23.121.128': 36.0, '172.23.121.129': 36.0, '172.23.120.170': 36.0} ############################################################ # # Step 10: Crash Magma/memc with Loading of docs # ############################################################ ############################################################ # # RAM FootPrint: 4379.83203125 # ############################################################ 2022-02-08 06:13:32,763 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 06:13:32,765 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 114 seconds. Reason: Iteration:0 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 8524 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 8544 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:15:26,772 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 06:15:26,772 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 06:15:31,775 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:15:31,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:15:32,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:15:32,240 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:15:32,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:15:32,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:15:32,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:15:32,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:15:32,326 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:15:32,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:15:32,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:15:32,385 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:15:32,385 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:15:32,460 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['6387 memcached\n'], []) 2022-02-08 06:15:32,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 06:15:32,497 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:15:32,520 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['6387 memcached\n'], []) 2022-02-08 06:15:32,522 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:15:32,523 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:15:32,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:15:32,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:15:32,950 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:15:32,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:15:32,964 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:15:32,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:15:32,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:15:33,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:15:33,025 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:15:33,038 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:15:33,052 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:15:33,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:15:33,108 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['51800 memcached\n'], []) 2022-02-08 06:15:33,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 06:15:33,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:15:33,167 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['51800 memcached\n'], []) 2022-02-08 06:15:33,167 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:15:33,171 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:15:33,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:15:33,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:15:33,571 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:15:33,571 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:15:33,589 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:15:33,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:15:33,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:15:33,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:15:33,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:15:33,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:15:33,684 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:15:33,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:15:33,744 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['38262 memcached\n'], []) 2022-02-08 06:15:33,744 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 06:15:33,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:15:33,799 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['38262 memcached\n'], []) 2022-02-08 06:15:33,801 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:15:33,802 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:15:33,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:15:34,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:15:34,161 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:15:34,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:15:34,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:15:34,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:15:34,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:15:34,219 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:15:34,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:15:34,247 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:15:34,260 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:15:34,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:15:34,302 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['70997 memcached\n'], []) 2022-02-08 06:15:34,303 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 06:15:34,329 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:15:34,352 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['70997 memcached\n'], []) 2022-02-08 06:15:34,352 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:15:34,355 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:15:34,448 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:15:34,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:15:34,727 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:15:34,729 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:15:34,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:15:34,755 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:15:34,767 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:15:34,783 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:15:34,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:15:34,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:15:34,822 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:15:34,823 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:15:34,878 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['64831 memcached\n'], []) 2022-02-08 06:15:34,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 06:15:34,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:15:34,928 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['64831 memcached\n'], []) 2022-02-08 06:15:34,930 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:15:34,940 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:15:35,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:15:35,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:15:35,296 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:15:35,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:15:35,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:15:35,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:15:35,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:15:35,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:15:35,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:15:35,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:15:35,398 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:15:35,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:15:35,450 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['99822 memcached\n'], []) 2022-02-08 06:15:35,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 06:15:35,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:15:35,497 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['99822 memcached\n'], []) 2022-02-08 06:15:35,497 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:15:35,500 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:15:35,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:15:35,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:15:35,898 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:15:35,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:15:35,913 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:15:35,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:15:35,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:15:35,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:15:35,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:15:35,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:15:35,996 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:15:35,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:15:36,049 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['33625 memcached\n'], []) 2022-02-08 06:15:36,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 06:15:36,073 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:15:36,096 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['33625 memcached\n'], []) 2022-02-08 06:15:36,096 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:15:36,098 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:15:36,178 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:15:36,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:15:36,497 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:15:36,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:15:36,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:15:36,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:15:36,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:15:36,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:15:36,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:15:36,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:15:36,654 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:15:36,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:15:36,720 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['48546 memcached\n'], []) 2022-02-08 06:15:36,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 06:15:36,744 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:15:36,773 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['48546 memcached\n'], []) 2022-02-08 06:15:36,775 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:15:36,779 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:15:36,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:15:37,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:15:37,135 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:15:37,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:15:37,150 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:15:37,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:15:37,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:15:37,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:15:37,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:15:37,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:15:37,236 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:15:37,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:15:37,276 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['91391 memcached\n'], []) 2022-02-08 06:15:37,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 06:15:37,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:15:37,318 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['91391 memcached\n'], []) 2022-02-08 06:15:37,318 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:15:37,319 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:15:37,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:15:37,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:15:37,716 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:15:37,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:15:37,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:15:37,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:15:37,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:15:37,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:15:37,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:15:37,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:15:37,819 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:15:37,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:15:37,862 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['115905 memcached\n'], []) 2022-02-08 06:15:37,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 06:15:37,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:15:37,911 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['115905 memcached\n'], []) 2022-02-08 06:15:38,035 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:15:38,036 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:15:38,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:15:38,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:15:38,397 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:15:38,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:15:38,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:15:38,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:15:38,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:15:38,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:15:38,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:15:38,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:15:38,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:15:38,536 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 06:15:38,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:15:38,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:15:42,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:15:43,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:15:43,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:15:43,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:15:43,609 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 06:15:43,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:15:43,624 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:15:43,678 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:15:43,709 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 06:15:43,710 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:15:43,717 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:15:43,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:15:44,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:15:44,111 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:15:44,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:15:44,140 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:15:44,160 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:15:44,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:15:44,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:15:44,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:15:44,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:15:44,232 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:15:44,246 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 06:15:44,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:15:44,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:15:49,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 5 queries submitted, 0 failed, 8568 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:15:49,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:15:50,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:15:50,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:15:50,503 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 06:15:50,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:15:50,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:15:50,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:15:50,592 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 06:15:50,592 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:15:50,595 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:15:50,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:15:50,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:15:50,967 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:15:50,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:15:50,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:15:50,997 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:15:51,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:15:51,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:15:51,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:15:51,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:15:51,066 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:15:51,085 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 06:15:51,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:15:51,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:15:54,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:15:55,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:15:55,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:15:56,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:15:56,045 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 06:15:56,045 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:15:56,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:15:56,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:15:56,378 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 06:15:56,378 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:15:56,380 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 06:15:56,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:15:56,681 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 06:15:56,746 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:15:56,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 06:15:56,760 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 06:15:56,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 06:15:56,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 06:15:56,801 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 06:15:56,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 06:15:56,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 06:15:56,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:15:56,854 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 06:15:56,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:15:56,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:15:56,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:15:56,923 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:15:56,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:15:56,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:15:56,960 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 06:15:56,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:15:56,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:15:56,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:15:56,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:15:57,017 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:15:57,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:15:57,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:15:57,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:15:57,104 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:15:57,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:15:57,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:15:57,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:15:57,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:15:57,214 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:15:57,239 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:15:57,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:15:57,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:15:57,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:15:57,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:15:57,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:15:57,339 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:15:57,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:15:57,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:15:57,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:15:57,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:15:57,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:15:57,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:15:57,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:15:57,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:15:57,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:15:57,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:15:57,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:15:57,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:15:57,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:15:57,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:15:57,607 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:15:57,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:15:57,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:15:57,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:15:57,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:15:57,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:15:57,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:15:57,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:15:57,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:15:57,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:15:57,762 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:15:57,778 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:15:57,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:15:57,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:15:57,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:15:57,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:15:57,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:15:57,869 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:15:57,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:15:57,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:15:57,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:15:57,933 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:15:57,948 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:15:57,964 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:15:57,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:15:57,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:15:58,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:15:58,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:15:58,046 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 06:15:58,049 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:15:58,150 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:15:58,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:15:58,444 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:15:58,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:15:58,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:15:58,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:15:58,484 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:15:58,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:15:58,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:15:58,535 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:15:58,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:15:58,563 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 06:15:58,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:15:58,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:01,331 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:16:01,693 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:01,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:02,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:16:02,066 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 06:16:02,068 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:16:02,084 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:02,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:16:02,148 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 06:16:02,148 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:16:02,150 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 06:16:02,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:16:02,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 06:16:02,552 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:16:02,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 06:16:02,566 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 06:16:02,579 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 06:16:02,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 06:16:02,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 06:16:02,624 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 06:16:02,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 06:16:02,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:16:02,671 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 06:16:02,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:16:02,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:02,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:16:02,752 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:02,765 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:02,778 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:16:02,790 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 06:16:02,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:16:02,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:02,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:16:02,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:02,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:16:02,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:02,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:02,923 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:02,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:02,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:02,989 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:03,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:03,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:03,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:03,071 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:16:03,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:03,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:03,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:03,140 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:03,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:03,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:03,193 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:03,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:03,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:03,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:16:03,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:03,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:03,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:03,329 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:03,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:03,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:03,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:03,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:03,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:03,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:16:03,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:03,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:03,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:03,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:03,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:03,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:03,566 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:03,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:03,596 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:03,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:16:03,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:03,642 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:03,658 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:03,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:03,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:03,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:03,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:03,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:03,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:03,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:16:03,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:03,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:03,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:03,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:03,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:03,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:03,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:03,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:03,947 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 06:16:03,950 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:16:04,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:16:04,283 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:16:04,381 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:16:04,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:16:04,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:16:04,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:16:04,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:16:04,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:16:04,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:16:04,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:16:04,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:16:04,523 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 06:16:04,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:16:04,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:08,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:16:09,341 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:09,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:10,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:16:10,036 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 06:16:10,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:16:10,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:10,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:16:10,112 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 06:16:10,112 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:16:10,115 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:16:10,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:16:10,448 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:16:10,533 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:16:10,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:16:10,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:16:10,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:16:10,584 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:16:10,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:16:10,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:16:10,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:16:10,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:16:10,690 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 06:16:10,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:16:10,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:15,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:16:15,596 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:15,903 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:16,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:16:16,200 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 06:16:16,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:16:16,216 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:16,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:16:16,289 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 06:16:16,290 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:16:16,292 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 06:16:16,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:16:16,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 06:16:16,710 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:16:16,711 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 06:16:16,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 06:16:16,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 06:16:16,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 06:16:16,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 06:16:16,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 06:16:16,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 06:16:16,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:16:16,836 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 06:16:16,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:16:16,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:16,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:16:16,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:16,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:16,943 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:16:16,956 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 06:16:16,956 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:16:16,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:16,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:16:16,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:17,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:16:17,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:17,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:17,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:17,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:17,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:17,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:17,201 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:17,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:17,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:17,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:16:17,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:17,295 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:17,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:17,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:17,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:17,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:17,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:17,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:17,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:17,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:16:17,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:17,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:17,484 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:17,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:17,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:17,545 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:17,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:17,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:17,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:17,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:16:17,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:17,660 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:17,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:17,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:17,714 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:17,732 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:17,749 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:17,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:17,783 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:17,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:16:17,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:17,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:17,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:17,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:17,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:17,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:17,918 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:17,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:17,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:17,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:16:17,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:18,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:18,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:18,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:18,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:18,081 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:18,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:18,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:18,135 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 06:16:18,138 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 06:16:18,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:16:18,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 06:16:18,517 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:16:18,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 06:16:18,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 06:16:18,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 06:16:18,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 06:16:18,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 06:16:18,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 06:16:18,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 06:16:18,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:16:18,668 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 06:16:18,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:16:18,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:18,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:16:18,744 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:18,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:18,770 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:16:18,783 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 06:16:18,783 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:16:18,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:18,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:16:18,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:18,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:16:18,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:18,890 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:18,913 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:18,933 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:18,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:18,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:18,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:19,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:19,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:19,058 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:16:19,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:19,091 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:19,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:19,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:19,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:19,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:19,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:19,191 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:19,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:19,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:16:19,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:19,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:19,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:19,295 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:19,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:19,326 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:19,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:19,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:19,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:19,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:16:19,405 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:19,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:19,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:19,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:19,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:19,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:19,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:19,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:19,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:19,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:16:19,579 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:19,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:19,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:19,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:19,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:19,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:19,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:19,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:19,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:19,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:16:19,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:19,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:19,788 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:19,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:19,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:19,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:19,854 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:19,869 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:19,885 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 06:16:19,895 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:16:19,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:16:20,216 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:16:20,289 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:16:20,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:16:20,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:16:20,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:16:20,329 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:16:20,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:16:20,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:16:20,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:16:20,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:16:20,407 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 06:16:20,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:16:20,454 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:24,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:16:25,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:25,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:25,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:16:25,681 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 06:16:25,681 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:16:25,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:25,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:16:25,782 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 06:16:25,782 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:16:25,785 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:16:25,869 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:16:26,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:16:26,187 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:16:26,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:16:26,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:16:26,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:16:26,234 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:16:26,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:16:26,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:16:26,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:16:26,306 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:16:26,325 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 06:16:26,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:16:26,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:30,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:16:30,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:30,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:30,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:16:30,921 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 06:16:30,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:16:30,936 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:30,982 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:16:30,996 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 06:16:30,997 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:16:31,000 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:16:31,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:16:31,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:16:31,979 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:16:31,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:16:31,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:16:32,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:16:32,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:16:32,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:16:32,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:16:32,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:16:32,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:16:32,104 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 06:16:32,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:16:32,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:35,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:16:35,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:35,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:36,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:16:36,213 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 06:16:36,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:16:36,227 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:36,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:16:36,278 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 06:16:36,279 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:16:36,280 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:16:36,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:16:36,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:16:36,713 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:16:36,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:16:36,729 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:16:36,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:16:36,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:16:36,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:16:36,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:16:36,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:16:36,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:16:36,825 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 06:16:36,849 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:16:36,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:41,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:16:42,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:42,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:42,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:16:42,970 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 06:16:42,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:16:42,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:43,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:16:43,052 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 06:16:43,052 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:16:43,055 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 06:16:43,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:16:43,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 06:16:43,424 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:16:43,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 06:16:43,441 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 06:16:43,456 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 06:16:43,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 06:16:43,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 06:16:43,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 06:16:43,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 06:16:43,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:16:43,551 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 06:16:43,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:16:43,596 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:43,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:16:43,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:43,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:16:43,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:16:43,680 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 06:16:43,681 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:16:43,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:16:43,711 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:16:43,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:43,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:16:43,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:43,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:43,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:43,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:43,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:43,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:43,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:43,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:16:43,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:43,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:16:43,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:44,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:44,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:44,043 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:44,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:44,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:44,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:44,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:16:44,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:44,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:16:44,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:44,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:44,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:44,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:44,219 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:44,234 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:44,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:44,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:16:44,280 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:44,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:16:44,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:44,326 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:44,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:44,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:44,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:44,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:44,405 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:44,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:16:44,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:44,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:16:44,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:44,482 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:44,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:44,512 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:44,528 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:44,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:44,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:44,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:16:44,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:44,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:16:44,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:44,640 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:44,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:44,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:44,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:44,709 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:44,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:44,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:16:44,759 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 06:16:44,759 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:16:44,762 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:16:44,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:16:45,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:16:45,121 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:16:45,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:16:45,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:16:45,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:16:45,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:16:45,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:16:45,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:16:45,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:16:45,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:16:45,507 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:16:45,507 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.74799990654. 2022-02-08 06:16:45,509 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:16:45,510 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:16:45,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:16:45,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:16:45,957 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:16:45,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:16:45,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:16:45,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:16:46,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:16:46,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:16:46,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:16:46,068 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:16:46,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:16:46,262 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:16:46,263 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:16:48,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:16:48,431 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:16:48,433 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 8570 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:16:50,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:16:50,584 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:16:50,584 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:16:52,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:16:52,736 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:16:52,737 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:16:54,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:16:55,028 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:16:55,029 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:16:57,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:16:57,253 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:16:57,253 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:16:59,255 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:16:59,533 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:16:59,533 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:17:01,535 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:01,687 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:17:01,688 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:17:03,688 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:03,834 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:17:03,834 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:17:05,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:05,976 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:17:05,976 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:17:07,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:08,121 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:17:08,121 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:17:10,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:10,273 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:17:10,275 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:17:12,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:12,427 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:17:12,427 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:17:14,428 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:14,582 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:17:14,582 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:17:16,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:16,734 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:17:16,734 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:17:18,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:18,872 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:17:18,874 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 33.3650000095. 2022-02-08 06:17:18,874 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:17:18,875 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:17:19,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:17:19,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:17:19,937 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:17:19,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:17:19,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:17:19,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:17:19,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:17:20,005 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:17:20,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:17:20,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:17:20,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:20,302 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:17:20,303 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.42999982834. 2022-02-08 06:17:20,305 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:17:20,306 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:17:21,127 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:17:21,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:17:21,546 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:17:21,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:17:21,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:17:21,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:17:21,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:17:21,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:17:21,624 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:17:21,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:17:21,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:22,007 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:17:22,009 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.70399999619. 2022-02-08 06:17:22,009 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:17:22,010 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:17:22,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:17:22,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:17:23,022 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:17:23,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:17:23,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:17:23,053 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:17:23,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:17:23,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:17:23,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:17:23,127 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:17:23,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:23,411 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:17:23,411 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.40299987793. 2022-02-08 06:17:23,411 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:17:23,414 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:17:24,109 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:17:24,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:17:24,460 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:17:24,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:17:24,474 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:17:24,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:17:24,512 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:17:24,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:17:24,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:17:24,579 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:17:24,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:24,842 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:17:24,842 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.43000006676. 2022-02-08 06:17:24,842 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:17:24,845 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:17:25,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:17:25,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:17:25,816 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:17:25,816 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:17:25,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:17:25,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:17:25,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:17:25,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:17:25,890 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:17:25,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:17:25,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:26,223 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:17:26,223 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.38100004196. 2022-02-08 06:17:26,224 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:17:26,226 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:17:26,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:17:27,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:17:27,332 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:17:27,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:17:27,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:17:27,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:17:27,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:17:27,415 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:17:27,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:17:27,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:17:27,474 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:27,776 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:17:27,776 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.55100011826. 2022-02-08 06:17:27,776 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:17:27,779 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:17:28,423 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:17:28,665 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:17:28,759 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:17:28,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:17:28,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:17:28,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:17:28,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:17:28,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:17:28,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:17:28,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:17:28,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:29,144 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:17:29,144 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.36899995804. 2022-02-08 06:17:29,145 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:17:29,148 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:17:29,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:17:30,249 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:17:30,371 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:17:30,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:17:30,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:17:30,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:17:30,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:17:30,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:17:30,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:17:30,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:17:30,579 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:30,880 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:17:30,880 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 06:17:32,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:33,029 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:17:33,029 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 06:17:35,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:17:35,158 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:17:35,160 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.01400017738. ############################################################ # # RAM FootPrint: 4383.4140625 # ############################################################ 2022-02-08 06:17:35,190 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 06:17:35,190 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 92 seconds. Reason: Iteration:1 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 8583 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 8605 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:19:07,197 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 06:19:07,197 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 06:19:12,201 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:19:12,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:12,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:19:12,625 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:12,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:19:12,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:19:12,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:19:12,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:19:12,683 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:19:12,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:19:12,709 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:19:12,721 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:19:12,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:19:12,776 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['10283 memcached\n'], []) 2022-02-08 06:19:12,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 06:19:12,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:19:12,825 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['10283 memcached\n'], []) 2022-02-08 06:19:12,825 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:19:12,828 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:19:13,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:13,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:19:13,867 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:13,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:19:13,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:19:13,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:19:13,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:19:13,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:19:13,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:19:13,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:19:13,997 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:19:13,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:19:14,056 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['55850 memcached\n'], []) 2022-02-08 06:19:14,058 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 06:19:14,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:19:14,101 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['55850 memcached\n'], []) 2022-02-08 06:19:14,102 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:19:14,105 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:19:14,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:14,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:19:14,509 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:14,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:19:14,525 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:19:14,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:19:14,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:19:14,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:19:14,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:19:14,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:19:14,622 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:19:14,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:19:14,667 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['42456 memcached\n'], []) 2022-02-08 06:19:14,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 06:19:14,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:19:14,717 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['42456 memcached\n'], []) 2022-02-08 06:19:14,717 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:19:14,720 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:19:14,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:15,045 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:19:15,107 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:15,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:19:15,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:19:15,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:19:15,145 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:19:15,163 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:19:15,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:19:15,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:19:15,201 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:19:15,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:19:15,247 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['74899 memcached\n'], []) 2022-02-08 06:19:15,247 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 06:19:15,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:19:15,295 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['74899 memcached\n'], []) 2022-02-08 06:19:15,296 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:19:15,299 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:19:15,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:15,584 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:19:15,648 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:15,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:19:15,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:19:15,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:19:15,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:19:15,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:19:15,716 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:19:15,729 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:19:15,742 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:19:15,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:19:15,789 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['68761 memcached\n'], []) 2022-02-08 06:19:15,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 06:19:15,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:19:15,842 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['68761 memcached\n'], []) 2022-02-08 06:19:15,842 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:19:15,846 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:19:15,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:16,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:19:16,253 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:16,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:19:16,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:19:16,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:19:16,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:19:16,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:19:16,331 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:19:16,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:19:16,358 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:19:16,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:19:16,411 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['103754 memcached\n'], []) 2022-02-08 06:19:16,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 06:19:16,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:19:16,457 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['103754 memcached\n'], []) 2022-02-08 06:19:16,457 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:19:16,460 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:19:16,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:16,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:19:16,832 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:16,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:19:16,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:19:16,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:19:16,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:19:16,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:19:16,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:19:16,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:19:16,936 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:19:16,936 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:19:16,986 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['37524 memcached\n'], []) 2022-02-08 06:19:16,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 06:19:17,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:19:17,033 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['37524 memcached\n'], []) 2022-02-08 06:19:17,033 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:19:17,042 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:19:17,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:17,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:19:17,451 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:17,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:19:17,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:19:17,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:19:17,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:19:17,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:19:17,535 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:19:17,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:19:17,562 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:19:17,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:19:17,681 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['52444 memcached\n'], []) 2022-02-08 06:19:17,683 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 06:19:17,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:19:17,727 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['52444 memcached\n'], []) 2022-02-08 06:19:17,727 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:19:17,730 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:19:17,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:18,045 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:19:18,104 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:18,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:19:18,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:19:18,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:19:18,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:19:18,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:19:18,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:19:18,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:19:18,201 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:19:18,201 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:19:18,250 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['101834 memcached\n'], []) 2022-02-08 06:19:18,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 06:19:18,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:19:18,312 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['101834 memcached\n'], []) 2022-02-08 06:19:18,312 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:19:18,315 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:19:18,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:18,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:19:18,707 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:18,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:19:18,721 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:19:18,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:19:18,749 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:19:18,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:19:18,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:19:18,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:19:18,823 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:19:18,823 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:19:18,878 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['118496 memcached\n'], []) 2022-02-08 06:19:18,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 06:19:18,903 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:19:18,923 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['118496 memcached\n'], []) 2022-02-08 06:19:18,923 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:19:18,924 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:19:19,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:19,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:19:19,345 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:19,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:19:19,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:19:19,377 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:19:19,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:19:19,415 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:19:19,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:19:19,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:19:19,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:19:19,480 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 06:19:19,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:19:19,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:20,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:19:20,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:20,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:21,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:19:21,194 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 06:19:21,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:19:21,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:21,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:19:21,282 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 06:19:21,282 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:19:21,285 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:19:21,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:21,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:19:21,709 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:21,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:19:21,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:19:21,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:19:21,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:19:21,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:19:21,798 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:19:21,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:19:21,829 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:19:21,855 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 06:19:21,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:19:21,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:22,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:19:22,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:23,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:23,548 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:19:23,565 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 06:19:23,566 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:19:23,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:23,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:19:23,644 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 06:19:23,645 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:19:23,650 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:19:23,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:23,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:19:24,005 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:24,005 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:19:24,025 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:19:24,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:19:24,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:19:24,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:19:24,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:19:24,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:19:24,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:19:24,142 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 06:19:24,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:19:24,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:24,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:19:24,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:25,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:25,377 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:19:25,394 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 06:19:25,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:19:25,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:25,471 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:19:25,487 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 06:19:25,489 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:19:25,490 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 06:19:25,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:25,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 06:19:25,851 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:25,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 06:19:25,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 06:19:25,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 06:19:25,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 06:19:25,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 06:19:25,933 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 06:19:25,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 06:19:25,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:19:25,973 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 06:19:26,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:19:26,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:26,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:19:26,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:26,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:26,073 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:19:26,088 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 06:19:26,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:19:26,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:26,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:19:26,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:26,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:19:26,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:26,196 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:26,219 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:26,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:26,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:26,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:26,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:26,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:26,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:26,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:19:26,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:26,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:26,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:26,418 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:26,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:26,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:26,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:26,484 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:26,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:26,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:19:26,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:26,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:26,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:26,579 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:26,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:26,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:26,624 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:26,640 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:26,658 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:26,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:19:26,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:26,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:26,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:26,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:26,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:26,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:26,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:26,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:26,849 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:26,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:19:26,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:26,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:26,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:26,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:26,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:26,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:26,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:26,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:26,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:27,015 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:19:27,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:27,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:27,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:27,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:27,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:27,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:27,119 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:27,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:27,151 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 06:19:27,154 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:19:27,232 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:27,441 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:19:27,529 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:27,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:19:27,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:19:27,581 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:19:27,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:19:27,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:19:27,630 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:19:27,642 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:19:27,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:19:27,677 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 06:19:27,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:19:27,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:28,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:19:28,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:28,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:28,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:19:28,753 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 06:19:28,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:19:28,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:28,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:19:28,829 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 06:19:28,831 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:19:28,832 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 06:19:28,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:29,130 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 06:19:29,190 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:29,191 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 06:19:29,204 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 06:19:29,219 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 06:19:29,232 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 06:19:29,247 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 06:19:29,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 06:19:29,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 06:19:29,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:19:29,302 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 06:19:29,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:19:29,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:29,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:19:29,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:29,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:29,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:19:29,421 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 06:19:29,423 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:19:29,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:29,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:19:29,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:29,497 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:19:29,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:29,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:29,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:29,589 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:29,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:29,647 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:29,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:29,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:29,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:29,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:19:29,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:29,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:29,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:29,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:29,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:29,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:29,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:29,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:29,923 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:29,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:19:29,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:29,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:29,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:30,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:30,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:30,035 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:30,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:30,071 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:30,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:30,104 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:19:30,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:30,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:30,155 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:30,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:30,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:30,214 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:30,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:30,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:30,270 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:30,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:19:30,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:30,321 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:30,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:30,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:30,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:30,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:30,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:30,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:30,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:30,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:19:30,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:30,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:30,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:30,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:30,530 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:30,548 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:30,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:30,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:30,608 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 06:19:30,611 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:19:30,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:30,910 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:19:30,977 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:30,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:19:30,997 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:19:31,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:19:31,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:19:31,043 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:19:31,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:19:31,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:19:31,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:19:31,102 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 06:19:31,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:19:31,147 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:31,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:19:32,227 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:32,573 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:32,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:19:32,878 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 06:19:32,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:19:32,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:32,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:19:32,953 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 06:19:32,953 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:19:32,957 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:19:33,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:33,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:19:33,306 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:33,308 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:19:33,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:19:33,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:19:33,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:19:33,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:19:33,387 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:19:33,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:19:33,423 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:19:33,440 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 06:19:33,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:19:33,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:33,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:19:34,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:34,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:35,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:19:35,049 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 06:19:35,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:19:35,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:35,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:19:35,154 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 06:19:35,154 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:19:35,157 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 06:19:35,242 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:35,466 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 06:19:35,529 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:35,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 06:19:35,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 06:19:35,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 06:19:35,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 06:19:35,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 06:19:35,599 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 06:19:35,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 06:19:35,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:19:35,661 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 06:19:35,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:19:35,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:35,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:19:35,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:35,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:35,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:19:35,780 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 06:19:35,780 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:19:35,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:35,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:19:35,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:35,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:19:35,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:35,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:35,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:35,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:35,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:35,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:35,997 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:36,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:36,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:36,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:19:36,073 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:36,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:36,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:36,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:36,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:36,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:36,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:36,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:36,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:36,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:19:36,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:36,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:36,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:36,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:36,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:36,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:36,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:36,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:36,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:36,385 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:19:36,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:36,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:36,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:36,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:36,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:36,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:36,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:36,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:36,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:36,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:19:36,571 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:36,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:36,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:36,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:36,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:36,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:36,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:36,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:36,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:36,721 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:19:36,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:36,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:36,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:36,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:36,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:36,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:36,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:36,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:36,924 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 06:19:36,927 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 06:19:37,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:37,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 06:19:37,332 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:37,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 06:19:37,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 06:19:37,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 06:19:37,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 06:19:37,397 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 06:19:37,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 06:19:37,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 06:19:37,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:19:37,456 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 06:19:37,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:19:37,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:37,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:19:37,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:37,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:37,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:19:37,582 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 06:19:37,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:19:37,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:37,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:19:37,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:37,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:19:37,678 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:37,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:37,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:37,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:37,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:37,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:37,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:37,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:37,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:37,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:19:37,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:37,933 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:37,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:37,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:37,989 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:38,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:38,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:38,045 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:38,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:38,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:19:38,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:38,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:38,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:38,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:38,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:38,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:38,198 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:38,216 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:38,232 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:38,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:19:38,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:38,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:38,303 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:38,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:38,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:38,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:38,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:38,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:38,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:38,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:19:38,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:38,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:38,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:38,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:38,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:38,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:38,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:38,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:38,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:38,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:19:38,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:38,632 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:38,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:38,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:38,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:38,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:38,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:38,749 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:38,789 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 06:19:38,792 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:19:38,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:39,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:19:39,204 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:39,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:19:39,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:19:39,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:19:39,255 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:19:39,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:19:39,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:19:39,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:19:39,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:19:39,335 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 06:19:39,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:19:39,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:39,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:19:40,331 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:40,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:40,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:19:40,915 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 06:19:40,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:19:40,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:40,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:19:41,003 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 06:19:41,005 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:19:41,006 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:19:41,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:41,316 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:19:41,381 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:41,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:19:41,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:19:41,408 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:19:41,423 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:19:41,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:19:41,454 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:19:41,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:19:41,484 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:19:41,503 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 06:19:41,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:19:41,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:41,943 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:19:42,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:42,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:42,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:19:42,815 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 06:19:42,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:19:42,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:42,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:19:42,897 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 06:19:42,897 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:19:42,900 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:19:42,997 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:43,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:19:43,286 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:43,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:19:43,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:19:43,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:19:43,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:19:43,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:19:43,362 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:19:43,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:19:43,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:19:43,410 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 06:19:43,438 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:19:43,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:43,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:19:44,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:44,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:44,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:19:44,654 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 06:19:44,655 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:19:44,670 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:44,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:19:44,709 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 06:19:44,710 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:19:44,711 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:19:44,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:45,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:19:45,124 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:45,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:19:45,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:19:45,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:19:45,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:19:45,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:19:45,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:19:45,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:19:45,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:19:45,255 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 06:19:45,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:19:45,293 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:45,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:19:46,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:46,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:46,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:19:46,963 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 06:19:46,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:19:46,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:47,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:19:47,061 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 06:19:47,061 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:19:47,063 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 06:19:47,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:47,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 06:19:47,431 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:47,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 06:19:47,446 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 06:19:47,459 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 06:19:47,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 06:19:47,487 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 06:19:47,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 06:19:47,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 06:19:47,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:19:47,545 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 06:19:47,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:19:47,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:47,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:19:47,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:47,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:19:47,678 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:19:47,694 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 06:19:47,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:19:47,711 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:19:47,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:19:47,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:47,760 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:19:47,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:47,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:47,834 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:47,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:47,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:47,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:47,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:47,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:19:47,969 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:47,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:19:48,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:48,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:48,045 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:48,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:48,081 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:48,099 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:48,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:48,140 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:19:48,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:48,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:19:48,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:48,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:48,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:48,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:48,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:48,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:48,295 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:48,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:19:48,326 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:48,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:19:48,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:48,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:48,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:48,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:48,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:48,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:48,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:48,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:19:48,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:48,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:19:48,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:48,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:48,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:48,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:48,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:48,579 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:48,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:48,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:19:48,624 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:48,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:19:48,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:48,678 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:48,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:48,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:48,727 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:48,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:48,770 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:48,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:19:48,812 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 06:19:48,813 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:19:48,815 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:19:49,405 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 8626 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:19:49,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:19:49,780 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:49,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:19:49,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:19:49,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:19:49,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:19:49,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:19:49,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:19:49,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:19:49,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:19:50,206 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:19:50,206 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 06:19:52,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:19:52,351 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:19:52,352 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 06:19:54,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:19:54,493 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:19:54,494 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 06:19:56,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:19:56,628 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:19:56,628 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.81500005722. 2022-02-08 06:19:56,628 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:19:56,631 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:19:56,721 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:19:57,017 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:19:57,104 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:19:57,104 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:19:57,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:19:57,140 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:19:57,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:19:57,178 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:19:57,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:19:57,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:19:57,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:19:57,391 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:19:57,392 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:19:59,392 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:19:59,562 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:19:59,562 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:01,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:01,719 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:01,719 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:03,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:03,862 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:03,862 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:05,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:06,007 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:06,007 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:08,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:08,164 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:08,164 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:10,165 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:10,316 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:10,318 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:12,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:12,464 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:12,464 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:14,466 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:14,617 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:14,618 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:16,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:16,770 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:16,772 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:18,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:18,943 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:18,943 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:20,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:21,157 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:21,158 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:23,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:23,400 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:23,400 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:25,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:25,575 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:25,575 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:27,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:27,819 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:27,819 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:29,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:30,025 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:30,026 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:32,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:32,174 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:32,174 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:34,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:34,338 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:34,339 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:36,339 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:36,489 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:36,489 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:38,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:38,635 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:38,637 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:40,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:40,778 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:40,779 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:42,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:42,947 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:42,947 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:44,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:45,094 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:45,094 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:47,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:47,247 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:47,249 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:49,249 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:49,401 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:49,401 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 8626 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:20:51,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:51,565 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:51,565 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:53,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:53,719 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:53,719 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:55,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:55,865 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:55,865 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:20:57,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:20:58,010 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:20:58,012 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:21:00,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:21:00,154 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:21:00,155 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:21:02,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:21:02,301 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:21:02,301 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:21:04,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:21:04,444 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:21:04,446 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:21:06,446 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:21:06,582 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:21:06,582 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 69.9539999962. 2022-02-08 06:21:06,582 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:21:06,585 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:21:07,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:21:07,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:21:07,740 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:21:07,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:21:07,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:21:07,778 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:21:07,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:21:07,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:21:07,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:21:07,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:21:07,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:21:08,144 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:21:08,144 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.56200003624. 2022-02-08 06:21:08,144 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:21:08,147 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:21:08,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:21:09,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:21:09,269 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:21:09,270 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:21:09,285 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:21:09,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:21:09,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:21:09,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:21:09,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:21:09,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:21:09,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:21:09,707 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:21:09,707 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.56299996376. 2022-02-08 06:21:09,707 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:21:09,710 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:21:10,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:21:10,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:21:10,898 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:21:10,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:21:10,913 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:21:10,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:21:10,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:21:10,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:21:10,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:21:11,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:21:11,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:21:12,109 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:21:12,111 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.40300011635. 2022-02-08 06:21:12,111 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:21:12,112 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:21:12,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:21:13,066 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:21:13,157 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:21:13,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:21:13,170 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:21:13,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:21:13,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:21:13,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:21:13,285 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:21:13,316 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:21:13,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:21:13,894 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:21:13,894 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.78299999237. 2022-02-08 06:21:13,894 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:21:13,895 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:21:14,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:21:14,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:21:14,746 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:21:14,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:21:14,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:21:14,783 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:21:14,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:21:14,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:21:14,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:21:14,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:21:14,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:21:15,148 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:21:15,148 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.25499987602. 2022-02-08 06:21:15,150 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:21:15,151 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:21:16,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:21:16,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:21:16,825 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:21:16,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:21:16,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:21:16,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:21:16,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:21:16,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:21:16,903 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:21:16,928 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:21:16,941 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:21:17,746 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:21:17,747 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.59799981117. 2022-02-08 06:21:17,747 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:21:17,750 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:21:18,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:21:18,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:21:18,586 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:21:18,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:21:18,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:21:18,619 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:21:18,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:21:18,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:21:18,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:21:18,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:21:18,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:21:20,609 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:21:20,609 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.86199998856. 2022-02-08 06:21:20,611 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:21:20,612 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:21:20,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:21:20,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:21:20,997 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:21:20,997 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:21:21,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:21:21,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:21:21,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:21:21,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:21:21,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:21:21,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:21:21,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:21:21,698 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:21:21,700 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.08899998665. ############################################################ # # RAM FootPrint: 4384.41796875 # ############################################################ 2022-02-08 06:21:21,736 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 06:21:21,737 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 94 seconds. Reason: Iteration:2 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 8645 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 8669 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:22:55,743 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 06:22:55,743 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 06:23:00,746 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:23:00,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:01,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:23:01,187 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:01,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:23:01,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:23:01,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:23:01,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:23:01,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:23:01,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:23:01,293 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:23:01,308 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:23:01,308 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:23:01,368 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['10994 memcached\n'], []) 2022-02-08 06:23:01,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 06:23:01,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:23:01,418 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['10994 memcached\n'], []) 2022-02-08 06:23:01,420 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:23:01,421 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:23:01,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:02,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:23:02,290 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:02,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:23:02,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:23:02,329 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:23:02,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:23:02,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:23:02,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:23:02,397 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:23:02,411 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:23:02,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:23:02,467 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['56740 memcached\n'], []) 2022-02-08 06:23:02,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 06:23:02,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:23:02,519 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['56740 memcached\n'], []) 2022-02-08 06:23:02,520 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:23:02,522 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:23:02,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:02,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:23:02,931 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:02,931 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:23:02,948 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:23:02,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:23:02,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:23:02,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:23:03,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:23:03,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:23:03,048 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:23:03,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:23:03,094 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['43218 memcached\n'], []) 2022-02-08 06:23:03,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 06:23:03,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:23:03,147 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['43218 memcached\n'], []) 2022-02-08 06:23:03,147 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:23:03,150 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:23:03,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:03,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:23:03,539 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:03,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:23:03,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:23:03,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:23:03,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:23:03,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:23:03,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:23:03,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:23:03,635 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:23:03,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:23:03,703 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['75609 memcached\n'], []) 2022-02-08 06:23:03,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 06:23:03,732 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:23:03,759 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['75609 memcached\n'], []) 2022-02-08 06:23:03,759 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:23:03,762 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:23:03,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:04,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:23:04,115 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:04,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:23:04,130 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:23:04,142 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:23:04,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:23:04,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:23:04,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:23:04,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:23:04,210 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:23:04,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:23:04,257 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['69429 memcached\n'], []) 2022-02-08 06:23:04,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 06:23:04,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:23:04,306 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['69429 memcached\n'], []) 2022-02-08 06:23:04,306 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:23:04,309 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:23:04,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:04,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:23:04,671 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:04,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:23:04,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:23:04,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:23:04,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:23:04,727 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:23:04,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:23:04,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:23:04,766 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:23:04,767 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:23:04,819 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['104423 memcached\n'], []) 2022-02-08 06:23:04,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 06:23:04,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:23:04,865 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['104423 memcached\n'], []) 2022-02-08 06:23:04,865 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:23:04,868 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:23:04,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:05,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:23:05,233 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:05,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:23:05,247 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:23:05,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:23:05,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:23:05,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:23:05,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:23:05,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:23:05,328 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:23:05,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:23:05,378 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['38237 memcached\n'], []) 2022-02-08 06:23:05,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 06:23:05,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:23:05,424 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['38237 memcached\n'], []) 2022-02-08 06:23:05,426 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:23:05,427 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:23:05,507 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:05,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:23:05,789 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:05,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:23:05,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:23:05,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:23:05,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:23:05,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:23:05,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:23:05,869 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:23:05,882 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:23:05,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:23:05,944 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['53155 memcached\n'], []) 2022-02-08 06:23:05,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 06:23:05,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:23:05,989 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['53155 memcached\n'], []) 2022-02-08 06:23:05,989 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:23:05,992 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:23:06,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:06,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:23:06,328 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:06,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:23:06,341 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:23:06,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:23:06,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:23:06,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:23:06,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:23:06,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:23:06,424 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:23:06,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:23:06,479 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['102548 memcached\n'], []) 2022-02-08 06:23:06,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 06:23:06,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:23:06,532 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['102548 memcached\n'], []) 2022-02-08 06:23:06,532 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:23:06,533 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:23:06,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:06,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:23:06,907 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:06,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:23:06,923 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:23:06,936 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:23:06,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:23:06,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:23:06,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:23:06,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:23:07,013 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:23:07,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:23:07,072 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['119200 memcached\n'], []) 2022-02-08 06:23:07,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 06:23:07,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:23:07,119 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['119200 memcached\n'], []) 2022-02-08 06:23:07,119 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:23:07,121 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:23:07,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:07,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:23:07,494 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:07,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:23:07,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:23:07,525 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:23:07,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:23:07,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:23:07,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:23:07,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:23:07,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:23:07,624 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 06:23:07,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:23:07,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:11,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:23:11,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:12,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:12,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:23:12,601 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 06:23:12,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:23:12,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:12,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:23:12,684 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 06:23:12,684 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:23:12,687 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:23:12,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:13,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:23:13,088 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:13,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:23:13,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:23:13,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:23:13,127 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:23:13,142 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:23:13,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:23:13,170 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:23:13,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:23:13,197 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 06:23:13,224 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:23:13,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:13,721 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:23:14,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:14,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:14,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:23:14,865 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 06:23:14,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:23:14,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:14,943 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:23:14,956 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 06:23:14,957 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:23:14,959 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:23:15,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:15,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:23:15,299 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:15,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:23:15,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:23:15,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:23:15,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:23:15,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:23:15,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:23:15,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:23:15,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:23:15,414 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 06:23:15,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:23:15,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:15,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:23:16,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:16,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:16,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:23:16,714 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 06:23:16,716 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:23:16,727 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:16,783 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:23:16,798 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 06:23:16,798 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:23:16,801 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 06:23:16,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:17,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 06:23:17,203 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:17,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 06:23:17,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 06:23:17,234 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 06:23:17,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 06:23:17,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 06:23:17,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 06:23:17,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 06:23:17,321 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:23:17,338 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 06:23:17,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:23:17,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:17,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:23:17,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:17,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:17,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:23:17,457 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 06:23:17,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:23:17,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:17,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:23:17,507 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:17,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:23:17,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:17,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:17,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:17,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:17,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:17,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:17,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:17,749 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:17,770 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:17,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:23:17,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:17,829 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:17,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:17,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:17,885 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:17,903 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:17,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:17,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:17,956 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:17,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:23:17,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:18,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:18,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:18,045 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:18,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:18,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:18,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:18,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:18,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:18,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:23:18,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:18,183 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:18,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:18,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:18,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:18,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:18,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:18,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:18,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:18,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:23:18,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:18,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:18,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:18,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:18,405 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:18,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:18,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:18,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:18,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:18,487 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:23:18,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:18,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:18,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:18,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:18,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:18,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:18,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:18,630 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:18,648 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 06:23:18,650 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:23:18,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:19,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:23:19,733 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:19,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:23:19,778 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:23:19,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:23:19,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:23:19,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:23:20,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:23:20,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:23:20,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:23:20,263 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 06:23:20,290 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:23:20,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:20,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:23:21,211 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:21,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:21,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:23:21,854 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 06:23:21,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:23:21,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:21,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:23:21,977 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 06:23:21,979 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:23:21,980 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 06:23:22,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:22,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 06:23:22,361 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:22,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 06:23:22,377 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 06:23:22,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 06:23:22,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 06:23:22,428 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 06:23:22,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 06:23:22,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 06:23:22,471 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:23:22,487 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 06:23:22,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:23:22,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:22,545 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:23:22,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:22,573 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:22,586 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:23:22,601 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 06:23:22,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:23:22,614 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:22,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:23:22,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:22,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:23:22,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:22,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:22,749 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:22,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:22,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:22,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:22,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:22,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:22,913 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:22,931 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:23:22,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:22,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:22,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:23,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:23,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:23,045 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:23,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:23,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:23,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:23,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:23:23,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:23,145 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:23,163 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:23,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:23,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:23,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:23,232 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:23,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:23,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:23,283 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:23:23,303 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:23,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:23,336 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:23,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:23,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:23,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:23,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:23,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:23,438 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:23,454 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:23:23,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:23,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:23,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:23,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:23,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:23,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:23,563 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:23,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:23,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:23,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:23:23,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:23,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:23,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:23,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:23,688 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:23,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:23,721 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:23,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:23,752 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 06:23:23,755 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:23:23,834 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:24,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:23:24,095 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:24,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:23:24,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:23:24,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:23:24,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:23:24,147 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:23:24,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:23:24,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:23:24,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:23:24,206 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 06:23:24,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:23:24,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:24,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:23:25,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:25,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:25,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:23:25,984 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 06:23:25,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:23:25,997 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:26,043 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:23:26,056 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 06:23:26,056 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:23:26,059 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:23:26,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:26,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:23:26,421 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:26,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:23:26,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:23:26,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:23:26,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:23:26,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:23:26,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:23:26,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:23:26,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:23:26,555 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 06:23:26,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:23:26,599 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:27,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:23:27,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:27,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:28,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:23:28,111 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 06:23:28,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:23:28,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:28,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:23:28,232 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 06:23:28,233 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:23:28,236 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 06:23:28,329 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:28,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 06:23:28,608 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:28,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 06:23:28,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 06:23:28,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 06:23:28,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 06:23:28,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 06:23:28,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 06:23:28,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 06:23:28,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:23:28,726 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 06:23:28,752 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:23:28,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:28,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:23:28,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:28,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:28,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:23:28,831 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 06:23:28,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:23:28,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:28,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:23:28,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:28,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:23:28,918 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:28,941 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:28,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:28,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:29,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:29,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:29,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:29,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:29,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:29,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:23:29,130 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:29,145 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:29,163 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:29,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:29,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:29,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:29,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:29,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:29,280 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:29,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:23:29,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:29,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:29,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:29,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:29,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:29,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:29,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:29,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:29,448 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:29,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:23:29,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:29,497 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:29,512 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:29,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:29,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:29,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:29,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:29,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:29,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:29,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:23:29,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:29,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:29,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:29,683 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:29,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:29,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:29,729 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:29,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:29,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:29,780 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:23:29,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:29,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:29,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:29,849 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:29,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:29,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:29,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:29,920 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:29,937 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 06:23:29,940 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 06:23:30,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:30,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 06:23:30,312 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:30,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 06:23:30,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 06:23:30,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 06:23:30,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 06:23:30,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 06:23:30,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 06:23:30,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 06:23:30,408 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:23:30,424 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 06:23:30,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:23:30,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:30,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:23:30,497 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:30,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:30,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:23:30,538 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 06:23:30,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:23:30,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:30,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:23:30,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:30,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:23:30,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:30,683 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:30,709 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:30,732 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:30,760 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:30,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:30,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:30,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:30,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:30,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:23:30,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:30,910 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:30,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:30,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:30,956 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:30,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:30,989 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:31,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:31,017 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:31,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:23:31,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:31,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:31,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:31,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:31,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:31,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:31,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:31,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:31,165 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:31,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:23:31,196 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:31,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:31,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:31,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:31,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:31,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:31,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:31,295 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:31,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:31,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:23:31,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:31,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:31,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:31,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:31,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:31,415 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:31,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:31,448 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:31,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:31,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:23:31,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:31,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:31,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:31,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:31,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:31,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:31,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:31,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:31,642 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 06:23:31,644 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:23:31,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:31,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:23:32,049 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:32,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:23:32,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:23:32,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:23:32,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:23:32,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:23:32,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:23:32,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:23:32,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:23:32,180 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 06:23:32,221 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:23:32,239 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:32,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:23:33,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:33,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:33,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:23:33,750 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 06:23:33,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:23:33,765 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:33,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:23:33,851 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 06:23:33,851 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:23:33,852 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:23:33,936 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:34,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:23:34,236 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:34,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:23:34,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:23:34,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:23:34,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:23:34,295 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:23:34,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:23:34,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:23:34,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:23:34,375 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 06:23:34,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:23:34,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:36,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:23:36,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:37,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:37,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:23:37,298 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 06:23:37,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:23:37,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:37,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:23:37,358 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 06:23:37,358 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:23:37,361 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:23:37,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:38,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:23:38,400 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:38,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:23:38,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:23:38,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:23:38,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:23:38,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:23:38,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:23:38,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:23:38,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:23:38,555 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 06:23:38,581 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:23:38,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:41,528 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:23:41,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:42,163 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:42,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:23:42,378 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 06:23:42,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:23:42,392 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:42,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:23:42,430 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 06:23:42,430 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:23:42,433 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:23:42,515 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:42,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:23:42,819 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:42,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:23:42,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:23:42,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:23:42,869 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:23:42,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:23:42,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:23:42,918 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:23:42,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:23:42,959 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 06:23:42,989 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:23:43,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:43,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:23:44,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:44,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:44,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:23:44,667 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 06:23:44,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:23:44,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:44,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:23:44,744 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 06:23:44,744 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:23:44,747 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 06:23:44,829 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:45,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 06:23:45,111 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:45,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 06:23:45,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 06:23:45,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 06:23:45,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 06:23:45,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 06:23:45,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 06:23:45,193 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 06:23:45,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:23:45,221 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 06:23:45,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:23:45,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:45,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:23:45,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:45,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:23:45,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:23:45,328 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 06:23:45,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:23:45,341 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:23:45,357 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:23:45,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:45,392 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:23:45,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:45,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:45,466 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:45,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:45,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:45,548 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:45,571 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:45,596 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:23:45,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:45,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:23:45,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:45,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:45,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:45,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:45,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:45,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:45,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:45,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:23:45,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:45,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:23:45,816 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:45,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:45,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:45,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:45,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:45,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:45,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:45,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:23:45,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:45,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:23:45,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:46,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:46,022 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:46,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:46,053 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:46,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:46,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:46,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:23:46,112 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:46,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:23:46,142 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:46,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:46,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:46,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:46,201 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:46,216 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:46,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:46,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:23:46,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:46,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:23:46,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:46,308 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:46,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:46,341 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:46,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:46,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:46,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:46,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:23:46,420 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 06:23:46,420 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:23:46,423 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:23:46,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:46,716 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:23:46,773 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:46,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:23:46,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:23:46,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:23:46,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:23:46,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:23:46,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:23:46,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:23:46,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:23:47,299 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:23:47,301 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.881000041962. 2022-02-08 06:23:47,302 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:23:47,302 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:23:47,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:23:47,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:23:47,684 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:23:47,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:23:47,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:23:47,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:23:47,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:23:47,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:23:47,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:23:47,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:23:47,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:23:47,980 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:23:47,980 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 8684 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:23:49,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:23:50,138 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:23:50,140 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:23:52,140 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:23:52,414 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:23:52,415 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:23:54,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:23:54,592 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:23:54,592 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:23:56,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:23:56,805 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:23:56,805 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:23:58,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:23:58,973 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:23:58,973 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:00,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:01,170 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:01,170 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:03,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:03,375 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:03,375 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:05,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:05,542 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:05,542 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:07,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:07,713 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:07,713 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:09,716 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:09,865 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:09,865 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:11,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:12,022 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:12,022 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:14,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:14,164 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:14,164 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:16,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:16,315 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:16,315 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:18,316 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:18,470 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:18,471 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:20,471 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:20,621 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:20,621 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:22,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:22,766 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:22,766 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:24,767 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:24,915 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:24,915 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:26,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:27,065 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:27,065 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:29,066 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:29,217 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:29,217 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:31,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:31,368 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:31,368 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:33,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:33,517 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:33,517 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:35,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:35,663 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:35,664 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:37,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:37,811 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:37,811 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:39,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:39,960 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:39,960 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:41,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:42,108 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:42,108 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:24:44,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:44,247 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:24:44,249 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 56.9459998608. 2022-02-08 06:24:44,249 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:24:44,250 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:24:44,829 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:24:45,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:24:45,173 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:24:45,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:24:45,191 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:24:45,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:24:45,229 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:24:45,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:24:45,270 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:24:45,303 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:24:45,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:45,559 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:24:45,559 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.3109998703. 2022-02-08 06:24:45,559 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:24:45,562 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:24:46,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:24:46,454 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:24:46,552 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:24:46,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:24:46,566 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:24:46,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:24:46,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:24:46,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:24:46,630 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:24:46,658 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:24:46,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:47,359 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:24:47,359 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.79999995232. 2022-02-08 06:24:47,361 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:24:47,362 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:24:48,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:24:48,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:24:48,438 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:24:48,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:24:48,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:24:48,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:24:48,497 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:24:48,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:24:48,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:24:48,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:24:48,586 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:49,092 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:49,092 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 5 queries submitted, 0 failed, 8686 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:24:51,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:51,247 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:24:51,249 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.88900017738. 2022-02-08 06:24:51,249 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:24:51,250 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:24:52,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:24:52,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:24:52,381 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:24:52,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:24:52,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:24:52,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:24:52,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:24:52,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:24:52,464 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:24:52,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:24:52,525 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:53,062 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:24:53,062 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.81299996376. 2022-02-08 06:24:53,062 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:24:53,065 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:24:53,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:24:54,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:24:54,384 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:24:54,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:24:54,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:24:54,413 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:24:54,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:24:54,441 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:24:54,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:24:54,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:24:54,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:54,944 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:24:54,944 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.88199996948. 2022-02-08 06:24:54,944 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:24:54,947 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:24:55,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:24:56,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:24:56,855 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:24:56,857 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:24:57,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:24:57,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:24:57,349 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:24:57,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:24:57,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:24:57,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:24:57,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:24:58,440 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:24:58,440 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 06:25:00,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:25:02,858 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:25:02,859 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.91499996185. 2022-02-08 06:25:02,859 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:25:02,861 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:25:03,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:25:03,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:25:03,809 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:25:03,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:25:03,823 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:25:03,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:25:03,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:25:03,895 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:25:03,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:25:03,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:25:03,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:25:04,813 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:25:04,815 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.95500016212. 2022-02-08 06:25:04,815 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:25:04,816 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:25:05,645 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:25:06,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:25:06,210 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:25:06,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:25:06,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:25:06,242 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:25:06,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:25:06,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:25:06,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:25:06,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:25:06,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:25:08,076 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:25:08,076 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.2619998455. ############################################################ # # RAM FootPrint: 4393.18359375 # ############################################################ 2022-02-08 06:25:08,275 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 06:25:08,275 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 76 seconds. Reason: Iteration:3 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 8709 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:26:24,282 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 06:26:24,282 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 06:26:29,288 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:26:29,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:29,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:26:29,694 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:29,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:26:29,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:26:29,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:26:29,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:26:29,762 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:26:29,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:26:29,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:26:29,805 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:26:29,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:26:29,869 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['11754 memcached\n'], []) 2022-02-08 06:26:29,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 06:26:29,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:26:29,917 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['11754 memcached\n'], []) 2022-02-08 06:26:29,917 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:26:29,920 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:26:30,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:30,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:26:30,825 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:30,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:26:30,841 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:26:30,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:26:30,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:26:30,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:26:30,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:26:30,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:26:30,966 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:26:30,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:26:31,030 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['57799 memcached\n'], []) 2022-02-08 06:26:31,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 06:26:31,056 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:26:31,076 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['57799 memcached\n'], []) 2022-02-08 06:26:31,078 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:26:31,079 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:26:31,150 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:31,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:26:31,447 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:31,448 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:26:31,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:26:31,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:26:31,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:26:31,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:26:31,528 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:26:31,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:26:31,559 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:26:31,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:26:31,608 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['44038 memcached\n'], []) 2022-02-08 06:26:31,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 06:26:31,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:26:31,661 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['44038 memcached\n'], []) 2022-02-08 06:26:31,661 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:26:31,664 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:26:31,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:31,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:26:32,051 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:32,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:26:32,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:26:32,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:26:32,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:26:32,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:26:32,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:26:32,130 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:26:32,144 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:26:32,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:26:32,190 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['76329 memcached\n'], []) 2022-02-08 06:26:32,191 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 06:26:32,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:26:32,246 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['76329 memcached\n'], []) 2022-02-08 06:26:32,246 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:26:32,250 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:26:32,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:32,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:26:32,621 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:32,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:26:32,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:26:32,647 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:26:32,660 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:26:32,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:26:32,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:26:32,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:26:32,717 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:26:32,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:26:32,766 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['70147 memcached\n'], []) 2022-02-08 06:26:32,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 06:26:32,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:26:32,815 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['70147 memcached\n'], []) 2022-02-08 06:26:32,815 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:26:32,819 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:26:32,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:33,130 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:26:33,190 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:33,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:26:33,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:26:33,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:26:33,234 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:26:33,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:26:33,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:26:33,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:26:33,290 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:26:33,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:26:33,349 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['105144 memcached\n'], []) 2022-02-08 06:26:33,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 06:26:33,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:26:33,408 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['105144 memcached\n'], []) 2022-02-08 06:26:33,408 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:26:33,413 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:26:33,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:33,711 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:26:33,796 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:33,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:26:33,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:26:33,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:26:33,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:26:33,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:26:33,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:26:33,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:26:33,895 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:26:33,895 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:26:33,951 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['38961 memcached\n'], []) 2022-02-08 06:26:33,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 06:26:33,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:26:33,999 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['38961 memcached\n'], []) 2022-02-08 06:26:34,000 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:26:34,002 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:26:34,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:34,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:26:34,361 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:34,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:26:34,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:26:34,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:26:34,408 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:26:34,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:26:34,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:26:34,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:26:34,466 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:26:34,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:26:34,529 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['53872 memcached\n'], []) 2022-02-08 06:26:34,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 06:26:34,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:26:34,573 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['53872 memcached\n'], []) 2022-02-08 06:26:34,575 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:26:34,576 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:26:34,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:34,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:26:34,944 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:34,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:26:34,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:26:34,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:26:34,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:26:35,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:26:35,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:26:35,045 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:26:35,059 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:26:35,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:26:35,105 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['103274 memcached\n'], []) 2022-02-08 06:26:35,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 06:26:35,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:26:35,155 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['103274 memcached\n'], []) 2022-02-08 06:26:35,157 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:26:35,158 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:26:35,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:35,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:26:35,533 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:35,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:26:35,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:26:35,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:26:35,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:26:35,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:26:35,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:26:35,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:26:35,631 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:26:35,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:26:35,713 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['119923 memcached\n'], []) 2022-02-08 06:26:35,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 06:26:35,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:26:35,763 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['119923 memcached\n'], []) 2022-02-08 06:26:35,765 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:26:35,769 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:26:35,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:36,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:26:36,167 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:36,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:26:36,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:26:36,193 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:26:36,204 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:26:36,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:26:36,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:26:36,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:26:36,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:26:36,279 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 06:26:36,306 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:26:36,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:37,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:26:37,885 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:38,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:38,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:26:38,573 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 06:26:38,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:26:38,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:38,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:26:38,653 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 06:26:38,653 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:26:38,657 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:26:38,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:38,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:26:39,020 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:39,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:26:39,035 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:26:39,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:26:39,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:26:39,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:26:39,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:26:39,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:26:39,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:26:39,140 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 06:26:39,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:26:39,191 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:39,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:26:40,234 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:40,573 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:40,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:26:40,871 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 06:26:40,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:26:40,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:40,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:26:40,960 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 06:26:40,961 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:26:40,963 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:26:41,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:41,247 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:26:41,309 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:41,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:26:41,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:26:41,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:26:41,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:26:41,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:26:41,385 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:26:41,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:26:41,418 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:26:41,434 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 06:26:41,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:26:41,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:41,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:26:42,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:42,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:42,678 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:26:42,693 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 06:26:42,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:26:42,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:42,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:26:42,789 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 06:26:42,789 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:26:42,792 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 06:26:42,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:43,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 06:26:43,173 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:43,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 06:26:43,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 06:26:43,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 06:26:43,214 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 06:26:43,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 06:26:43,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 06:26:43,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 06:26:43,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:26:43,292 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 06:26:43,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:26:43,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:43,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:26:43,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:43,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:43,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:26:43,411 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 06:26:43,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:26:43,426 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:43,441 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:26:43,456 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:43,482 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:26:43,512 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:43,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:43,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:43,589 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:43,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:43,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:43,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:43,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:43,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:43,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:26:43,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:43,780 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:43,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:43,816 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:43,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:43,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:43,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:43,890 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:43,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:43,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:26:43,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:43,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:43,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:43,989 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:44,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:44,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:44,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:44,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:44,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:44,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:26:44,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:44,140 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:44,155 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:44,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:44,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:44,204 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:44,221 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:44,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:44,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:44,272 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:26:44,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:44,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:44,329 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:44,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:44,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:44,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:44,397 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:44,413 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:44,428 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:44,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:26:44,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:44,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:44,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:44,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:44,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:44,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:44,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:44,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:44,611 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 06:26:44,612 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:26:44,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:45,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:26:45,821 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:45,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:26:45,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:26:46,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:26:46,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:26:46,466 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:26:46,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:26:46,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:26:46,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:26:47,095 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 06:26:47,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:26:47,227 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 8737 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:26:50,015 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:26:50,505 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:50,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:51,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:26:51,053 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 06:26:51,053 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:26:51,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:51,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:26:51,282 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 06:26:51,283 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:26:51,286 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 06:26:51,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:51,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 06:26:51,671 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:51,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 06:26:51,688 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 06:26:51,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 06:26:51,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 06:26:51,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 06:26:51,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 06:26:51,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 06:26:51,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:26:51,802 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 06:26:51,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:26:51,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:51,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:26:51,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:51,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:51,910 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:26:51,924 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 06:26:51,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:26:51,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:51,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:26:51,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:51,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:26:52,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:52,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:52,073 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:52,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:52,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:52,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:52,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:52,201 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:52,227 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:52,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:26:52,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:52,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:52,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:52,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:52,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:52,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:52,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:52,387 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:52,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:52,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:26:52,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:52,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:52,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:52,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:52,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:52,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:52,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:52,563 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:52,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:52,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:26:52,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:52,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:52,642 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:52,658 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:52,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:52,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:52,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:52,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:52,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:52,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:26:52,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:52,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:52,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:52,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:52,834 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:52,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:52,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:52,895 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:52,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:52,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:26:52,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:52,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:52,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:53,015 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:53,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:53,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:53,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:53,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:53,114 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 06:26:53,115 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:26:53,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:53,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:26:53,513 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:53,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:26:53,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:26:53,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:26:53,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:26:53,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:26:53,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:26:53,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:26:53,619 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:26:53,637 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 06:26:53,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:26:53,678 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:54,196 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:26:54,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:55,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:55,387 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:26:55,405 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 06:26:55,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:26:55,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:55,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:26:55,482 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 06:26:55,483 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:26:55,486 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:26:55,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:55,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:26:55,849 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:55,849 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:26:55,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:26:55,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:26:55,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:26:55,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:26:55,931 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:26:55,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:26:55,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:26:55,986 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 06:26:56,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:26:56,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:56,456 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:26:56,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:57,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:57,507 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:26:57,523 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 06:26:57,525 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:26:57,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:57,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:26:57,618 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 06:26:57,619 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:26:57,621 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 06:26:57,711 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:57,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 06:26:58,013 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:58,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 06:26:58,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 06:26:58,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 06:26:58,053 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 06:26:58,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 06:26:58,084 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 06:26:58,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 06:26:58,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:26:58,128 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 06:26:58,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:26:58,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:58,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:26:58,204 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:58,219 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:58,232 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:26:58,246 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 06:26:58,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:26:58,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:58,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:26:58,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:58,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:26:58,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:58,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:58,387 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:58,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:58,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:58,454 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:58,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:58,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:58,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:58,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:26:58,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:58,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:58,586 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:58,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:58,619 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:58,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:58,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:58,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:26:58,683 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:58,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:26:58,716 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:58,732 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:58,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:58,760 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:58,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:58,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:58,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:58,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:26:58,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:58,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:26:58,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:58,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:58,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:58,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:58,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:58,943 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:58,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:58,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:26:58,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:59,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:26:59,017 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:59,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:59,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:59,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:59,081 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:59,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:59,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:59,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:26:59,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:59,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:26:59,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:59,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:59,214 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:59,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:59,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:59,270 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:59,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:59,303 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:26:59,319 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 06:26:59,322 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 06:26:59,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:26:59,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 06:26:59,680 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:26:59,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 06:26:59,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 06:26:59,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 06:26:59,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 06:26:59,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 06:26:59,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 06:26:59,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 06:26:59,783 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:26:59,799 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 06:26:59,829 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:26:59,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:59,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:26:59,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:59,885 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:26:59,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:26:59,911 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 06:26:59,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:26:59,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:26:59,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:26:59,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:26:59,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:27:00,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:27:00,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:27:00,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:27:00,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:27:00,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:27:00,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:27:00,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:27:00,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:27:00,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:00,239 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:27:00,257 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:00,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:00,293 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:00,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:00,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:00,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:00,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:00,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:00,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:00,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:27:00,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:00,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:00,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:00,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:00,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:00,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:00,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:00,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:00,599 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:00,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:27:00,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:00,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:00,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:00,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:00,693 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:00,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:00,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:00,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:00,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:00,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:27:00,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:00,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:00,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:00,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:00,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:00,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:00,885 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:00,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:00,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:00,941 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:27:00,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:00,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:00,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:01,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:01,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:01,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:01,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:01,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:01,099 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 06:27:01,104 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:27:01,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:27:01,413 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:27:01,486 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:27:01,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:27:01,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:27:01,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:27:01,530 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:27:01,548 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:27:01,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:27:01,573 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:27:01,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:27:01,604 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 06:27:01,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:27:01,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:27:02,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:27:02,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:27:02,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:27:03,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:27:03,187 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 06:27:03,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:27:03,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:27:03,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:27:03,262 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 06:27:03,263 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:27:03,266 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:27:03,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:27:03,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:27:03,638 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:27:03,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:27:03,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:27:03,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:27:03,681 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:27:03,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:27:03,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:27:03,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:27:03,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:27:03,762 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 06:27:03,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:27:03,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:27:04,247 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:27:04,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:27:04,928 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:27:05,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:27:05,206 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 06:27:05,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:27:05,224 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:27:05,272 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:27:05,301 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 06:27:05,302 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:27:05,305 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:27:05,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:27:06,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:27:06,367 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:27:06,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:27:06,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:27:06,413 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:27:06,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:27:06,459 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:27:06,471 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:27:06,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:27:06,507 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:27:06,522 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 06:27:06,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:27:06,566 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:27:06,943 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:27:07,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:27:07,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:27:07,760 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:27:07,775 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 06:27:07,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:27:07,788 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:27:07,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:27:07,838 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 06:27:07,839 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:27:07,842 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:27:07,928 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:27:08,198 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:27:08,275 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:27:08,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:27:08,295 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:27:08,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:27:08,329 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:27:08,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:27:08,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:27:08,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:27:08,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:27:08,420 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 06:27:08,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:27:08,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:27:08,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:27:09,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:27:09,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:27:10,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:27:10,265 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 06:27:10,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:27:10,295 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:27:10,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:27:10,427 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 06:27:10,428 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:27:10,430 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 06:27:10,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:27:10,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 06:27:10,792 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:27:10,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 06:27:10,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 06:27:10,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 06:27:10,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 06:27:10,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 06:27:10,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 06:27:10,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 06:27:10,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:27:10,927 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 06:27:10,956 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:27:10,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:27:10,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:27:11,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:27:11,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:27:11,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:27:11,049 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 06:27:11,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:27:11,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:27:11,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:27:11,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:27:11,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:27:11,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:27:11,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:27:11,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:27:11,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:27:11,242 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:27:11,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:27:11,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:27:11,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:27:11,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:11,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:27:11,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:11,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:11,405 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:11,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:11,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:11,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:11,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:11,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:27:11,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:11,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:27:11,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:11,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:11,566 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:11,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:11,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:11,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:11,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:11,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:27:11,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:11,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:27:11,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:11,716 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:11,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:11,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:11,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:11,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:11,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:11,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:27:11,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:11,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:27:11,854 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:11,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:11,890 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:11,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:11,920 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:11,936 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:11,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:11,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:27:11,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:12,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:27:12,017 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:12,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:12,058 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:12,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:12,091 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:12,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:12,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:12,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:27:12,151 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 06:27:12,151 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:27:12,154 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:27:12,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:27:12,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:27:12,489 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:27:12,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:27:12,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:27:12,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:27:12,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:27:12,543 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:27:12,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:27:12,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:27:12,584 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:13,006 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:27:13,006 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.855000019073. 2022-02-08 06:27:13,006 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:27:13,009 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:27:13,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:27:13,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:27:13,418 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:27:13,418 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:27:13,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:27:13,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:27:13,466 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:27:13,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:27:13,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:27:13,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:27:13,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:13,687 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:13,687 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:15,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:15,842 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:15,842 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:17,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:17,990 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:17,992 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:19,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:20,138 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:20,138 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:22,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:22,289 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:22,289 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:24,290 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:24,440 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:24,440 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:26,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:26,594 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:26,595 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:28,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:28,744 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:28,744 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:30,744 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:30,892 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:30,892 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:32,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:33,048 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:33,048 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:35,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:35,194 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:35,196 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:37,196 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:37,355 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:37,357 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:39,357 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:39,505 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:39,506 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:41,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:41,663 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:41,664 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:43,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:43,812 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:43,812 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:45,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:46,124 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:46,125 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:48,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:48,329 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:48,331 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 8751 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:27:50,331 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:50,519 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:50,519 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:52,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:52,868 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:52,868 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:54,869 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:55,354 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:55,354 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:57,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:57,523 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:57,523 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:27:59,525 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:27:59,697 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:27:59,698 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:28:01,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:28:01,911 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:28:01,911 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:28:03,913 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:28:04,121 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:28:04,122 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:28:06,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:28:06,292 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:28:06,293 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:28:08,293 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:28:08,448 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:28:08,450 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:28:10,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:28:10,591 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:28:10,591 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:28:12,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:28:12,756 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:28:12,756 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 59.75. 2022-02-08 06:28:12,756 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:28:12,759 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:28:14,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:28:14,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:28:14,976 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:28:14,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:28:14,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:28:15,038 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:28:15,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:28:15,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:28:15,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:28:15,196 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:28:15,214 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:28:16,042 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:28:16,042 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.28500008583. 2022-02-08 06:28:16,042 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:28:16,128 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:28:16,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:28:17,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:28:17,556 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:28:17,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:28:17,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:28:17,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:28:17,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:28:17,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:28:17,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:28:17,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:28:17,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:28:18,302 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:28:18,302 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.25900006294. 2022-02-08 06:28:18,302 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:28:18,305 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:28:18,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:28:19,150 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:28:19,301 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:28:19,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:28:19,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:28:19,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:28:19,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:28:19,387 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:28:19,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:28:19,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:28:19,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:28:19,740 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:28:19,740 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.43799996376. 2022-02-08 06:28:19,740 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:28:19,743 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:28:20,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:28:20,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:28:21,012 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:28:21,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:28:21,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:28:21,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:28:21,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:28:21,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:28:21,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:28:21,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:28:21,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:28:21,585 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:28:21,585 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.84500002861. 2022-02-08 06:28:21,585 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:28:21,588 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:28:22,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:28:22,548 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:28:22,634 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:28:22,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:28:22,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:28:22,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:28:22,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:28:22,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:28:22,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:28:22,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:28:22,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:28:23,098 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:28:23,098 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.51300001144. 2022-02-08 06:28:23,098 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:28:23,101 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:28:23,928 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:28:24,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:28:24,444 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:28:24,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:28:24,459 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:28:24,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:28:24,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:28:24,505 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:28:24,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:28:24,543 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:28:24,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:28:25,003 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:28:25,003 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.90499997139. 2022-02-08 06:28:25,003 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:28:25,006 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:28:26,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:28:26,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:28:26,592 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:28:26,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:28:26,607 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:28:26,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:28:26,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:28:26,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:28:26,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:28:26,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:28:26,706 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:28:27,599 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:28:27,599 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.59599995613. 2022-02-08 06:28:27,599 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:28:27,601 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:28:28,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:28:28,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:28:28,851 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:28:28,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:28:28,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:28:28,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:28:29,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:28:29,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:28:29,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:28:29,196 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:28:29,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:28:29,980 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:28:29,980 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.38100004196. ############################################################ # # RAM FootPrint: 4389.95703125 # ############################################################ 2022-02-08 06:28:30,013 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 06:28:30,013 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 80 seconds. Reason: Iteration:4 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 8759 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 8785 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:29:50,019 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 06:29:50,019 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 06:29:55,022 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:29:55,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:29:55,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:29:55,533 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:29:55,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:29:55,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:29:55,566 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:29:55,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:29:55,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:29:55,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:29:55,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:29:55,686 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:29:55,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:29:55,767 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['12420 memcached\n'], []) 2022-02-08 06:29:55,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 06:29:55,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:29:55,825 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['12420 memcached\n'], []) 2022-02-08 06:29:55,825 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:29:55,828 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:29:56,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:29:56,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:29:56,767 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:29:56,767 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:29:56,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:29:56,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:29:56,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:29:56,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:29:56,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:29:56,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:29:56,901 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:29:56,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:29:56,959 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['58807 memcached\n'], []) 2022-02-08 06:29:56,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 06:29:56,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:29:57,013 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['58807 memcached\n'], []) 2022-02-08 06:29:57,013 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:29:57,016 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:29:57,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:29:57,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:29:57,385 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:29:57,387 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:29:57,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:29:57,418 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:29:57,441 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:29:57,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:29:57,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:29:57,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:29:57,509 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:29:57,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:29:57,555 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['44828 memcached\n'], []) 2022-02-08 06:29:57,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 06:29:57,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:29:57,605 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['44828 memcached\n'], []) 2022-02-08 06:29:57,605 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:29:57,608 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:29:57,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:29:57,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:29:57,957 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:29:57,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:29:57,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:29:57,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:29:58,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:29:58,017 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:29:58,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:29:58,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:29:58,055 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:29:58,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:29:58,099 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['76980 memcached\n'], []) 2022-02-08 06:29:58,099 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 06:29:58,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:29:58,151 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['76980 memcached\n'], []) 2022-02-08 06:29:58,151 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:29:58,153 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:29:58,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:29:58,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:29:58,513 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:29:58,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:29:58,528 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:29:58,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:29:58,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:29:58,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:29:58,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:29:58,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:29:58,608 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:29:58,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:29:58,657 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['70859 memcached\n'], []) 2022-02-08 06:29:58,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 06:29:58,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:29:58,720 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['70859 memcached\n'], []) 2022-02-08 06:29:58,720 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:29:58,729 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:29:58,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:29:59,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:29:59,078 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:29:59,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:29:59,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:29:59,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:29:59,117 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:29:59,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:29:59,147 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:29:59,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:29:59,176 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:29:59,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:29:59,223 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['105838 memcached\n'], []) 2022-02-08 06:29:59,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 06:29:59,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:29:59,279 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['105838 memcached\n'], []) 2022-02-08 06:29:59,279 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:29:59,282 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:29:59,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:29:59,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:29:59,653 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:29:59,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:29:59,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:29:59,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:29:59,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:29:59,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:29:59,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:29:59,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:29:59,752 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:29:59,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:29:59,796 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['39631 memcached\n'], []) 2022-02-08 06:29:59,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 06:29:59,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:29:59,841 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['39631 memcached\n'], []) 2022-02-08 06:29:59,841 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:29:59,844 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:29:59,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:30:00,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:30:00,210 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:30:00,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:30:00,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:30:00,234 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:30:00,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:30:00,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:30:00,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:30:00,293 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:30:00,338 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:30:00,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:30:00,493 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['54539 memcached\n'], []) 2022-02-08 06:30:00,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 06:30:00,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:30:00,542 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['54539 memcached\n'], []) 2022-02-08 06:30:00,542 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:30:00,545 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:30:00,647 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:30:00,849 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:30:00,921 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:30:00,923 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:30:00,936 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:30:00,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:30:00,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:30:00,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:30:00,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:30:00,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:30:01,013 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:30:01,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:30:01,055 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['103926 memcached\n'], []) 2022-02-08 06:30:01,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 06:30:01,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:30:01,115 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['103926 memcached\n'], []) 2022-02-08 06:30:01,115 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:30:01,118 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:30:01,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:30:01,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:30:01,486 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:30:01,487 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:30:01,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:30:01,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:30:01,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:30:01,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:30:01,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:30:01,579 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:30:01,594 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:30:01,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:30:01,674 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['120624 memcached\n'], []) 2022-02-08 06:30:01,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 06:30:01,706 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:30:01,729 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['120624 memcached\n'], []) 2022-02-08 06:30:01,729 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:30:01,730 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:30:01,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:30:02,035 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:30:02,101 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:30:02,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:30:02,117 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:30:02,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:30:02,145 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:30:02,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:30:02,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:30:02,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:30:02,204 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:30:02,220 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 06:30:02,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:30:02,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:05,859 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:30:06,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:06,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:07,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:30:07,046 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 06:30:07,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:30:07,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:07,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:30:07,134 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 06:30:07,134 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:30:07,135 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:30:07,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:30:07,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:30:07,500 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:30:07,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:30:07,528 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:30:07,563 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:30:07,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:30:07,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:30:07,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:30:07,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:30:07,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:30:07,660 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 06:30:07,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:30:07,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:10,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:30:11,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:11,829 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:12,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:30:12,141 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 06:30:12,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:30:12,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:12,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:30:12,240 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 06:30:12,240 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:30:12,243 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:30:12,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:30:12,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:30:12,632 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:30:12,632 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:30:12,655 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:30:12,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:30:12,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:30:12,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:30:12,706 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:30:12,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:30:12,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:30:12,749 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 06:30:12,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:30:12,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:16,272 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:30:16,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:16,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:17,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:30:17,167 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 06:30:17,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:30:17,183 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:17,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:30:17,263 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 06:30:17,263 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:30:17,266 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 06:30:17,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:30:17,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 06:30:17,661 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:30:17,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 06:30:17,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 06:30:17,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 06:30:17,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 06:30:17,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 06:30:17,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 06:30:17,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 06:30:17,767 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:30:17,783 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 06:30:17,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:30:17,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:17,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:30:17,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:17,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:17,885 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:30:17,900 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 06:30:17,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:30:17,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:17,931 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:30:17,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:17,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:30:18,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:18,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:18,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:18,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:18,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:18,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:18,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:18,191 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:18,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:18,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:30:18,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:18,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:18,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:18,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:18,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:18,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:18,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:18,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:18,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:18,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:30:18,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:18,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:18,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:18,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:18,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:18,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:18,545 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:18,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:18,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:18,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:30:18,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:18,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:18,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:18,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:18,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:18,693 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:18,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:18,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:18,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:18,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:30:18,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:18,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:18,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:18,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:18,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:18,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:18,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:18,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:18,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:18,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:30:18,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:18,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:19,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:19,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:19,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:19,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:19,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:19,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:19,101 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 06:30:19,105 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:30:19,211 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:30:19,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:30:19,505 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:30:19,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:30:19,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:30:19,535 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:30:19,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:30:19,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:30:19,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:30:19,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:30:19,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:30:19,634 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 06:30:19,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:30:19,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:20,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:30:20,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:20,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:20,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:30:20,846 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 06:30:20,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:30:20,862 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:20,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:30:20,934 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 06:30:20,934 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:30:20,937 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 06:30:21,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:30:21,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 06:30:21,319 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:30:21,321 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 06:30:21,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 06:30:21,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 06:30:21,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 06:30:21,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 06:30:21,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 06:30:21,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 06:30:21,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:30:21,434 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 06:30:21,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:30:21,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:21,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:30:21,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:21,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:21,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:30:21,581 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 06:30:21,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:30:21,596 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:21,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:30:21,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:21,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:30:21,683 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:21,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:21,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:21,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:21,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:21,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:21,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:21,869 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:21,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:21,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:30:21,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:21,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:21,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:21,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:21,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:22,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:22,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:22,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:22,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:22,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:30:22,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:22,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:22,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:22,178 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:22,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:22,216 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:22,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:22,249 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:22,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:22,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:30:22,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:22,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:22,331 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:22,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:22,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:22,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:22,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:22,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:22,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:22,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:30:22,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:22,487 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:22,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:22,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:22,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:22,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:22,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:22,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:22,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:22,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:30:22,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:22,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:22,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:22,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:22,721 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:22,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:22,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:22,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:22,789 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 06:30:22,792 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:30:22,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:30:23,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:30:23,158 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:30:23,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:30:23,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:30:23,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:30:23,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:30:23,216 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:30:23,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:30:23,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:30:23,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:30:23,285 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 06:30:23,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:30:23,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:27,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:30:28,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:28,614 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:28,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:30:28,951 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 06:30:28,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:30:28,964 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:29,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:30:29,020 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 06:30:29,020 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:30:29,025 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:30:29,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:30:29,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:30:29,460 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:30:29,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:30:29,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:30:29,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:30:29,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:30:29,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:30:29,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:30:29,548 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:30:29,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:30:29,579 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 06:30:29,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:30:29,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:32,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:30:33,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:33,749 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:34,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:30:34,065 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 06:30:34,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:30:34,084 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:34,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:30:34,158 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 06:30:34,160 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:30:34,163 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 06:30:34,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:30:34,505 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 06:30:34,578 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:30:34,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 06:30:34,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 06:30:34,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 06:30:34,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 06:30:34,645 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 06:30:34,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 06:30:34,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 06:30:34,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:30:34,710 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 06:30:34,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:30:34,755 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:34,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:30:34,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:34,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:34,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:30:34,835 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 06:30:34,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:30:34,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:34,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:30:34,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:34,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:30:34,936 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:34,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:34,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:35,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:35,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:35,073 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:35,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:35,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:35,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:35,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:30:35,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:35,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:35,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:35,257 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:35,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:35,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:35,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:35,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:35,357 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:35,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:30:35,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:35,413 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:35,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:35,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:35,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:35,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:35,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:35,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:35,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:35,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:30:35,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:35,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:35,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:35,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:35,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:35,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:35,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:35,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:35,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:35,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:30:35,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:35,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:35,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:35,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:35,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:35,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:35,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:35,979 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:35,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:36,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:30:36,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:36,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:36,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:36,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:36,127 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:36,147 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:36,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:36,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:36,206 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 06:30:36,209 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 06:30:36,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:30:36,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 06:30:36,621 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:30:36,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 06:30:36,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 06:30:36,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 06:30:36,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 06:30:36,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 06:30:36,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 06:30:36,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 06:30:36,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:30:36,753 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 06:30:36,783 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:30:36,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:36,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:30:36,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:36,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:36,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:30:36,871 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 06:30:36,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:30:36,885 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:36,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:30:36,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:36,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:30:36,969 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:36,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:37,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:37,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:37,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:37,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:37,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:37,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:30:37,198 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:37,219 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:30:37,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:37,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:37,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:37,295 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:37,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:37,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:37,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:37,362 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:30:37,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:37,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:30:37,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:37,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:37,446 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:37,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:37,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:37,497 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:37,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:37,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:30:37,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:37,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:30:37,579 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:37,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:37,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:37,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:37,650 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:37,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:37,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:37,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:30:37,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:37,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:30:37,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:37,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:37,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:37,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:37,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:37,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:37,857 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:37,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:30:37,890 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:37,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:30:37,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:37,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:37,964 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:37,982 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:37,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:38,017 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:38,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:38,056 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:30:38,075 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 06:30:38,078 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:30:38,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:30:38,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:30:38,618 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:30:38,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:30:38,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:30:38,655 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:30:38,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:30:38,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:30:38,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:30:38,732 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:30:38,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:30:38,788 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 06:30:38,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:30:38,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:42,043 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:30:42,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:43,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:43,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:30:43,933 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 06:30:43,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:30:44,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:44,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:30:44,331 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 06:30:44,331 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:30:44,334 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:30:44,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:30:44,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:30:44,859 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:30:44,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:30:44,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:30:44,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:30:44,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:30:44,956 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:30:44,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:30:44,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:30:45,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:30:45,036 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 06:30:45,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:30:45,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:48,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:30:49,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 8797 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:30:49,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:49,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:30:49,867 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 06:30:49,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:30:49,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:49,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:30:49,957 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 06:30:49,957 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:30:49,960 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:30:50,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:30:50,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:30:50,934 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:30:50,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:30:50,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:30:50,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:30:50,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:30:51,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:30:51,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:30:51,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:30:51,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:30:51,101 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 06:30:51,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:30:51,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:52,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:30:52,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:53,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:30:53,392 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:30:53,408 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 06:30:53,408 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:30:53,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:53,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:30:53,489 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 06:30:53,489 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:30:53,493 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:30:53,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:30:53,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:30:53,967 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:30:53,969 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:30:53,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:30:53,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:30:54,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:30:54,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:30:54,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:30:54,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:30:54,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:30:54,131 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 06:30:54,170 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:30:54,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:30:59,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:31:00,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:31:00,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:31:01,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:31:01,651 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 06:31:01,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:31:01,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:31:01,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:31:01,980 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 06:31:01,980 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:31:01,983 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 06:31:02,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:31:02,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 06:31:02,404 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:31:02,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 06:31:02,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 06:31:02,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 06:31:02,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 06:31:02,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 06:31:02,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 06:31:02,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 06:31:02,507 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:31:02,523 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 06:31:02,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:31:02,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:31:02,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:31:02,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:31:02,688 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:31:02,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:31:02,717 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 06:31:02,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:31:02,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:31:02,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:31:02,760 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:31:02,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:31:02,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:31:02,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:31:02,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:31:02,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:31:02,920 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:31:02,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:31:02,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:31:03,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:31:03,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:31:03,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:31:03,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:31:03,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:31:03,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:31:03,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:31:03,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:31:03,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:31:03,160 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:31:03,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:31:03,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:31:03,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:31:03,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:31:03,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:31:03,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:31:03,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:31:03,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:31:03,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:31:03,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:31:03,349 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:31:03,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:31:03,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:31:03,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:31:03,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:31:03,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:31:03,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:31:03,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:31:03,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:31:03,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:31:03,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:31:03,525 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:31:03,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:31:03,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:31:03,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:31:03,596 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:31:03,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:31:03,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:31:03,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:31:03,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:31:03,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:31:03,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:31:03,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:31:03,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:31:03,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:31:03,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:31:03,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:31:03,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:31:03,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:31:03,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:31:03,859 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:31:03,875 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 06:31:03,875 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:31:03,878 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:31:03,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:31:04,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:31:04,236 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:31:04,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:31:04,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:31:04,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:31:04,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:31:04,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:31:04,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:31:04,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:31:04,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:04,868 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:31:04,868 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.993000030518. 2022-02-08 06:31:04,868 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:31:04,871 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:31:05,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:31:05,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:31:06,308 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:31:06,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:31:06,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:31:06,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:31:06,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:31:06,727 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:31:06,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:31:06,829 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:31:06,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:07,091 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:31:07,092 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:31:09,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:09,292 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:31:09,292 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:31:11,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:11,477 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:31:11,479 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:31:13,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:13,686 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:31:13,687 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:31:15,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:15,854 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:31:15,855 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:31:17,862 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:18,029 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:31:18,029 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:31:20,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:20,184 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:31:20,186 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:31:22,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:22,344 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:31:22,345 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:31:24,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:24,493 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:31:24,493 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:31:26,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:26,641 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:31:26,641 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:31:28,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:28,786 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:31:28,786 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:31:30,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:30,930 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:31:30,930 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:31:32,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:33,073 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:31:33,075 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:31:35,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:35,217 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:31:35,217 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 30.3489999771. 2022-02-08 06:31:35,217 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:31:35,220 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:31:35,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:31:36,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:31:36,257 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:31:36,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:31:36,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:31:36,290 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:31:36,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:31:36,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:31:36,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:31:36,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:31:36,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:36,713 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:31:36,713 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.49500012398. 2022-02-08 06:31:36,713 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:31:36,716 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:31:37,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:31:37,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:31:37,960 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:31:37,961 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:31:37,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:31:37,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:31:38,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:31:38,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:31:38,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:31:38,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:31:38,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:38,848 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:31:38,848 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.13400006294. 2022-02-08 06:31:38,848 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:31:38,849 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:31:39,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:31:39,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:31:39,815 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:31:39,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:31:39,829 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:31:39,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:31:39,859 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:31:39,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:31:39,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:31:39,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:31:39,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:40,875 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:31:40,877 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.02900004387. 2022-02-08 06:31:40,877 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:31:40,878 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:31:41,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:31:41,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:31:41,849 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:31:41,849 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:31:41,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:31:41,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:31:41,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:31:41,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:31:41,931 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:31:41,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:31:41,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:42,944 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:31:42,944 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.06699991226. 2022-02-08 06:31:42,944 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:31:42,947 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:31:43,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:31:43,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:31:43,910 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:31:43,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:31:43,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:31:43,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:31:43,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:31:43,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:31:43,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:31:44,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:31:44,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:44,482 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:31:44,483 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.53800010681. 2022-02-08 06:31:44,483 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:31:44,484 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:31:44,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:31:44,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:31:44,865 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:31:44,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:31:44,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:31:44,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:31:44,913 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:31:44,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:31:44,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:31:44,979 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:31:44,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:45,946 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:31:45,947 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.46300005913. 2022-02-08 06:31:45,947 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:31:45,948 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:31:46,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:31:46,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:31:46,951 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:31:46,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:31:46,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:31:47,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:31:47,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:31:47,035 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:31:47,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:31:47,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:31:47,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:48,246 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:31:48,247 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.30099987984. 2022-02-08 06:31:48,247 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:31:48,250 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:31:48,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:31:49,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 5 queries submitted, 0 failed, 8803 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:31:49,480 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:31:49,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:31:49,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:31:49,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:31:49,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:31:49,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:31:49,563 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:31:49,586 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:31:49,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:31:50,371 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:31:50,371 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.12400007248. ############################################################ # # RAM FootPrint: 4385.4140625 # ############################################################ 2022-02-08 06:31:50,404 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 06:31:50,404 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 89 seconds. Reason: Iteration:5 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 8826 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:33:19,413 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 06:33:19,414 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 06:33:24,417 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:33:24,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:24,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:33:24,871 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:24,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:33:24,890 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:33:24,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:33:24,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:33:24,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:33:24,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:33:24,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:33:24,993 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:33:24,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:33:25,065 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['13117 memcached\n'], []) 2022-02-08 06:33:25,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 06:33:25,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:33:25,140 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['13117 memcached\n'], []) 2022-02-08 06:33:25,140 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:33:25,144 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:33:25,778 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:26,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:33:26,142 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:26,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:33:26,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:33:26,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:33:26,186 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:33:26,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:33:26,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:33:26,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:33:26,256 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:33:26,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:33:26,318 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['59798 memcached\n'], []) 2022-02-08 06:33:26,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 06:33:26,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:33:26,367 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['59798 memcached\n'], []) 2022-02-08 06:33:26,368 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:33:26,371 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:33:26,448 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:26,670 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:33:26,740 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:26,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:33:26,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:33:26,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:33:26,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:33:26,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:33:26,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:33:26,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:33:26,849 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:33:26,849 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:33:26,898 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['45590 memcached\n'], []) 2022-02-08 06:33:26,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 06:33:26,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:33:26,947 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['45590 memcached\n'], []) 2022-02-08 06:33:26,948 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:33:26,950 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:33:27,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:27,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:33:27,357 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:27,357 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:33:27,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:33:27,385 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:33:27,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:33:27,415 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:33:27,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:33:27,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:33:27,457 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:33:27,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:33:27,505 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['77677 memcached\n'], []) 2022-02-08 06:33:27,505 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 06:33:27,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:33:27,553 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['77677 memcached\n'], []) 2022-02-08 06:33:27,553 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:33:27,558 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:33:27,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:27,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:33:27,898 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:27,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:33:27,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:33:27,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:33:27,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:33:27,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:33:27,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:33:27,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:33:28,005 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:33:28,005 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:33:28,052 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['71526 memcached\n'], []) 2022-02-08 06:33:28,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 06:33:28,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:33:28,099 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['71526 memcached\n'], []) 2022-02-08 06:33:28,099 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:33:28,101 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:33:28,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:28,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:33:28,473 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:28,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:33:28,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:33:28,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:33:28,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:33:28,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:33:28,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:33:28,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:33:28,565 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:33:28,566 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:33:28,611 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['106506 memcached\n'], []) 2022-02-08 06:33:28,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 06:33:28,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:33:28,664 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['106506 memcached\n'], []) 2022-02-08 06:33:28,664 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:33:28,665 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:33:28,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:28,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:33:29,026 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:29,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:33:29,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:33:29,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:33:29,068 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:33:29,084 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:33:29,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:33:29,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:33:29,124 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:33:29,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:33:29,171 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['40331 memcached\n'], []) 2022-02-08 06:33:29,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 06:33:29,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:33:29,220 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['40331 memcached\n'], []) 2022-02-08 06:33:29,221 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:33:29,223 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:33:29,910 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:30,165 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:33:30,266 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:30,267 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:33:30,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:33:30,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:33:30,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:33:30,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:33:30,341 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:33:30,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:33:30,367 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:33:30,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:33:30,431 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['55239 memcached\n'], []) 2022-02-08 06:33:30,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 06:33:30,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:33:30,479 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['55239 memcached\n'], []) 2022-02-08 06:33:30,479 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:33:30,489 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:33:30,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:30,816 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:33:30,882 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:30,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:33:30,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:33:30,913 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:33:30,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:33:30,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:33:30,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:33:30,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:33:30,993 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:33:30,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:33:31,049 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['104629 memcached\n'], []) 2022-02-08 06:33:31,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 06:33:31,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:33:31,098 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['104629 memcached\n'], []) 2022-02-08 06:33:31,098 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:33:31,105 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:33:31,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:31,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:33:31,469 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:31,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:33:31,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:33:31,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:33:31,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:33:31,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:33:31,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:33:31,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:33:31,565 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:33:31,566 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:33:31,615 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['121284 memcached\n'], []) 2022-02-08 06:33:31,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 06:33:31,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:33:31,664 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['121284 memcached\n'], []) 2022-02-08 06:33:31,664 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:33:31,667 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:33:31,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:31,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:33:32,051 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:32,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:33:32,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:33:32,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:33:32,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:33:32,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:33:32,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:33:32,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:33:32,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:33:32,163 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 06:33:32,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:33:32,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:33,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:33:34,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:34,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:35,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:33:35,026 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 06:33:35,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:33:35,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:35,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:33:35,112 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 06:33:35,114 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:33:35,117 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:33:35,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:35,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:33:35,467 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:35,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:33:35,482 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:33:35,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:33:35,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:33:35,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:33:35,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:33:35,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:33:35,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:33:35,588 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 06:33:35,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:33:35,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:36,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:33:36,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:37,127 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:37,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:33:37,430 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 06:33:37,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:33:37,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:37,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:33:37,523 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 06:33:37,525 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:33:37,528 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:33:37,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:37,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:33:37,891 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:37,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:33:37,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:33:37,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:33:37,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:33:37,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:33:37,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:33:37,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:33:38,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:33:38,028 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 06:33:38,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:33:38,073 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:38,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:33:38,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:39,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:39,285 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:33:39,299 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 06:33:39,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:33:39,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:39,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:33:39,385 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 06:33:39,387 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:33:39,388 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 06:33:39,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:39,727 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 06:33:39,796 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:39,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 06:33:39,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 06:33:39,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 06:33:39,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 06:33:39,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 06:33:39,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 06:33:39,885 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 06:33:39,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:33:39,917 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 06:33:39,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:33:39,961 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:39,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:33:39,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:40,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:40,022 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:33:40,036 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 06:33:40,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:33:40,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:40,068 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:33:40,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:40,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:33:40,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:40,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:40,198 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:40,224 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:40,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:40,283 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:40,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:40,341 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:40,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:40,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:33:40,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:40,423 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:40,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:40,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:40,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:40,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:40,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:40,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:40,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:40,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:33:40,586 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:40,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:40,624 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:40,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:40,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:40,693 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:40,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:40,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:40,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:40,770 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:33:40,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:40,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:40,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:40,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:40,857 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:40,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:40,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:40,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:40,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:40,941 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:33:40,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:40,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:40,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:41,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:41,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:41,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:41,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:41,073 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:41,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:41,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:33:41,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:41,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:41,170 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:41,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:41,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:41,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:41,239 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:41,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:41,276 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 06:33:41,278 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:33:41,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:41,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:33:42,595 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:42,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:33:42,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:33:42,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:33:42,877 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:33:42,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:33:42,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:33:42,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:33:42,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:33:43,009 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 06:33:43,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:33:43,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:44,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:33:45,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:45,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:45,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:33:45,533 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 06:33:45,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:33:45,596 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:45,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:33:45,736 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 06:33:45,736 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:33:45,740 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 06:33:45,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:46,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 06:33:46,119 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:46,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 06:33:46,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 06:33:46,147 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 06:33:46,160 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 06:33:46,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 06:33:46,191 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 06:33:46,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 06:33:46,221 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:33:46,236 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 06:33:46,267 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:33:46,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:46,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:33:46,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:46,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:46,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:33:46,351 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 06:33:46,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:33:46,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:46,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:33:46,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:46,428 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:33:46,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:46,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:46,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:46,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:46,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:46,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:46,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:46,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:46,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:46,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:33:46,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:46,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:46,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:46,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:46,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:46,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:46,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:46,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:46,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:46,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:33:46,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:46,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:46,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:46,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:46,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:46,964 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:46,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:47,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:47,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:47,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:33:47,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:47,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:47,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:47,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:47,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:47,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:47,155 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:47,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:47,193 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:47,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:33:47,229 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:47,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:47,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:47,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:47,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:47,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:47,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:47,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:47,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:47,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:33:47,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:47,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:47,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:47,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:47,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:47,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:47,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:47,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:47,536 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 06:33:47,539 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:33:47,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:47,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:33:47,897 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:47,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:33:47,913 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:33:47,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:33:47,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:33:47,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:33:47,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:33:47,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:33:47,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:33:48,009 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 06:33:48,053 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:33:48,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:48,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:33:49,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 8848 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:33:49,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:49,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:33:49,819 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 06:33:49,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:33:49,834 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:49,877 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:33:49,890 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 06:33:49,890 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:33:49,891 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:33:49,961 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:50,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:33:50,249 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:50,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:33:50,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:33:50,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:33:50,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:33:50,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:33:50,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:33:50,339 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:33:50,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:33:50,372 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 06:33:50,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:33:50,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:50,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:33:51,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:51,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:51,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:33:51,992 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 06:33:51,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:33:52,015 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:52,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:33:52,118 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 06:33:52,119 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:33:52,122 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 06:33:52,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:52,428 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 06:33:52,496 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:52,497 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 06:33:52,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 06:33:52,528 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 06:33:52,545 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 06:33:52,563 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 06:33:52,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 06:33:52,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 06:33:52,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:33:52,624 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 06:33:52,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:33:52,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:52,683 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:33:52,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:52,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:52,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:33:52,737 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 06:33:52,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:33:52,752 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:52,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:33:52,780 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:52,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:33:52,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:52,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:52,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:52,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:52,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:52,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:52,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:53,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:53,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:53,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:33:53,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:53,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:53,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:53,142 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:53,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:53,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:53,191 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:53,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:53,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:53,239 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:33:53,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:53,272 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:53,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:53,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:53,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:53,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:53,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:53,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:53,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:53,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:33:53,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:53,428 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:53,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:53,459 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:53,482 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:53,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:53,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:53,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:53,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:53,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:33:53,584 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:53,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:53,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:53,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:53,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:53,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:53,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:53,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:53,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:53,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:33:53,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:53,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:53,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:53,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:53,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:53,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:53,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:53,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:53,885 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 06:33:53,890 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 06:33:53,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:54,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 06:33:54,289 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:54,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 06:33:54,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 06:33:54,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 06:33:54,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 06:33:54,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 06:33:54,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 06:33:54,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 06:33:54,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:33:54,415 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 06:33:54,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:33:54,466 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:54,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:33:54,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:54,512 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:54,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:33:54,539 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 06:33:54,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:33:54,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:54,571 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:33:54,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:54,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:33:54,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:54,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:54,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:54,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:54,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:54,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:54,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:54,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:33:54,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:54,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:33:54,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:54,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:54,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:54,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:54,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:55,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:55,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:55,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:33:55,081 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:55,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:33:55,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:55,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:55,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:55,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:55,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:55,196 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:55,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:55,234 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:33:55,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:55,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:33:55,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:55,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:55,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:55,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:55,341 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:55,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:55,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:55,392 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:33:55,408 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:55,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:33:55,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:55,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:55,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:55,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:55,512 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:55,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:55,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:55,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:33:55,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:55,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:33:55,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:55,632 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:55,650 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:55,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:55,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:55,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:55,721 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:55,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:33:55,756 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 06:33:55,757 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:33:55,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:56,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:33:56,190 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:56,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:33:56,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:33:56,229 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:33:56,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:33:56,272 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:33:56,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:33:56,306 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:33:56,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:33:56,346 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 06:33:56,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:33:56,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:56,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:33:57,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:57,706 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:57,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:33:57,990 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 06:33:57,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:33:58,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:58,056 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:33:58,072 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 06:33:58,072 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:33:58,075 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:33:58,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:33:58,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:33:58,466 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:33:58,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:33:58,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:33:58,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:33:58,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:33:58,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:33:58,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:33:58,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:33:58,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:33:58,612 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 06:33:58,642 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:33:58,660 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:33:59,109 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:33:59,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:33:59,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:34:00,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:34:00,091 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 06:34:00,091 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:34:00,109 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:34:00,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:34:00,154 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 06:34:00,155 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:34:00,158 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:34:00,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:34:01,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:34:01,437 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:34:01,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:34:01,454 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:34:01,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:34:01,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:34:01,525 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:34:01,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:34:01,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:34:01,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:34:01,604 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 06:34:01,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:34:01,665 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:34:04,474 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:34:04,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:34:05,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:34:05,415 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:34:05,430 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 06:34:05,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:34:05,446 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:34:05,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:34:05,512 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 06:34:05,513 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:34:05,515 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:34:05,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:34:05,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:34:06,006 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:34:06,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:34:06,025 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:34:06,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:34:06,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:34:06,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:34:06,316 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:34:06,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:34:06,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:34:06,549 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 06:34:06,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:34:06,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:34:07,392 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:34:07,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:34:08,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:34:08,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:34:08,737 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 06:34:08,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:34:08,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:34:08,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:34:08,825 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 06:34:08,826 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:34:08,829 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 06:34:08,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:34:09,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 06:34:09,219 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:34:09,219 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 06:34:09,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 06:34:09,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 06:34:09,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 06:34:09,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 06:34:09,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 06:34:09,306 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 06:34:09,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:34:09,338 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 06:34:09,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:34:09,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:34:09,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:34:09,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:34:09,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:34:09,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:34:09,463 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 06:34:09,464 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:34:09,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:34:09,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:34:09,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:34:09,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:34:09,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:34:09,596 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:34:09,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:34:09,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:34:09,688 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:34:09,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:34:09,749 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:34:09,780 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:34:09,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:34:09,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:34:09,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:34:09,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:34:09,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:34:09,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:34:09,918 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:34:09,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:34:09,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:34:09,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:34:09,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:34:10,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:34:10,028 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:34:10,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:34:10,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:34:10,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:34:10,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:34:10,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:34:10,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:34:10,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:34:10,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:34:10,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:34:10,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:34:10,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:34:10,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:34:10,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:34:10,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:34:10,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:34:10,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:34:10,349 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:34:10,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:34:10,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:34:10,413 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:34:10,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:34:10,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:34:10,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:34:10,484 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:34:10,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:34:10,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:34:10,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:34:10,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:34:10,573 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:34:10,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:34:10,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:34:10,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:34:10,650 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:34:10,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:34:10,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:34:10,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:34:10,727 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:34:10,746 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 06:34:10,746 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:34:10,749 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:34:10,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:34:11,053 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:34:11,128 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:34:11,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:34:11,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:34:11,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:34:11,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:34:11,211 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:34:11,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:34:11,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:34:11,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:11,773 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:34:11,773 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.02699995041. 2022-02-08 06:34:11,773 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:34:11,776 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:34:11,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:34:12,607 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:34:13,121 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:34:13,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:34:13,247 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:34:13,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:34:13,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:34:13,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:34:13,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:34:13,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:34:13,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:13,871 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:13,871 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:15,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:16,084 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:16,084 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:18,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:18,358 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:18,358 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:20,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:20,572 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:20,572 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:22,573 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:22,927 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:22,927 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:24,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:25,266 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:25,266 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:27,267 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:27,576 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:27,578 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:29,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:29,782 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:29,782 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:31,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:31,953 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:31,954 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:33,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:34,108 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:34,109 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:36,109 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:36,289 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:36,289 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:38,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:38,451 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:38,453 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:40,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:40,663 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:40,663 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:42,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:42,983 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:42,983 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:44,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:45,131 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:45,131 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:47,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:47,292 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:47,292 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:49,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:49,440 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:49,440 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 8861 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:34:51,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:51,598 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:51,598 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:53,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:53,752 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:53,753 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:55,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:55,903 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:55,904 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:34:57,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:34:58,049 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:34:58,049 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:35:00,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:35:00,201 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:35:00,203 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:35:02,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:35:02,345 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:35:02,346 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 50.5720000267. 2022-02-08 06:35:02,346 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:35:02,348 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:35:03,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:35:03,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:35:03,447 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:35:03,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:35:03,464 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:35:03,482 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:35:03,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:35:03,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:35:03,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:35:03,566 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:35:03,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:35:04,216 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:35:04,216 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.86900019646. 2022-02-08 06:35:04,216 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:35:04,217 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:35:05,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:35:05,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:35:05,786 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:35:05,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:35:05,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:35:05,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:35:05,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:35:05,854 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:35:05,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:35:05,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:35:05,910 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:35:06,493 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:35:06,493 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.27699995041. 2022-02-08 06:35:06,493 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:35:06,496 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:35:07,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:35:07,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:35:07,617 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:35:07,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:35:07,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:35:07,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:35:07,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:35:07,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:35:07,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:35:07,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:35:07,744 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:35:08,223 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:35:08,224 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.73099994659. 2022-02-08 06:35:08,224 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:35:08,226 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:35:09,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:35:09,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:35:09,714 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:35:09,714 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:35:09,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:35:09,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:35:09,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:35:09,788 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:35:09,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:35:09,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:35:09,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:35:10,421 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:35:10,421 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.1970000267. 2022-02-08 06:35:10,423 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:35:10,424 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:35:11,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:35:11,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:35:11,684 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:35:11,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:35:11,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:35:11,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:35:11,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:35:11,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:35:11,770 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:35:11,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:35:11,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:35:12,318 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:35:12,318 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.89499998093. 2022-02-08 06:35:12,319 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:35:12,319 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:35:12,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:35:12,714 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:35:12,809 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:35:12,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:35:12,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:35:12,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:35:12,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:35:12,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:35:12,913 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:35:12,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:35:12,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:35:13,448 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:35:13,450 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.13100004196. 2022-02-08 06:35:13,450 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:35:13,453 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:35:13,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:35:14,224 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:35:14,305 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:35:14,306 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:35:14,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:35:14,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:35:14,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:35:14,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:35:14,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:35:14,408 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:35:14,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:35:14,931 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:35:14,933 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.48300004005. 2022-02-08 06:35:14,933 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:35:14,934 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:35:15,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:35:15,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:35:15,894 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:35:15,895 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:35:15,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:35:15,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:35:15,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:35:15,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:35:15,979 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:35:16,005 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:35:16,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:35:16,755 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:35:16,755 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.8220000267. ############################################################ # # RAM FootPrint: 4402.34375 # ############################################################ 2022-02-08 06:35:16,789 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 06:35:16,789 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 86 seconds. Reason: Iteration:6 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 8879 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:36:42,796 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 06:36:42,796 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 06:36:47,799 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:36:47,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:36:48,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:36:48,275 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:36:48,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:36:48,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:36:48,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:36:48,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:36:48,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:36:48,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:36:48,423 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:36:48,441 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:36:48,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:36:48,519 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['13772 memcached\n'], []) 2022-02-08 06:36:48,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 06:36:48,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:36:48,585 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['13772 memcached\n'], []) 2022-02-08 06:36:48,585 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:36:48,588 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:36:49,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 8904 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:36:49,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:36:49,773 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:36:49,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:36:49,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:36:49,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:36:49,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:36:49,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:36:49,854 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:36:49,890 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:36:49,903 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:36:49,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:36:49,950 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['60650 memcached\n'], []) 2022-02-08 06:36:49,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 06:36:49,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:36:50,006 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['60650 memcached\n'], []) 2022-02-08 06:36:50,006 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:36:50,009 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:36:50,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:36:50,306 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:36:50,375 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:36:50,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:36:50,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:36:50,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:36:50,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:36:50,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:36:50,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:36:50,466 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:36:50,482 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:36:50,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:36:50,529 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['46391 memcached\n'], []) 2022-02-08 06:36:50,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 06:36:50,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:36:50,579 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['46391 memcached\n'], []) 2022-02-08 06:36:50,579 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:36:50,585 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:36:50,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:36:50,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:36:50,944 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:36:50,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:36:50,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:36:50,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:36:50,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:36:51,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:36:51,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:36:51,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:36:51,040 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:36:51,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:36:51,134 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['78347 memcached\n'], []) 2022-02-08 06:36:51,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 06:36:51,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:36:51,183 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['78347 memcached\n'], []) 2022-02-08 06:36:51,183 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:36:51,186 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:36:51,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:36:51,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:36:51,529 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:36:51,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:36:51,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:36:51,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:36:51,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:36:51,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:36:51,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:36:51,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:36:51,618 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:36:51,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:36:51,677 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['72226 memcached\n'], []) 2022-02-08 06:36:51,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 06:36:51,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:36:51,726 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['72226 memcached\n'], []) 2022-02-08 06:36:51,726 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:36:51,730 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:36:51,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:36:52,038 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:36:52,108 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:36:52,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:36:52,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:36:52,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:36:52,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:36:52,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:36:52,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:36:52,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:36:52,206 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:36:52,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:36:52,286 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['107207 memcached\n'], []) 2022-02-08 06:36:52,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 06:36:52,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:36:52,336 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['107207 memcached\n'], []) 2022-02-08 06:36:52,336 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:36:52,338 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:36:52,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:36:52,647 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:36:52,737 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:36:52,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:36:52,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:36:52,762 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:36:52,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:36:52,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:36:52,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:36:52,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:36:52,828 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:36:52,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:36:52,887 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['40994 memcached\n'], []) 2022-02-08 06:36:52,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 06:36:52,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:36:52,936 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['40994 memcached\n'], []) 2022-02-08 06:36:52,937 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:36:52,940 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:36:53,022 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:36:53,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:36:53,302 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:36:53,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:36:53,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:36:53,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:36:53,339 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:36:53,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:36:53,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:36:53,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:36:53,403 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:36:53,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:36:53,461 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['55909 memcached\n'], []) 2022-02-08 06:36:53,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 06:36:53,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:36:53,509 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['55909 memcached\n'], []) 2022-02-08 06:36:53,509 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:36:53,512 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:36:53,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:36:53,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:36:53,871 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:36:53,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:36:53,885 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:36:53,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:36:53,913 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:36:53,928 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:36:53,941 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:36:53,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:36:53,969 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:36:53,969 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:36:54,017 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['105301 memcached\n'], []) 2022-02-08 06:36:54,017 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 06:36:54,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:36:54,069 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['105301 memcached\n'], []) 2022-02-08 06:36:54,071 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:36:54,073 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:36:54,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:36:54,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:36:54,466 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:36:54,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:36:54,482 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:36:54,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:36:54,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:36:54,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:36:54,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:36:54,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:36:54,568 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:36:54,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:36:54,621 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['121988 memcached\n'], []) 2022-02-08 06:36:54,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 06:36:54,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:36:54,713 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['121988 memcached\n'], []) 2022-02-08 06:36:54,713 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:36:54,717 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:36:54,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:36:55,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:36:55,108 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:36:55,109 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:36:55,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:36:55,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:36:55,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:36:55,165 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:36:55,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:36:55,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:36:55,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:36:55,226 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 06:36:55,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:36:55,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:36:58,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:36:59,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:36:59,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:36:59,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:36:59,700 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 06:36:59,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:36:59,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:36:59,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:36:59,789 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 06:36:59,789 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:36:59,792 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:36:59,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:37:00,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:37:00,141 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:37:00,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:37:00,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:37:00,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:37:00,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:37:00,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:37:00,216 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:37:00,229 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:37:00,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:37:00,259 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 06:37:00,295 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:37:00,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:05,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:37:05,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:06,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:06,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:37:06,335 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 06:37:06,336 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:37:06,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:06,423 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:37:06,437 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 06:37:06,437 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:37:06,440 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:37:06,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:37:06,727 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:37:06,803 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:37:06,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:37:06,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:37:06,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:37:06,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:37:06,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:37:06,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:37:06,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:37:06,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:37:06,921 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 06:37:06,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:37:06,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:07,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:37:07,770 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:08,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:08,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:37:08,247 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 06:37:08,249 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:37:08,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:08,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:37:08,342 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 06:37:08,342 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:37:08,345 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 06:37:08,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:37:08,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 06:37:08,776 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:37:08,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 06:37:08,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 06:37:08,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 06:37:08,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 06:37:08,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 06:37:08,849 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 06:37:08,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 06:37:08,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:37:08,905 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 06:37:08,933 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:37:08,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:08,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:37:08,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:08,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:09,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:37:09,016 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 06:37:09,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:37:09,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:09,043 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:37:09,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:09,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:37:09,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:09,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:09,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:09,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:09,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:09,247 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:09,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:09,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:09,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:09,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:37:09,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:09,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:09,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:09,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:09,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:09,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:09,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:09,497 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:09,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:09,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:37:09,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:09,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:09,586 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:09,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:09,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:09,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:09,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:09,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:09,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:09,709 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:37:09,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:09,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:09,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:09,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:09,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:09,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:09,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:09,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:09,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:09,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:37:09,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:09,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:09,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:09,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:09,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:09,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:10,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:10,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:10,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:10,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:37:10,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:10,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:10,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:10,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:10,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:10,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:10,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:10,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:10,226 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 06:37:10,229 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:37:10,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:37:10,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:37:10,809 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:37:10,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:37:10,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:37:10,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:37:10,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:37:11,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:37:11,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:37:11,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:37:11,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:37:11,467 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 06:37:11,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:37:11,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:12,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:37:12,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:12,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:12,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:37:12,903 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 06:37:12,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:37:12,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:12,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:37:12,980 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 06:37:12,980 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:37:12,982 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 06:37:13,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:37:13,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 06:37:13,365 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:37:13,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 06:37:13,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 06:37:13,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 06:37:13,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 06:37:13,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 06:37:13,446 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 06:37:13,459 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 06:37:13,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:37:13,487 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 06:37:13,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:37:13,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:13,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:37:13,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:13,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:13,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:37:13,604 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 06:37:13,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:37:13,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:13,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:37:13,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:13,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:37:13,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:13,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:13,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:13,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:13,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:13,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:13,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:13,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:13,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:13,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:37:13,989 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:14,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:14,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:14,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:14,066 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:14,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:14,104 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:14,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:14,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:14,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:37:14,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:14,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:14,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:14,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:14,239 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:14,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:14,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:14,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:14,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:14,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:37:14,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:14,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:14,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:14,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:14,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:14,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:14,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:14,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:14,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:14,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:37:14,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:14,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:14,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:14,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:14,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:14,614 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:14,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:14,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:14,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:14,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:37:14,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:14,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:14,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:14,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:14,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:14,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:14,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:14,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:14,845 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 06:37:14,848 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:37:14,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:37:15,155 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:37:15,210 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:37:15,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:37:15,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:37:15,239 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:37:15,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:37:15,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:37:15,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:37:15,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:37:15,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:37:15,328 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 06:37:15,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:37:15,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:19,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:37:19,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:20,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:20,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:37:20,451 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 06:37:20,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:37:20,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:20,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:37:20,539 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 06:37:20,539 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:37:20,542 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:37:20,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:37:20,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:37:21,033 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:37:21,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:37:21,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:37:21,071 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:37:21,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:37:21,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:37:21,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:37:21,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:37:21,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:37:21,341 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 06:37:21,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:37:21,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:26,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:37:26,885 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:27,293 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:27,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:37:27,723 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 06:37:27,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:37:27,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:27,829 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:37:27,852 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 06:37:27,852 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:37:27,858 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 06:37:27,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:37:28,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 06:37:28,371 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:37:28,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 06:37:28,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 06:37:28,410 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 06:37:28,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 06:37:28,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 06:37:28,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 06:37:28,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 06:37:28,507 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:37:28,529 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 06:37:28,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:37:28,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:28,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:37:28,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:28,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:28,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:37:28,688 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 06:37:28,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:37:28,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:28,752 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:37:28,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:28,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:37:28,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:28,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:28,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:28,979 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:29,015 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:29,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:29,084 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:29,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:29,155 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:29,178 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:37:29,211 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:29,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:29,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:29,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:29,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:29,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:29,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:29,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:29,405 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:29,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:37:29,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:29,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:29,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:29,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:29,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:29,566 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:29,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:29,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:29,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:29,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:37:29,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:29,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:29,716 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:29,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:29,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:29,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:29,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:29,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:29,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:29,877 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:37:29,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:29,910 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:29,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:29,941 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:29,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:29,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:29,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:30,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:30,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:30,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:37:30,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:30,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:30,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:30,160 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:30,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:30,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:30,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:30,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:30,279 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 06:37:30,282 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 06:37:30,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:37:30,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 06:37:30,655 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:37:30,655 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 06:37:30,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 06:37:30,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 06:37:30,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 06:37:30,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 06:37:30,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 06:37:30,749 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 06:37:30,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:37:30,783 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 06:37:30,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:37:30,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:30,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:37:30,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:30,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:30,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:37:30,907 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 06:37:30,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:37:30,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:30,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:37:30,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:30,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:37:31,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:31,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:31,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:31,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:31,142 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:31,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:31,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:31,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:31,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:31,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:37:31,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:31,316 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:31,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:31,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:31,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:31,385 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:31,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:31,423 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:31,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:31,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:37:31,474 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:31,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:31,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:31,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:31,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:31,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:31,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:31,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:31,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:31,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:37:31,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:31,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:31,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:31,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:31,706 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:31,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:31,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:31,755 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:31,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:31,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:37:31,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:31,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:31,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:31,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:31,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:31,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:31,895 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:31,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:31,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:31,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:37:31,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:31,979 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:31,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:32,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:32,028 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:32,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:32,066 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:32,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:32,101 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 06:37:32,104 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:37:32,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:37:32,474 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:37:32,565 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:37:32,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:37:32,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:37:32,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:37:32,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:37:32,647 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:37:32,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:37:32,681 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:37:32,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:37:32,727 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 06:37:32,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:37:32,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:37,339 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:37:37,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:38,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:38,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:37:38,563 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 06:37:38,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:37:38,599 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:38,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:37:38,684 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 06:37:38,684 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:37:38,687 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:37:38,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:37:39,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:37:39,127 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:37:39,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:37:39,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:37:39,160 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:37:39,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:37:39,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:37:39,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:37:39,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:37:39,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:37:39,282 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 06:37:39,316 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:37:39,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:39,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:37:40,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:40,596 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:40,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:37:40,891 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 06:37:40,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:37:40,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:40,936 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:37:40,953 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 06:37:40,953 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:37:40,956 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:37:41,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:37:41,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:37:41,943 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:37:41,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:37:41,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:37:41,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:37:42,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:37:42,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:37:42,058 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:37:42,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:37:42,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:37:42,125 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 06:37:42,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:37:42,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:42,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:37:43,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:43,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:43,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:37:43,526 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 06:37:43,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:37:43,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:43,584 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:37:43,599 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 06:37:43,599 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:37:43,601 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:37:43,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:37:43,920 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:37:44,002 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:37:44,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:37:44,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:37:44,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:37:44,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:37:44,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:37:44,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:37:44,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:37:44,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:37:44,153 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 06:37:44,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:37:44,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:49,377 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 5 queries submitted, 0 failed, 8913 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:37:50,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:50,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:50,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:37:50,940 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 06:37:50,941 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:37:50,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:51,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:37:51,075 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 06:37:51,075 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:37:51,079 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 06:37:51,216 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:37:51,426 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 06:37:51,486 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:37:51,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 06:37:51,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 06:37:51,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 06:37:51,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 06:37:51,543 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 06:37:51,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 06:37:51,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 06:37:51,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:37:51,601 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 06:37:51,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:37:51,642 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:51,660 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:37:51,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:51,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:37:51,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:37:51,719 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 06:37:51,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:37:51,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:37:51,752 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:37:51,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:51,795 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:37:51,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:51,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:51,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:51,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:51,931 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:51,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:51,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:52,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:37:52,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:52,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:37:52,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:52,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:52,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:52,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:52,145 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:52,163 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:52,178 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:52,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:37:52,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:52,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:37:52,247 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:52,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:52,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:52,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:52,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:52,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:52,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:52,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:37:52,385 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:52,405 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:37:52,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:52,438 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:52,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:52,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:52,497 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:52,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:52,530 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:52,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:37:52,563 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:52,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:37:52,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:52,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:52,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:52,640 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:52,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:52,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:52,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:52,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:37:52,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:52,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:37:52,752 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:52,770 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:52,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:52,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:52,829 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:52,849 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:52,869 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:52,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:37:52,905 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 06:37:52,905 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:37:52,907 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:37:52,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:37:53,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:37:53,319 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:37:53,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:37:53,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:37:53,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:37:53,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:37:53,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:37:53,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:37:53,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:37:53,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:37:53,868 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:37:53,872 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.967000007629. 2022-02-08 06:37:53,872 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:37:53,875 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:37:53,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:37:54,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:37:54,371 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:37:54,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:37:54,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:37:54,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:37:54,426 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:37:54,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:37:54,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:37:54,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:37:54,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:37:54,829 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:37:54,829 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:37:56,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:37:57,022 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:37:57,023 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:37:59,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:37:59,198 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:37:59,198 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:38:01,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:01,457 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:38:01,457 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:38:03,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:03,657 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:38:03,657 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:38:05,658 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:05,855 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:38:05,857 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:38:07,857 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:08,030 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:38:08,030 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:38:10,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:10,532 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:38:10,532 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:38:12,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:12,957 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:38:12,957 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:38:14,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:15,155 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:38:15,157 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:38:17,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:17,434 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:38:17,434 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:38:19,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:19,634 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:38:19,634 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:38:21,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:21,792 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:38:21,793 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:38:23,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:23,947 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:38:23,948 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:38:25,948 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:26,092 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:38:26,092 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:38:28,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:28,249 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:38:28,249 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:38:30,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:30,398 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:38:30,398 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:38:32,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:32,536 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:38:32,536 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 38.6630001068. 2022-02-08 06:38:32,536 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:38:32,538 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:38:33,145 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:38:33,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:38:33,494 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:38:33,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:38:33,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:38:33,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:38:33,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:38:33,573 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:38:33,589 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:38:33,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:38:33,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:34,124 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:38:34,125 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.58899998665. 2022-02-08 06:38:34,125 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:38:34,127 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:38:34,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:38:35,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:38:35,210 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:38:35,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:38:35,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:38:35,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:38:35,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:38:35,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:38:35,283 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:38:35,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:38:35,326 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:35,931 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:38:35,933 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.80800008774. 2022-02-08 06:38:35,933 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:38:35,934 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:38:36,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:38:37,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:38:37,368 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:38:37,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:38:37,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:38:37,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:38:37,413 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:38:37,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:38:37,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:38:37,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:38:37,487 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:37,954 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:38:37,954 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.02199983597. 2022-02-08 06:38:37,954 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:38:37,957 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:38:38,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:38:38,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:38:38,822 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:38:38,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:38:38,834 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:38:38,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:38:38,869 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:38:38,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:38:38,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:38:38,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:38:38,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:39,414 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:38:39,414 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.45900011063. 2022-02-08 06:38:39,414 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:38:39,417 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:38:40,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:38:40,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:38:40,318 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:38:40,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:38:40,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:38:40,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:38:40,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:38:40,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:38:40,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:38:40,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:38:40,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:40,882 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:38:40,884 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.47000002861. 2022-02-08 06:38:40,884 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:38:40,885 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:38:41,418 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:38:41,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:38:41,808 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:38:41,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:38:41,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:38:41,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:38:41,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:38:41,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:38:41,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:38:41,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:38:41,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:42,760 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:38:42,762 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.87699985504. 2022-02-08 06:38:42,762 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:38:42,763 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:38:43,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:38:43,624 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:38:43,706 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:38:43,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:38:43,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:38:43,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:38:43,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:38:43,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:38:43,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:38:43,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:38:43,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:44,484 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:38:44,484 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.72299981117. 2022-02-08 06:38:44,484 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:38:44,486 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:38:45,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:38:45,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:38:45,430 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:38:45,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:38:45,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:38:45,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:38:45,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:38:45,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:38:45,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:38:45,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:38:45,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:38:47,121 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:38:47,121 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.6360001564. ############################################################ # # RAM FootPrint: 4385.23828125 # ############################################################ 2022-02-08 06:38:47,155 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 06:38:47,157 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 88 seconds. Reason: Iteration:7 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 8925 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 8949 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:40:15,164 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 06:40:15,164 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 06:40:20,167 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:40:20,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:20,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:40:20,630 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:20,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:40:20,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:40:20,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:40:20,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:40:20,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:40:20,716 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:40:20,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:40:20,753 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:40:20,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:40:20,818 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['14467 memcached\n'], []) 2022-02-08 06:40:20,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 06:40:20,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:40:20,887 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['14467 memcached\n'], []) 2022-02-08 06:40:20,887 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:40:20,890 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:40:21,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:21,658 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:40:21,743 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:21,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:40:21,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:40:21,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:40:21,795 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:40:21,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:40:21,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:40:21,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:40:21,871 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:40:21,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:40:21,938 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['61568 memcached\n'], []) 2022-02-08 06:40:21,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 06:40:21,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:40:21,984 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['61568 memcached\n'], []) 2022-02-08 06:40:21,986 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:40:21,989 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:40:22,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:22,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:40:22,371 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:22,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:40:22,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:40:22,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:40:22,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:40:22,441 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:40:22,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:40:22,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:40:22,493 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:40:22,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:40:22,539 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['47137 memcached\n'], []) 2022-02-08 06:40:22,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 06:40:22,566 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:40:22,592 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['47137 memcached\n'], []) 2022-02-08 06:40:22,592 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:40:22,595 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:40:22,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:22,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:40:22,961 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:22,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:40:22,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:40:22,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:40:23,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:40:23,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:40:23,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:40:23,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:40:23,065 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:40:23,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:40:23,118 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['79039 memcached\n'], []) 2022-02-08 06:40:23,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 06:40:23,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:40:23,181 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['79039 memcached\n'], []) 2022-02-08 06:40:23,183 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:40:23,184 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:40:23,267 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:23,482 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:40:23,542 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:23,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:40:23,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:40:23,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:40:23,579 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:40:23,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:40:23,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:40:23,619 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:40:23,632 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:40:23,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:40:23,690 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['72875 memcached\n'], []) 2022-02-08 06:40:23,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 06:40:23,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:40:23,739 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['72875 memcached\n'], []) 2022-02-08 06:40:23,739 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:40:23,742 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:40:23,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:24,071 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:40:24,131 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:24,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:40:24,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:40:24,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:40:24,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:40:24,186 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:40:24,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:40:24,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:40:24,227 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:40:24,227 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:40:24,279 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['107857 memcached\n'], []) 2022-02-08 06:40:24,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 06:40:24,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:40:24,325 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['107857 memcached\n'], []) 2022-02-08 06:40:24,325 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:40:24,328 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:40:24,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:24,632 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:40:24,694 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:24,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:40:24,709 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:40:24,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:40:24,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:40:24,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:40:24,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:40:24,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:40:24,788 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:40:24,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:40:24,831 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['41691 memcached\n'], []) 2022-02-08 06:40:24,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 06:40:24,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:40:24,875 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['41691 memcached\n'], []) 2022-02-08 06:40:24,875 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:40:24,880 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:40:24,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:25,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:40:25,272 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:25,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:40:25,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:40:25,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:40:25,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:40:25,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:40:25,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:40:25,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:40:25,369 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:40:25,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:40:25,430 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['56606 memcached\n'], []) 2022-02-08 06:40:25,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 06:40:25,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:40:25,473 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['56606 memcached\n'], []) 2022-02-08 06:40:25,473 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:40:25,476 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:40:25,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:25,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:40:25,859 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:25,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:40:25,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:40:25,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:40:25,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:40:25,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:40:25,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:40:25,961 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:40:25,976 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:40:25,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:40:26,020 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['105993 memcached\n'], []) 2022-02-08 06:40:26,022 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 06:40:26,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:40:26,072 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['105993 memcached\n'], []) 2022-02-08 06:40:26,072 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:40:26,075 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:40:26,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:26,387 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:40:26,448 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:26,448 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:40:26,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:40:26,474 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:40:26,487 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:40:26,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:40:26,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:40:26,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:40:26,549 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:40:26,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:40:26,594 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['122642 memcached\n'], []) 2022-02-08 06:40:26,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 06:40:26,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:40:26,638 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['122642 memcached\n'], []) 2022-02-08 06:40:26,638 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:40:26,641 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:40:26,714 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:26,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:40:27,015 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:27,015 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:40:27,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:40:27,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:40:27,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:40:27,073 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:40:27,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:40:27,104 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:40:27,119 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:40:27,138 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 06:40:27,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:40:27,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:27,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:40:28,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:28,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:28,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:40:28,877 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 06:40:28,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:40:28,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:28,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:40:28,961 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 06:40:28,963 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:40:28,967 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:40:29,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:29,270 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:40:29,331 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:29,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:40:29,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:40:29,357 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:40:29,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:40:29,466 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:40:29,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:40:29,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:40:29,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:40:29,520 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 06:40:29,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:40:29,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:30,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:40:30,640 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:30,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:31,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:40:31,267 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 06:40:31,267 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:40:31,283 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:31,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:40:31,367 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 06:40:31,367 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:40:31,371 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:40:31,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:31,647 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:40:31,710 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:31,711 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:40:31,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:40:31,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:40:31,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:40:31,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:40:31,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:40:31,801 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:40:31,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:40:31,832 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 06:40:31,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:40:31,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:32,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:40:32,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:32,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:33,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:40:33,194 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 06:40:33,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:40:33,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:33,306 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:40:33,322 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 06:40:33,322 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:40:33,325 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 06:40:33,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:33,647 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 06:40:33,709 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:33,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 06:40:33,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 06:40:33,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 06:40:33,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 06:40:33,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 06:40:33,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 06:40:33,801 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 06:40:33,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:40:33,832 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 06:40:33,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:40:33,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:33,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:40:33,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:33,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:33,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:40:33,961 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 06:40:33,961 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:40:33,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:33,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:40:34,005 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:34,035 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:40:34,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:34,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:34,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:34,145 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:34,178 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:34,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:34,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:34,272 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:34,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:34,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:40:34,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:34,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:34,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:34,392 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:34,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:34,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:34,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:34,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:34,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:34,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:40:34,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:34,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:34,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:34,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:34,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:34,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:34,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:34,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:34,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:34,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:40:34,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:34,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:34,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:34,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:34,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:34,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:34,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:34,801 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:34,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:34,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:40:34,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:34,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:34,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:34,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:34,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:34,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:34,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:34,989 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:35,005 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:35,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:40:35,058 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:35,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:35,112 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:35,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:35,150 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:35,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:35,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:35,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:35,223 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 06:40:35,229 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:40:35,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:35,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:40:36,224 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:36,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:40:36,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:40:36,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:40:36,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:40:36,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:40:36,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:40:37,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:40:37,127 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:40:37,269 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 06:40:37,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:40:37,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:37,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:40:38,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:38,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:39,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:40:39,289 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 06:40:39,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:40:39,308 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:39,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:40:39,569 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 06:40:39,569 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:40:39,572 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 06:40:39,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:39,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 06:40:39,963 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:39,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 06:40:39,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 06:40:40,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 06:40:40,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 06:40:40,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 06:40:40,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 06:40:40,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 06:40:40,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:40:40,096 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 06:40:40,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:40:40,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:40,170 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:40:40,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:40,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:40,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:40:40,234 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 06:40:40,234 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:40:40,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:40,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:40:40,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:40,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:40:40,341 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:40,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:40,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:40,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:40,456 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:40,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:40,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:40,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:40,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:40,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:40:40,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:40,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:40,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:40,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:40,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:40,714 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:40,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:40,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:40,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:40,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:40:40,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:40,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:40,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:40,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:40,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:40,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:40,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:40,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:40,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:40,961 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:40:40,982 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:41,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:41,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:41,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:41,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:41,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:41,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:41,099 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:41,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:41,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:40:41,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:41,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:41,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:41,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:41,224 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:41,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:41,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:41,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:41,293 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:41,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:40:41,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:41,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:41,362 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:41,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:41,397 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:41,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:41,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:41,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:41,470 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 06:40:41,471 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:40:41,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:41,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:40:41,829 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:41,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:40:41,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:40:41,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:40:41,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:40:41,890 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:40:41,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:40:41,918 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:40:41,933 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:40:41,947 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 06:40:41,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:40:42,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:43,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:40:44,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:44,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:44,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:40:44,790 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 06:40:44,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:40:44,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:44,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:40:44,865 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 06:40:44,865 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:40:44,868 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:40:44,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:45,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:40:45,252 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:45,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:40:45,267 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:40:45,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:40:45,308 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:40:45,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:40:45,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:40:45,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:40:45,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:40:45,392 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 06:40:45,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:40:45,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:46,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:40:46,665 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:46,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:47,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:40:47,292 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 06:40:47,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:40:47,308 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:47,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:40:47,387 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 06:40:47,388 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:40:47,390 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 06:40:47,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:47,711 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 06:40:47,779 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:47,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 06:40:47,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 06:40:47,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 06:40:47,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 06:40:47,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 06:40:47,849 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 06:40:47,862 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 06:40:47,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:40:47,894 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 06:40:47,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:40:47,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:47,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:40:47,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:47,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:47,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:40:48,006 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 06:40:48,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:40:48,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:48,035 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:40:48,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:48,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:40:48,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:48,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:48,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:48,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:48,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:48,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:48,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:48,316 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:48,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:48,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:40:48,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:48,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:48,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:48,426 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:48,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:48,456 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:48,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:48,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:48,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:48,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:40:48,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:48,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:48,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:48,589 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:48,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:48,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:48,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:48,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:48,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:48,688 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:40:48,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:48,721 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:48,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:48,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:48,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:48,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:48,801 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:48,816 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:48,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:48,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:40:48,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:48,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:48,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:48,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:48,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:48,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:48,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:48,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:49,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:49,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:40:49,056 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:49,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:49,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:49,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:49,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:49,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:49,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:49,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:49,213 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 06:40:49,214 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 06:40:49,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 8969 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:40:49,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 06:40:49,634 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:49,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 06:40:49,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 06:40:49,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 06:40:49,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 06:40:49,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 06:40:49,711 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 06:40:49,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 06:40:49,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:40:49,759 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 06:40:49,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:40:49,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:49,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:40:49,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:49,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:49,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:40:49,885 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 06:40:49,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:40:49,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:49,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:40:49,931 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:49,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:40:49,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:50,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:50,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:50,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:50,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:50,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:50,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:50,201 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:40:50,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:50,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:40:50,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:50,280 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:50,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:50,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:50,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:50,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:50,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:50,405 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:40:50,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:50,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:40:50,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:50,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:50,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:50,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:50,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:50,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:50,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:50,573 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:40:50,589 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:50,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:40:50,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:50,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:50,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:50,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:50,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:50,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:50,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:50,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:40:50,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:50,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:40:50,783 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:50,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:50,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:50,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:50,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:50,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:50,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:50,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:40:50,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:50,936 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:40:50,964 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:50,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:51,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:51,017 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:51,035 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:51,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:51,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:51,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:40:51,105 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 06:40:51,108 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:40:51,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:51,428 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:40:51,493 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:51,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:40:51,507 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:40:51,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:40:51,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:40:51,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:40:51,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:40:51,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:40:51,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:40:51,607 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 06:40:51,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:40:51,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:52,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:40:52,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:52,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:53,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:40:53,315 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 06:40:53,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:40:53,329 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:53,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:40:53,421 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 06:40:53,423 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:40:53,424 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:40:53,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:53,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:40:53,808 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:53,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:40:53,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:40:53,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:40:53,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:40:53,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:40:53,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:40:53,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:40:53,918 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:40:53,940 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 06:40:53,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:40:54,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:54,408 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:40:54,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:55,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:55,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:40:55,378 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 06:40:55,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:40:55,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:55,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:40:55,447 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 06:40:55,447 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:40:55,450 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:40:56,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:56,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:40:56,607 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:56,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:40:56,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:40:56,645 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:40:56,681 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:40:56,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:40:56,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:40:56,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:40:56,760 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:40:56,786 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 06:40:56,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:40:56,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:57,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:40:58,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:58,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:40:58,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:40:58,921 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 06:40:58,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:40:58,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:40:59,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:40:59,046 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 06:40:59,046 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:40:59,048 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:40:59,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:40:59,385 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:40:59,473 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:40:59,474 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:40:59,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:40:59,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:40:59,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:40:59,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:40:59,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:40:59,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:40:59,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:40:59,664 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 06:40:59,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:40:59,829 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:41:00,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:41:01,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:41:01,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:41:02,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:41:02,367 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 06:41:02,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:41:02,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:41:02,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:41:02,821 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 06:41:02,822 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:41:02,828 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 06:41:02,920 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:41:03,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 06:41:03,220 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:41:03,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 06:41:03,239 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 06:41:03,255 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 06:41:03,270 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 06:41:03,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 06:41:03,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 06:41:03,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 06:41:03,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:41:03,359 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 06:41:03,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:41:03,413 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:41:03,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:41:03,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:41:03,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:41:03,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:41:03,502 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 06:41:03,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:41:03,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:41:03,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:41:03,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:41:03,584 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:41:03,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:41:03,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:41:03,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:41:03,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:41:03,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:41:03,788 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:41:03,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:41:03,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:41:03,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:41:03,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:41:03,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:41:03,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:41:03,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:41:03,982 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:41:04,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:41:04,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:41:04,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:41:04,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:41:04,084 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:41:04,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:41:04,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:41:04,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:41:04,160 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:41:04,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:41:04,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:41:04,211 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:41:04,229 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:41:04,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:41:04,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:41:04,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:41:04,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:41:04,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:41:04,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:41:04,349 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:41:04,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:41:04,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:41:04,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:41:04,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:41:04,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:41:04,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:41:04,464 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:41:04,482 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:41:04,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:41:04,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:41:04,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:41:04,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:41:04,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:41:04,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:41:04,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:41:04,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:41:04,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:41:04,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:41:04,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:41:04,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:41:04,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:41:04,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:41:04,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:41:04,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:41:04,776 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 06:41:04,778 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:41:04,780 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:41:04,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:41:05,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:41:05,246 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:41:05,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:41:05,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:41:05,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:41:05,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:41:05,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:41:05,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:41:05,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:41:05,392 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:05,973 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:41:05,973 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.19599986076. 2022-02-08 06:41:05,974 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:41:05,976 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:41:06,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:41:06,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:41:06,721 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:41:06,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:41:06,749 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:41:06,765 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:41:06,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:41:06,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:41:06,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:41:06,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:41:06,869 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:07,081 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:07,081 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:09,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:09,319 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:09,321 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:11,321 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:11,644 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:11,644 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:13,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:14,032 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:14,033 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:16,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:16,280 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:16,282 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:18,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:18,506 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:18,507 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:20,507 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:20,743 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:20,743 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:22,744 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:23,006 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:23,006 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:25,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:25,243 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:25,244 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:27,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:27,457 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:27,457 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:29,459 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:29,987 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:29,987 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:31,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:32,203 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:32,203 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:34,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:34,378 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:34,378 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:36,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:36,569 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:36,569 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:38,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:38,746 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:38,746 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:40,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:40,904 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:40,905 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:42,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:43,065 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:43,065 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:45,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:45,229 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:45,230 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:47,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:47,387 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:47,387 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:49,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 8974 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:41:49,545 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:49,545 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:51,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:51,697 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:51,698 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:53,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:53,858 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:53,859 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:55,859 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:56,017 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:56,019 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:41:58,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:41:58,167 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:41:58,168 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:42:00,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:42:00,334 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:42:00,334 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:42:02,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:42:02,490 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:42:02,490 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:42:04,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:42:04,647 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:42:04,647 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:42:06,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:42:06,795 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:42:06,795 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 60.8200001717. 2022-02-08 06:42:06,796 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:42:06,799 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:42:07,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:42:08,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:42:08,164 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:42:08,165 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:42:08,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:42:08,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:42:08,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:42:08,247 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:42:08,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:42:08,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:42:08,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:42:08,964 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:42:08,964 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.16999983788. 2022-02-08 06:42:08,964 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:42:08,967 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:42:09,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:42:10,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:42:10,108 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:42:10,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:42:10,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:42:10,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:42:10,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:42:10,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:42:10,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:42:10,214 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:42:10,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:42:10,821 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:42:10,822 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.85700011253. 2022-02-08 06:42:10,822 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:42:10,823 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:42:11,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:42:12,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:42:12,793 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:42:12,795 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:42:12,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:42:12,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:42:13,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:42:13,165 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:42:13,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:42:13,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:42:13,329 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:42:13,992 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:42:13,993 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.17100000381. 2022-02-08 06:42:13,993 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:42:13,996 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:42:14,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:42:15,280 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:42:15,390 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:42:15,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:42:15,405 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:42:15,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:42:15,482 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:42:15,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:42:15,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:42:15,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:42:15,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:42:17,506 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:42:17,506 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.51300001144. 2022-02-08 06:42:17,506 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:42:17,509 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:42:18,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:42:18,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:42:18,453 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:42:18,454 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:42:18,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:42:18,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:42:18,505 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:42:18,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:42:18,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:42:18,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:42:18,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:42:19,036 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:42:19,036 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.52999997139. 2022-02-08 06:42:19,036 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:42:19,039 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:42:19,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:42:20,487 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:42:20,658 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:42:20,660 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:42:20,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:42:20,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:42:20,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:42:20,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:42:20,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:42:20,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:42:20,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:42:23,509 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:42:23,509 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.47200012207. 2022-02-08 06:42:23,509 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:42:23,512 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:42:24,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:42:24,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:42:25,020 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:42:25,022 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:42:25,035 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:42:25,058 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:42:25,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:42:25,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:42:25,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:42:25,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:42:25,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:42:26,005 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:42:26,005 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.49500012398. 2022-02-08 06:42:26,006 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:42:26,007 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:42:26,545 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:42:26,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:42:26,910 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:42:26,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:42:26,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:42:26,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:42:26,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:42:27,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:42:27,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:42:27,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:42:27,068 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:42:28,094 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:42:28,095 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.08899998665. ############################################################ # # RAM FootPrint: 4390.12890625 # ############################################################ 2022-02-08 06:42:28,132 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 06:42:28,132 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 80 seconds. Reason: Iteration:8 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 8993 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:43:48,138 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 06:43:48,138 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 5 queries submitted, 0 failed, 9014 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:43:53,141 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:43:53,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:43:53,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:43:53,530 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:43:53,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:43:53,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:43:53,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:43:53,589 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:43:53,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:43:53,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:43:53,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:43:53,651 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:43:53,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:43:53,717 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['15142 memcached\n'], []) 2022-02-08 06:43:53,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 06:43:53,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:43:53,767 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['15142 memcached\n'], []) 2022-02-08 06:43:53,769 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:43:53,772 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:43:54,339 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:43:54,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:43:54,710 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:43:54,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:43:54,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:43:54,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:43:54,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:43:54,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:43:54,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:43:54,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:43:54,832 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:43:54,834 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:43:54,894 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['62469 memcached\n'], []) 2022-02-08 06:43:54,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 06:43:54,928 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:43:54,950 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['62469 memcached\n'], []) 2022-02-08 06:43:54,951 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:43:54,954 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:43:55,038 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:43:55,257 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:43:55,321 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:43:55,321 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:43:55,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:43:55,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:43:55,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:43:55,385 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:43:55,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:43:55,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:43:55,434 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:43:55,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:43:55,486 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['47905 memcached\n'], []) 2022-02-08 06:43:55,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 06:43:55,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:43:55,542 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['47905 memcached\n'], []) 2022-02-08 06:43:55,542 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:43:55,546 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:43:55,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:43:55,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:43:55,917 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:43:55,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:43:55,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:43:55,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:43:55,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:43:55,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:43:55,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:43:56,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:43:56,013 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:43:56,015 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:43:56,059 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['79696 memcached\n'], []) 2022-02-08 06:43:56,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 06:43:56,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:43:56,124 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['79696 memcached\n'], []) 2022-02-08 06:43:56,125 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:43:56,125 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:43:56,227 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:43:56,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:43:56,494 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:43:56,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:43:56,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:43:56,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:43:56,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:43:56,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:43:56,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:43:56,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:43:56,592 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:43:56,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:43:56,638 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['73549 memcached\n'], []) 2022-02-08 06:43:56,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 06:43:56,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:43:56,684 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['73549 memcached\n'], []) 2022-02-08 06:43:56,684 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:43:56,686 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:43:56,765 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:43:56,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:43:57,040 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:43:57,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:43:57,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:43:57,066 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:43:57,081 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:43:57,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:43:57,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:43:57,140 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:43:57,154 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:43:57,155 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:43:57,280 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['108528 memcached\n'], []) 2022-02-08 06:43:57,280 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 06:43:57,306 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:43:57,328 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['108528 memcached\n'], []) 2022-02-08 06:43:57,329 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:43:57,332 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:43:57,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:43:57,645 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:43:57,710 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:43:57,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:43:57,727 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:43:57,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:43:57,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:43:57,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:43:57,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:43:57,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:43:57,811 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:43:57,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:43:57,852 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['42373 memcached\n'], []) 2022-02-08 06:43:57,854 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 06:43:57,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:43:57,901 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['42373 memcached\n'], []) 2022-02-08 06:43:57,901 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:43:57,904 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:43:57,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:43:58,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:43:58,260 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:43:58,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:43:58,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:43:58,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:43:58,303 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:43:58,321 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:43:58,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:43:58,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:43:58,358 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:43:58,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:43:58,421 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['57265 memcached\n'], []) 2022-02-08 06:43:58,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 06:43:58,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:43:58,467 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['57265 memcached\n'], []) 2022-02-08 06:43:58,467 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:43:58,470 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:43:58,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:43:58,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:43:58,867 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:43:58,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:43:58,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:43:58,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:43:58,910 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:43:58,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:43:58,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:43:58,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:43:58,966 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:43:58,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:43:59,013 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['106661 memcached\n'], []) 2022-02-08 06:43:59,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 06:43:59,038 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:43:59,058 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['106661 memcached\n'], []) 2022-02-08 06:43:59,059 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:43:59,061 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:43:59,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:43:59,408 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:43:59,477 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:43:59,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:43:59,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:43:59,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:43:59,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:43:59,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:43:59,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:43:59,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:43:59,581 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:43:59,581 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:43:59,630 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['123318 memcached\n'], []) 2022-02-08 06:43:59,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 06:43:59,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:43:59,677 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['123318 memcached\n'], []) 2022-02-08 06:43:59,677 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:43:59,678 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:43:59,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:43:59,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:44:00,059 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:44:00,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:44:00,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:44:00,091 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:44:00,104 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:44:00,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:44:00,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:44:00,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:44:00,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:44:00,183 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 06:44:00,214 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:44:00,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:03,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:44:04,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:04,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:04,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:44:04,842 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 06:44:04,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:44:04,857 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:04,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:44:04,924 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 06:44:04,926 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:44:04,927 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:44:05,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:44:05,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:44:05,289 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:44:05,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:44:05,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:44:05,316 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:44:05,329 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:44:05,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:44:05,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:44:05,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:44:05,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:44:05,411 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 06:44:05,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:44:05,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:10,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:44:10,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:11,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:11,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:44:11,448 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 06:44:11,448 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:44:11,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:11,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:44:11,536 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 06:44:11,538 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:44:11,540 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:44:11,624 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:44:11,823 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:44:11,898 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:44:11,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:44:11,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:44:11,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:44:11,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:44:11,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:44:11,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:44:12,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:44:12,015 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:44:12,032 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 06:44:12,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:44:12,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:15,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:44:16,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:16,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:16,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:44:16,737 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 06:44:16,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:44:16,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:16,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:44:16,832 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 06:44:16,834 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:44:16,835 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 06:44:16,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:44:17,119 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 06:44:17,184 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:44:17,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 06:44:17,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 06:44:17,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 06:44:17,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 06:44:17,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 06:44:17,267 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 06:44:17,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 06:44:17,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:44:17,312 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 06:44:17,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:44:17,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:17,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:44:17,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:17,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:17,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:44:17,428 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 06:44:17,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:44:17,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:17,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:44:17,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:17,515 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:44:17,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:17,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:17,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:17,632 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:17,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:17,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:17,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:17,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:17,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:17,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:44:17,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:17,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:17,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:17,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:17,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:17,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:17,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:17,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:17,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:17,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:44:17,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:17,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:18,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:18,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:18,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:18,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:18,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:18,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:18,099 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:18,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:44:18,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:18,147 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:18,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:18,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:18,193 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:18,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:18,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:18,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:18,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:18,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:44:18,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:18,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:18,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:18,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:18,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:18,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:18,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:18,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:18,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:18,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:44:18,428 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:18,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:18,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:18,482 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:18,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:18,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:18,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:18,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:18,566 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 06:44:18,569 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:44:18,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:44:18,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:44:18,927 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:44:18,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:44:18,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:44:18,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:44:18,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:44:19,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:44:19,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:44:19,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:44:19,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:44:19,069 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 06:44:19,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:44:19,112 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:19,454 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:44:19,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:20,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:20,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:44:20,295 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 06:44:20,295 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:44:20,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:20,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:44:20,384 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 06:44:20,385 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:44:20,388 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 06:44:20,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:44:20,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 06:44:20,772 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:44:20,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 06:44:20,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 06:44:20,798 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 06:44:20,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 06:44:20,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 06:44:20,841 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 06:44:20,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 06:44:20,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:44:20,888 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 06:44:20,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:44:20,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:20,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:44:20,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:20,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:21,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:44:21,012 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 06:44:21,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:44:21,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:21,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:44:21,053 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:21,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:44:21,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:21,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:21,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:21,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:21,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:21,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:21,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:21,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:21,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:21,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:44:21,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:21,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:21,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:21,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:21,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:21,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:21,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:21,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:21,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:21,535 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:44:21,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:21,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:21,579 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:21,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:21,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:21,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:21,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:21,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:21,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:21,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:44:21,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:21,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:21,732 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:21,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:21,760 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:21,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:21,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:21,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:21,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:21,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:44:21,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:21,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:21,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:21,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:21,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:21,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:21,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:21,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:21,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:22,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:44:22,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:22,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:22,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:22,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:22,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:22,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:22,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:22,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:22,155 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 06:44:22,157 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:44:22,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:44:22,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:44:22,506 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:44:22,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:44:22,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:44:22,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:44:22,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:44:22,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:44:22,586 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:44:22,599 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:44:22,614 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:44:22,628 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 06:44:22,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:44:22,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:25,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:44:25,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:26,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:26,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:44:26,624 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 06:44:26,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:44:26,640 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:26,693 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:44:26,750 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 06:44:26,750 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:44:26,753 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:44:26,862 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:44:27,155 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:44:27,289 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:44:27,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:44:27,308 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:44:27,326 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:44:27,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:44:27,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:44:27,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:44:27,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:44:27,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:44:27,497 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 06:44:27,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:44:27,543 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:31,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:44:32,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:33,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:33,650 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:44:33,671 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 06:44:33,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:44:33,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:33,770 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:44:33,792 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 06:44:33,793 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:44:33,796 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 06:44:33,941 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:44:34,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 06:44:34,283 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:44:34,285 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 06:44:34,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 06:44:34,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 06:44:34,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 06:44:34,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 06:44:34,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 06:44:34,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 06:44:34,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:44:34,427 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 06:44:34,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:44:34,482 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:34,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:44:34,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:34,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:34,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:44:34,576 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 06:44:34,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:44:34,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:34,614 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:44:34,630 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:34,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:44:34,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:34,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:34,770 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:34,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:34,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:34,877 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:34,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:34,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:34,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:34,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:44:35,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:35,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:35,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:35,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:35,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:35,112 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:35,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:35,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:35,170 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:35,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:44:35,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:35,229 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:35,249 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:35,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:35,290 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:35,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:35,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:35,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:35,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:35,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:44:35,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:35,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:35,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:35,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:35,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:35,535 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:35,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:35,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:35,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:35,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:44:35,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:35,670 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:35,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:35,711 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:35,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:35,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:35,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:35,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:35,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:35,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:44:35,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:35,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:35,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:35,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:35,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:35,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:35,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:36,005 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:36,026 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 06:44:36,029 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 06:44:36,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:44:36,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 06:44:36,404 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:44:36,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 06:44:36,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 06:44:36,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 06:44:36,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 06:44:36,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 06:44:36,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 06:44:36,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 06:44:36,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:44:36,532 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 06:44:36,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:44:36,581 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:36,596 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:44:36,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:36,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:36,642 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:44:36,657 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 06:44:36,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:44:36,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:36,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:44:36,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:36,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:44:36,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:36,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:36,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:36,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:36,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:36,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:36,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:37,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:44:37,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:37,043 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:44:37,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:37,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:37,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:37,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:37,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:37,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:37,170 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:37,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:44:37,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:37,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:44:37,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:37,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:37,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:37,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:37,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:37,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:37,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:37,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:44:37,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:37,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:44:37,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:37,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:37,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:37,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:37,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:37,497 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:37,515 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:37,530 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:44:37,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:37,566 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:44:37,584 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:37,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:37,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:37,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:37,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:37,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:37,693 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:37,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:44:37,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:37,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:44:37,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:37,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:37,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:37,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:37,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:37,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:37,885 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:37,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:44:37,921 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 06:44:37,924 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:44:38,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:44:38,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:44:38,546 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:44:38,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:44:38,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:44:38,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:44:38,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:44:38,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:44:38,678 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:44:38,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:44:38,729 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:44:38,753 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 06:44:38,783 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:44:38,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:42,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:44:42,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:43,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:43,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:44:43,563 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 06:44:43,563 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:44:43,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:43,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:44:43,750 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 06:44:43,750 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:44:43,753 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:44:43,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:44:44,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:44:44,178 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:44:44,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:44:44,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:44:44,216 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:44:44,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:44:44,257 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:44:44,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:44:44,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:44:44,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:44:44,335 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 06:44:44,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:44:44,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:48,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:44:48,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:49,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:49,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:44:49,336 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 06:44:49,336 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:44:49,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:49,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:44:49,413 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 06:44:49,414 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:44:49,415 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 5 queries submitted, 0 failed, 9024 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:44:50,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:44:50,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:44:50,657 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:44:50,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:44:50,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:44:50,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:44:50,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:44:50,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:44:50,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:44:50,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:44:50,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:44:50,846 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 06:44:50,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:44:50,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:54,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:44:55,270 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:55,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:44:55,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:44:55,812 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 06:44:55,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:44:55,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:44:55,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:44:55,890 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 06:44:55,890 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:44:55,891 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:44:56,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:44:56,505 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:44:56,703 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:44:56,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:44:56,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:44:56,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:44:56,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:44:56,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:44:57,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:44:57,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:44:57,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:44:57,374 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 06:44:57,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:44:57,423 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:45:01,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:45:02,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:45:03,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:45:03,931 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:45:04,066 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 06:45:04,068 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:45:04,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:45:04,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:45:04,240 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 06:45:04,240 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:45:04,243 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 06:45:04,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:45:04,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 06:45:04,640 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:45:04,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 06:45:04,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 06:45:04,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 06:45:04,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 06:45:04,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 06:45:04,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 06:45:04,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 06:45:04,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:45:04,766 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 06:45:04,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:45:04,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:45:04,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:45:04,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:45:04,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:45:04,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:45:04,882 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 06:45:04,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:45:04,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:45:04,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:45:04,928 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:45:04,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:45:04,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:45:05,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:45:05,056 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:45:05,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:45:05,127 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:45:05,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:45:05,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:45:05,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:45:05,249 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:45:05,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:45:05,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:45:05,306 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:45:05,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:45:05,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:45:05,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:45:05,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:45:05,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:45:05,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:45:05,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:45:05,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:45:05,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:45:05,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:45:05,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:45:05,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:45:05,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:45:05,571 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:45:05,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:45:05,607 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:45:05,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:45:05,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:45:05,660 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:45:05,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:45:05,693 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:45:05,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:45:05,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:45:05,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:45:05,755 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:45:05,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:45:05,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:45:05,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:45:05,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:45:05,834 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:45:05,849 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:45:05,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:45:05,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:45:05,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:45:05,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:45:05,931 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:45:05,948 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:45:05,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:45:05,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:45:06,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:45:06,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:45:06,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:45:06,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:45:06,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:45:06,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:45:06,119 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:45:06,138 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 06:45:06,138 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:45:06,141 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:45:06,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:45:06,448 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:45:06,533 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:45:06,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:45:06,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:45:06,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:45:06,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:45:06,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:45:06,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:45:06,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:45:06,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:07,098 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:45:07,098 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.960000038147. 2022-02-08 06:45:07,099 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:45:07,101 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:45:08,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:45:08,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:45:08,598 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:45:08,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:45:08,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:45:08,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:45:08,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:45:08,788 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:45:08,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:45:08,877 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:45:08,903 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:09,144 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:45:09,144 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:45:11,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:11,312 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:45:11,312 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:45:13,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:13,482 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:45:13,482 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:45:15,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:15,654 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:45:15,654 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:45:17,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:17,813 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:45:17,813 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:45:19,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:19,971 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:45:19,971 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:45:21,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:22,134 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:45:22,134 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:45:24,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:24,285 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:45:24,285 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 17.1850001812. 2022-02-08 06:45:24,286 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:45:24,288 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:45:25,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:45:25,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:45:25,569 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:45:25,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:45:25,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:45:25,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:45:25,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:45:25,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:45:25,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:45:25,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:45:25,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:26,269 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:45:26,269 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.98399996758. 2022-02-08 06:45:26,270 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:45:26,272 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:45:27,150 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:45:27,423 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:45:27,565 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:45:27,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:45:27,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:45:27,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:45:27,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:45:27,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:45:27,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:45:27,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:45:27,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:28,440 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:45:28,440 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.17000007629. 2022-02-08 06:45:28,440 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:45:28,443 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:45:29,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:45:30,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:45:30,380 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:45:30,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:45:30,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:45:30,446 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:45:30,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:45:30,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:45:30,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:45:30,543 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:45:30,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:31,088 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:45:31,088 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.64700007439. 2022-02-08 06:45:31,088 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:45:31,092 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:45:31,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:45:31,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:45:31,973 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:45:31,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:45:31,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:45:32,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:45:32,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:45:32,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:45:32,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:45:32,104 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:45:32,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:32,703 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:45:32,704 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.61500000954. 2022-02-08 06:45:32,704 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:45:32,707 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:45:33,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:45:33,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:45:33,990 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:45:33,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:45:34,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:45:34,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:45:34,038 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:45:34,053 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:45:34,066 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:45:34,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:45:34,109 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:34,641 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:45:34,641 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.93700003624. 2022-02-08 06:45:34,641 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:45:34,644 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:45:34,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:45:35,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:45:35,676 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:45:35,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:45:35,765 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:45:35,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:45:35,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:45:36,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:45:36,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:45:36,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:45:36,487 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:36,795 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:45:36,796 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 06:45:38,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:39,299 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:45:39,299 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 06:45:41,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:41,490 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:45:41,490 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 06:45:43,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:44,144 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:45:44,144 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 06:45:46,145 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:46,697 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:45:46,697 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 06:45:48,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:48,851 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:45:48,852 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 14.2109999657. 2022-02-08 06:45:48,852 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:45:48,855 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 5 queries submitted, 0 failed, 9031 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:45:49,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:45:49,834 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:45:49,954 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:45:49,956 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:45:49,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:45:49,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:45:50,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:45:50,043 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:45:50,056 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:45:50,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:45:50,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:50,753 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:45:50,753 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.90100002289. 2022-02-08 06:45:50,753 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:45:50,756 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:45:51,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:45:51,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:45:51,677 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:45:51,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:45:51,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:45:51,714 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:45:51,729 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:45:51,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:45:51,760 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:45:51,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:45:51,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:45:52,634 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:45:52,635 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.88199996948. ############################################################ # # RAM FootPrint: 4389.23828125 # ############################################################ 2022-02-08 06:45:52,673 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 06:45:52,673 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 79 seconds. Reason: Iteration:9 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 9054 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:47:11,677 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 06:47:11,678 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 06:47:16,681 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:47:16,778 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:17,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:47:17,151 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:17,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:47:17,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:47:17,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:47:17,201 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:47:17,232 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:47:17,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:47:17,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:47:17,299 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:47:17,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:47:17,382 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['15842 memcached\n'], []) 2022-02-08 06:47:17,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 06:47:17,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:47:17,459 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['15842 memcached\n'], []) 2022-02-08 06:47:17,460 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:47:17,461 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:47:17,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:17,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:47:17,937 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:17,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:47:17,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:47:17,964 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:47:17,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:47:17,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:47:18,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:47:18,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:47:18,036 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:47:18,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:47:18,115 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['63465 memcached\n'], []) 2022-02-08 06:47:18,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 06:47:18,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:47:18,177 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['63465 memcached\n'], []) 2022-02-08 06:47:18,177 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:47:18,180 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:47:18,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:18,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:47:18,608 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:18,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:47:18,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:47:18,640 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:47:18,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:47:18,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:47:18,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:47:18,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:47:18,719 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:47:18,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:47:18,766 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['48701 memcached\n'], []) 2022-02-08 06:47:18,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 06:47:18,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:47:18,819 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['48701 memcached\n'], []) 2022-02-08 06:47:18,819 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:47:18,822 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:47:18,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:19,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:47:19,181 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:19,183 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:47:19,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:47:19,211 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:47:19,227 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:47:19,244 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:47:19,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:47:19,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:47:19,290 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:47:19,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:47:19,361 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['80398 memcached\n'], []) 2022-02-08 06:47:19,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 06:47:19,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:47:19,411 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['80398 memcached\n'], []) 2022-02-08 06:47:19,411 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:47:19,414 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:47:19,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:19,714 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:47:19,805 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:19,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:47:19,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:47:19,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:47:19,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:47:19,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:47:19,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:47:19,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:47:19,900 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:47:19,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:47:19,947 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['74250 memcached\n'], []) 2022-02-08 06:47:19,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 06:47:19,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:47:19,999 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['74250 memcached\n'], []) 2022-02-08 06:47:19,999 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:47:20,002 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:47:20,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:20,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:47:20,381 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:20,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:47:20,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:47:20,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:47:20,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:47:20,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:47:20,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:47:20,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:47:20,480 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:47:20,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:47:20,529 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['109233 memcached\n'], []) 2022-02-08 06:47:20,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 06:47:20,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:47:20,588 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['109233 memcached\n'], []) 2022-02-08 06:47:20,588 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:47:20,589 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:47:20,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:20,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:47:21,023 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:21,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:47:21,038 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:47:21,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:47:21,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:47:21,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:47:21,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:47:21,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:47:21,118 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:47:21,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:47:21,171 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['43077 memcached\n'], []) 2022-02-08 06:47:21,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 06:47:21,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:47:21,227 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['43077 memcached\n'], []) 2022-02-08 06:47:21,227 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:47:21,230 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:47:21,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:22,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:47:22,236 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:22,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:47:22,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:47:22,285 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:47:22,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:47:22,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:47:22,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:47:22,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:47:22,381 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:47:22,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:47:22,453 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['57964 memcached\n'], []) 2022-02-08 06:47:22,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 06:47:22,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:47:22,515 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['57964 memcached\n'], []) 2022-02-08 06:47:22,515 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:47:22,517 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:47:22,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:22,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:47:22,864 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:22,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:47:22,877 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:47:22,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:47:22,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:47:22,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:47:22,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:47:22,941 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:47:22,957 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:47:22,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:47:23,002 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['107359 memcached\n'], []) 2022-02-08 06:47:23,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 06:47:23,028 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:47:23,055 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['107359 memcached\n'], []) 2022-02-08 06:47:23,056 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:47:23,059 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:47:23,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:23,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:47:23,447 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:23,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:47:23,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:47:23,474 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:47:23,487 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:47:23,505 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:47:23,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:47:23,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:47:23,555 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:47:23,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:47:23,619 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['124015 memcached\n'], []) 2022-02-08 06:47:23,619 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 06:47:23,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:47:23,665 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['124015 memcached\n'], []) 2022-02-08 06:47:23,665 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:47:23,667 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:47:23,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:23,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:47:24,048 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:24,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:47:24,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:47:24,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:47:24,091 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:47:24,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:47:24,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:47:24,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:47:24,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:47:24,167 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 06:47:24,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:47:24,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:24,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:47:25,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:25,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:26,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:47:26,073 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 06:47:26,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:47:26,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:26,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:47:26,158 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 06:47:26,160 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:47:26,161 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:47:26,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:26,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:47:26,532 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:26,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:47:26,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:47:26,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:47:26,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:47:26,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:47:26,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:47:26,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:47:26,640 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:47:26,655 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 06:47:26,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:47:26,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:27,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:47:27,841 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:28,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:28,484 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:47:28,502 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 06:47:28,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:47:28,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:28,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:47:28,619 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 06:47:28,621 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:47:28,622 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:47:28,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:28,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:47:28,966 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:28,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:47:28,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:47:28,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:47:29,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:47:29,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:47:29,038 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:47:29,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:47:29,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:47:29,082 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 06:47:29,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:47:29,127 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:29,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:47:29,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:30,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:30,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:47:30,424 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 06:47:30,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:47:30,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:30,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:47:30,529 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 06:47:30,529 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:47:30,532 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 06:47:30,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:30,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 06:47:30,923 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:30,923 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 06:47:30,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 06:47:30,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 06:47:30,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 06:47:30,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 06:47:31,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 06:47:31,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 06:47:31,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:47:31,051 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 06:47:31,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:47:31,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:31,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:47:31,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:31,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:31,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:47:31,247 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 06:47:31,247 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:47:31,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:31,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:47:31,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:31,316 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:47:31,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:31,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:31,408 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:31,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:31,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:31,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:31,530 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:31,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:31,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:31,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:47:31,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:31,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:31,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:31,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:31,706 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:31,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:31,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:31,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:31,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:31,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:47:31,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:31,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:31,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:31,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:31,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:31,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:31,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:31,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:31,941 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:31,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:47:31,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:31,989 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:32,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:32,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:32,035 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:32,053 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:32,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:32,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:32,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:32,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:47:32,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:32,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:32,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:32,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:32,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:32,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:32,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:32,249 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:32,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:32,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:47:32,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:32,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:32,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:32,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:32,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:32,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:32,410 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:32,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:32,446 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 06:47:32,448 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:47:32,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:33,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:47:33,642 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:33,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:47:33,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:47:33,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:47:33,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:47:34,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:47:34,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:47:34,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:47:34,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:47:34,842 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 06:47:34,869 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:47:35,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:38,255 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:47:38,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:39,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:39,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:47:39,346 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 06:47:39,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:47:39,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:39,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:47:39,530 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 06:47:39,532 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:47:39,533 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 06:47:39,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:39,854 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 06:47:39,917 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:39,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 06:47:39,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 06:47:39,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 06:47:39,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 06:47:39,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 06:47:39,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 06:47:40,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 06:47:40,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:47:40,032 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 06:47:40,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:47:40,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:40,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:47:40,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:40,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:40,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:47:40,141 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 06:47:40,142 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:47:40,155 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:40,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:47:40,186 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:40,219 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:47:40,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:40,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:40,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:40,339 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:40,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:40,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:40,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:40,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:40,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:40,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:47:40,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:40,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:40,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:40,581 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:40,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:40,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:40,632 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:40,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:40,670 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:40,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:47:40,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:40,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:40,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:40,755 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:40,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:40,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:40,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:40,823 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:40,841 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:40,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:47:40,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:40,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:40,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:40,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:40,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:40,956 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:40,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:40,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:41,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:41,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:47:41,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:41,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:41,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:41,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:41,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:41,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:41,221 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:41,267 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:41,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:41,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:47:41,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:41,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:41,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:41,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:41,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:41,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:41,441 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:41,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:41,476 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 06:47:41,479 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:47:41,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:41,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:47:41,885 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:41,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:47:41,910 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:47:41,923 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:47:41,936 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:47:41,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:47:41,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:47:41,979 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:47:41,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:47:42,009 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 06:47:42,038 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:47:42,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:45,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:47:45,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:45,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:46,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:47:46,325 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 06:47:46,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:47:46,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:46,385 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:47:46,401 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 06:47:46,401 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:47:46,403 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:47:46,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:46,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:47:46,766 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:46,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:47:46,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:47:46,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:47:46,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:47:46,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:47:46,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:47:46,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:47:46,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:47:46,908 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 06:47:46,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:47:46,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:47,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:47:47,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:48,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:48,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:47:48,575 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 06:47:48,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:47:48,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:48,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:47:48,710 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 06:47:48,710 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:47:48,721 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 06:47:48,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:49,112 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 06:47:49,188 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:49,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 06:47:49,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 06:47:49,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 06:47:49,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 06:47:49,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 06:47:49,280 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 06:47:49,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 06:47:49,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:47:49,334 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 06:47:49,362 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:47:49,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:49,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:47:49,408 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:49,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:49,438 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:47:49,454 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 06:47:49,454 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:47:49,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 9073 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:47:49,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:47:49,505 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:49,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:47:49,571 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:49,599 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:49,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:49,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:49,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:49,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:49,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:49,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:49,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:49,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:47:49,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:49,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:49,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:49,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:49,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:49,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:50,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:50,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:50,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:50,066 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:47:50,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:50,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:50,127 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:50,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:50,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:50,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:50,198 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:50,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:50,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:50,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:47:50,280 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:50,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:50,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:50,339 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:50,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:50,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:50,410 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:50,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:50,456 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:50,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:47:50,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:50,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:50,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:50,543 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:50,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:50,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:50,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:50,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:50,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:50,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:47:50,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:50,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:50,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:50,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:50,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:50,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:50,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:50,823 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:50,842 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 06:47:50,844 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 06:47:50,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:51,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 06:47:51,206 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:51,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 06:47:51,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 06:47:51,232 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 06:47:51,244 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 06:47:51,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 06:47:51,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 06:47:51,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 06:47:51,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:47:51,328 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 06:47:51,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:47:51,377 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:51,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:47:51,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:51,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:51,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:47:51,446 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 06:47:51,446 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:47:51,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:51,474 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:47:51,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:51,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:47:51,548 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:51,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:51,607 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:51,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:51,665 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:51,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:51,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:51,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:47:51,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:51,816 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:47:51,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:51,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:51,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:51,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:51,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:51,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:51,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:51,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:47:51,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:51,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:47:52,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:52,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:52,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:52,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:52,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:52,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:52,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:52,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:47:52,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:52,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:47:52,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:52,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:52,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:52,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:52,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:52,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:52,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:52,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:47:52,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:52,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:47:52,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:52,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:52,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:52,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:52,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:52,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:52,397 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:52,415 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:47:52,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:52,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:47:52,464 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:52,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:52,497 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:52,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:52,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:52,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:52,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:52,579 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:47:52,595 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 06:47:52,598 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:47:52,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:52,918 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:47:53,016 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:53,017 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:47:53,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:47:53,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:47:53,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:47:53,084 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:47:53,099 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:47:53,117 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:47:53,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:47:53,165 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 06:47:53,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:47:53,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:54,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:47:55,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:55,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:56,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:47:56,061 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 06:47:56,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:47:56,081 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:56,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:47:56,154 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 06:47:56,154 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:47:56,157 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:47:56,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:47:56,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:47:56,614 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:47:56,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:47:56,632 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:47:56,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:47:56,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:47:56,706 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:47:56,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:47:56,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:47:56,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:47:57,026 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 06:47:57,053 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:47:57,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:57,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:47:58,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:58,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:47:58,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:47:58,926 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 06:47:58,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:47:59,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:47:59,204 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:47:59,359 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 06:47:59,359 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:47:59,361 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:48:00,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:48:00,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:48:00,561 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:48:00,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:48:00,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:48:00,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:48:00,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:48:00,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:48:00,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:48:00,711 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:48:00,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:48:00,755 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 06:48:00,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:48:00,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:48:01,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:48:01,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:48:01,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:48:02,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:48:02,216 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 06:48:02,216 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:48:02,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:48:02,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:48:02,290 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 06:48:02,290 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:48:02,292 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:48:02,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:48:02,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:48:02,740 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:48:02,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:48:02,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:48:02,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:48:02,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:48:02,816 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:48:02,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:48:02,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:48:02,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:48:02,908 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 06:48:03,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:48:03,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:48:03,961 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:48:04,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:48:04,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:48:05,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:48:05,322 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 06:48:05,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:48:05,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:48:05,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:48:05,420 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 06:48:05,421 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:48:05,424 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 06:48:05,505 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:48:05,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 06:48:05,815 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:48:05,816 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 06:48:05,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 06:48:05,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 06:48:05,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 06:48:05,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 06:48:05,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 06:48:05,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 06:48:05,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:48:05,948 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 06:48:05,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:48:05,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:48:06,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:48:06,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:48:06,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:48:06,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:48:06,091 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 06:48:06,091 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:48:06,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:48:06,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:48:06,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:48:06,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:48:06,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:48:06,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:48:06,270 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:48:06,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:48:06,331 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:48:06,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:48:06,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:48:06,426 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:48:06,454 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:48:06,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:48:06,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:48:06,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:48:06,525 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:48:06,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:48:06,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:48:06,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:48:06,596 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:48:06,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:48:06,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:48:06,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:48:06,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:48:06,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:48:06,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:48:06,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:48:06,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:48:06,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:48:06,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:48:06,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:48:06,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:48:06,877 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:48:06,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:48:06,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:48:06,928 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:48:06,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:48:06,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:48:06,979 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:48:06,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:48:07,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:48:07,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:48:07,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:48:07,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:48:07,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:48:07,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:48:07,117 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:48:07,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:48:07,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:48:07,165 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:48:07,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:48:07,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:48:07,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:48:07,239 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:48:07,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:48:07,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:48:07,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:48:07,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:48:07,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:48:07,349 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:48:07,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:48:07,394 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 06:48:07,394 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:48:07,398 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:48:07,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:48:07,711 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:48:07,798 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:48:07,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:48:07,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:48:07,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:48:07,862 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:48:07,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:48:07,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:48:07,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:48:07,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:08,480 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:48:08,482 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.08599996567. 2022-02-08 06:48:08,482 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:48:08,483 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:48:08,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:48:08,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:48:09,089 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:48:09,091 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:48:09,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:48:09,165 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:48:09,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:48:09,244 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:48:09,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:48:09,290 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:48:09,306 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:09,601 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:09,601 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:11,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:11,815 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:11,815 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:13,816 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:14,017 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:14,017 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:16,017 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:16,263 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:16,263 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:18,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:18,428 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:18,430 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:20,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:20,648 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:20,648 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:22,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:22,842 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:22,844 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:24,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:25,036 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:25,038 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:27,038 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:27,249 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:27,250 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:29,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:29,502 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:29,503 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:31,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:31,713 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:31,713 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:33,714 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:33,970 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:33,970 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:35,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:36,171 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:36,171 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:38,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:38,377 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:38,377 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:40,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:40,585 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:40,585 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:42,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:42,756 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:42,757 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:44,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:44,908 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:44,910 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:46,910 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:47,075 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:47,075 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:49,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:49,273 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:49,273 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 9076 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:48:51,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:51,428 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:51,428 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:53,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:53,579 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:53,581 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:55,581 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:55,733 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:55,734 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:57,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:48:57,888 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:48:57,888 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:48:59,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:49:00,036 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:49:00,036 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:49:02,038 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:49:02,180 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:49:02,180 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:49:04,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:49:04,319 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:49:04,321 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 55.8389999866. 2022-02-08 06:49:04,321 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:49:04,322 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:49:05,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:49:05,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:49:05,680 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:49:05,681 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:49:05,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:49:05,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:49:05,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:49:05,762 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:49:05,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:49:05,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:49:05,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:49:06,325 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:49:06,326 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.00599980354. 2022-02-08 06:49:06,326 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:49:06,328 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:49:07,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:49:07,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:49:07,522 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:49:07,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:49:07,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:49:07,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:49:07,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:49:07,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:49:07,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:49:07,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:49:07,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:49:08,289 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:49:08,290 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.96300005913. 2022-02-08 06:49:08,290 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:49:08,292 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:49:09,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:49:09,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:49:10,134 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:49:10,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:49:10,160 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:49:10,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:49:10,244 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:49:10,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:49:10,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:49:10,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:49:10,385 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:49:10,947 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:49:10,956 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.66400003433. 2022-02-08 06:49:10,956 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:49:10,959 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:49:11,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:49:12,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:49:12,576 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:49:12,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:49:12,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:49:12,614 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:49:12,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:49:12,683 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:49:12,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:49:12,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:49:12,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:49:14,056 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:49:14,058 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.10199999809. 2022-02-08 06:49:14,058 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:49:14,059 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:49:14,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:49:14,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:49:15,096 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:49:15,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:49:15,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:49:15,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:49:15,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:49:15,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:49:15,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:49:15,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:49:15,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:49:15,677 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:49:15,677 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.61999988556. 2022-02-08 06:49:15,678 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:49:15,680 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:49:16,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:49:16,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:49:16,733 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:49:16,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:49:16,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:49:16,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:49:16,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:49:16,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:49:16,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:49:16,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:49:16,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:49:17,653 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:49:17,654 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.97600007057. 2022-02-08 06:49:17,654 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:49:17,655 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:49:18,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:49:19,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:49:19,259 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:49:19,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:49:19,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:49:19,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:49:19,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:49:19,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:49:19,339 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:49:19,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:49:19,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:49:20,767 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:49:20,767 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.1139998436. 2022-02-08 06:49:20,769 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:49:20,769 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:49:21,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:49:21,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:49:21,625 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:49:21,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:49:21,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:49:21,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:49:21,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:49:21,688 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:49:21,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:49:21,729 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:49:21,744 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:49:22,319 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:49:22,319 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.55200004578. ############################################################ # # RAM FootPrint: 4379.80078125 # ############################################################ 2022-02-08 06:49:22,355 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 06:49:22,355 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 84 seconds. Reason: Iteration:10 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 9096 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:50:46,362 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 06:50:46,362 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 5 queries submitted, 0 failed, 9121 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:50:51,365 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:50:51,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:50:51,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:50:51,911 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:50:51,913 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:50:51,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:50:51,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:50:51,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:50:52,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:50:52,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:50:52,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:50:52,059 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:50:52,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:50:52,115 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['16498 memcached\n'], []) 2022-02-08 06:50:52,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 06:50:52,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:50:52,178 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['16498 memcached\n'], []) 2022-02-08 06:50:52,180 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:50:52,183 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:50:52,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:50:52,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:50:52,651 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:50:52,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:50:52,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:50:52,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:50:52,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:50:52,711 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:50:52,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:50:52,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:50:52,750 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:50:52,752 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:50:52,821 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['64243 memcached\n'], []) 2022-02-08 06:50:52,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 06:50:52,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:50:52,867 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['64243 memcached\n'], []) 2022-02-08 06:50:52,867 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:50:52,869 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:50:52,956 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:50:53,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:50:53,256 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:50:53,257 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:50:53,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:50:53,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:50:53,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:50:53,321 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:50:53,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:50:53,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:50:53,374 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:50:53,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:50:53,443 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['49445 memcached\n'], []) 2022-02-08 06:50:53,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 06:50:53,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:50:53,496 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['49445 memcached\n'], []) 2022-02-08 06:50:53,496 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:50:53,499 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:50:53,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:50:53,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:50:53,894 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:50:53,895 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:50:53,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:50:53,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:50:53,936 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:50:53,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:50:53,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:50:53,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:50:53,990 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:50:53,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:50:54,042 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['81061 memcached\n'], []) 2022-02-08 06:50:54,043 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 06:50:54,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:50:54,089 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['81061 memcached\n'], []) 2022-02-08 06:50:54,089 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:50:54,094 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:50:54,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:50:54,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:50:54,473 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:50:54,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:50:54,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:50:54,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:50:54,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:50:54,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:50:54,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:50:54,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:50:54,578 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:50:54,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:50:54,628 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['74904 memcached\n'], []) 2022-02-08 06:50:54,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 06:50:54,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:50:54,683 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['74904 memcached\n'], []) 2022-02-08 06:50:54,684 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:50:54,696 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:50:54,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:50:55,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:50:55,069 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:50:55,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:50:55,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:50:55,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:50:55,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:50:55,130 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:50:55,142 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:50:55,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:50:55,173 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:50:55,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:50:55,250 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['109885 memcached\n'], []) 2022-02-08 06:50:55,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 06:50:55,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:50:55,295 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['109885 memcached\n'], []) 2022-02-08 06:50:55,295 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:50:55,298 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:50:55,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:50:55,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:50:55,658 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:50:55,660 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:50:55,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:50:55,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:50:55,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:50:55,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:50:55,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:50:55,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:50:55,749 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:50:55,749 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:50:55,785 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['43731 memcached\n'], []) 2022-02-08 06:50:55,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 06:50:55,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:50:55,829 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['43731 memcached\n'], []) 2022-02-08 06:50:55,829 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:50:55,832 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:50:55,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:50:56,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:50:56,345 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:50:56,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:50:56,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:50:56,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:50:56,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:50:56,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:50:56,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:50:56,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:50:56,447 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:50:56,448 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:50:56,509 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['58691 memcached\n'], []) 2022-02-08 06:50:56,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 06:50:56,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:50:56,568 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['58691 memcached\n'], []) 2022-02-08 06:50:56,568 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:50:56,569 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:50:56,642 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:50:56,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:50:56,892 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:50:56,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:50:56,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:50:56,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:50:56,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:50:56,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:50:56,961 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:50:56,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:50:56,986 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:50:56,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:50:57,039 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['108025 memcached\n'], []) 2022-02-08 06:50:57,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 06:50:57,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:50:57,085 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['108025 memcached\n'], []) 2022-02-08 06:50:57,085 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:50:57,086 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:50:57,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:50:57,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:50:57,460 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:50:57,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:50:57,474 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:50:57,487 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:50:57,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:50:57,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:50:57,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:50:57,543 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:50:57,558 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:50:57,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:50:57,609 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['124686 memcached\n'], []) 2022-02-08 06:50:57,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 06:50:57,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:50:57,657 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['124686 memcached\n'], []) 2022-02-08 06:50:57,657 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:50:57,660 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:50:57,749 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:50:57,964 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:50:58,029 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:50:58,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:50:58,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:50:58,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:50:58,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:50:58,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:50:58,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:50:58,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:50:58,140 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:50:58,168 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 06:50:58,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:50:58,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:00,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:51:00,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:01,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:01,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:51:01,352 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 06:51:01,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:51:01,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:01,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:51:01,451 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 06:51:01,453 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:51:01,454 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:51:01,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:51:01,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:51:01,848 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:51:01,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:51:01,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:51:01,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:51:01,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:51:01,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:51:01,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:51:01,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:51:01,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:51:01,967 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 06:51:01,997 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:51:02,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:05,117 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:51:05,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:05,997 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:06,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:51:06,311 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 06:51:06,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:51:06,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:06,387 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:51:06,398 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 06:51:06,400 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:51:06,460 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:51:06,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:51:06,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:51:06,796 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:51:06,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:51:06,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:51:06,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:51:06,841 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:51:06,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:51:06,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:51:06,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:51:06,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:51:06,911 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 06:51:06,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:51:06,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:10,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:51:11,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:11,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:11,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:51:11,703 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 06:51:11,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:51:11,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:11,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:51:11,792 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 06:51:11,792 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:51:11,796 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 06:51:11,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:51:12,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 06:51:12,174 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:51:12,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 06:51:12,193 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 06:51:12,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 06:51:12,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 06:51:12,239 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 06:51:12,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 06:51:12,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 06:51:12,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:51:12,328 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 06:51:12,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:51:12,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:12,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:51:12,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:12,428 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:12,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:51:12,457 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 06:51:12,459 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:51:12,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:12,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:51:12,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:12,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:51:12,571 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:12,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:12,632 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:12,660 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:12,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:12,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:12,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:12,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:12,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:12,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:51:12,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:12,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:12,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:12,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:12,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:12,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:12,964 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:12,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:13,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:13,017 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:51:13,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:13,053 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:13,071 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:13,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:13,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:13,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:13,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:13,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:13,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:13,191 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:51:13,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:13,227 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:13,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:13,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:13,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:13,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:13,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:13,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:13,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:13,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:51:13,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:13,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:13,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:13,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:13,438 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:13,454 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:13,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:13,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:13,505 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:13,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:51:13,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:13,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:13,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:13,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:13,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:13,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:13,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:13,665 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:13,683 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 06:51:13,686 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:51:13,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:51:14,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:51:14,098 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:51:14,099 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:51:14,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:51:14,145 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:51:14,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:51:14,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:51:14,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:51:14,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:51:14,224 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:51:14,240 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 06:51:14,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:51:14,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:14,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:51:15,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:15,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:15,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:51:15,500 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 06:51:15,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:51:15,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:15,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:51:15,589 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 06:51:15,591 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:51:15,592 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 06:51:15,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:51:15,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 06:51:15,963 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:51:15,964 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 06:51:15,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 06:51:15,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 06:51:16,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 06:51:16,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 06:51:16,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 06:51:16,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 06:51:16,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:51:16,073 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 06:51:16,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:51:16,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:16,130 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:51:16,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:16,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:16,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:51:16,186 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 06:51:16,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:51:16,198 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:16,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:51:16,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:16,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:51:16,283 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:16,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:16,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:16,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:16,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:16,428 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:16,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:16,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:16,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:16,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:51:16,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:16,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:16,584 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:16,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:16,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:16,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:16,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:16,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:16,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:16,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:51:16,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:16,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:16,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:16,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:16,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:16,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:16,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:16,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:16,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:16,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:51:16,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:16,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:16,933 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:16,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:16,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:16,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:16,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:17,015 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:17,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:17,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:51:17,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:17,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:17,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:17,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:17,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:17,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:17,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:17,183 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:17,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:17,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:51:17,234 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:17,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:17,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:17,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:17,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:17,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:17,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:17,362 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:17,380 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 06:51:17,382 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:51:17,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:51:17,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:51:17,732 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:51:17,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:51:17,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:51:17,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:51:17,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:51:17,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:51:17,801 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:51:17,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:51:17,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:51:17,844 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 06:51:17,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:51:17,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:19,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:51:19,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:20,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:20,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:51:20,424 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 06:51:20,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:51:20,438 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:20,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:51:20,503 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 06:51:20,503 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:51:20,506 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:51:20,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:51:20,795 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:51:20,855 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:51:20,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:51:20,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:51:20,890 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:51:20,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:51:20,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:51:20,941 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:51:20,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:51:20,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:51:20,990 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 06:51:21,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:51:21,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:25,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:51:26,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:26,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:26,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:51:26,898 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 06:51:26,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:51:26,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:26,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:51:27,000 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 06:51:27,000 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:51:27,003 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 06:51:27,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:51:27,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 06:51:27,371 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:51:27,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 06:51:27,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 06:51:27,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 06:51:27,428 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 06:51:27,448 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 06:51:27,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 06:51:27,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 06:51:27,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:51:27,519 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 06:51:27,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:51:27,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:27,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:51:27,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:27,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:27,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:51:27,637 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 06:51:27,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:51:27,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:27,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:51:27,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:27,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:51:27,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:27,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:27,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:27,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:27,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:27,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:27,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:27,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:28,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:28,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:51:28,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:28,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:28,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:28,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:28,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:28,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:28,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:28,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:28,211 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:28,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:51:28,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:28,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:28,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:28,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:28,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:28,349 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:28,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:28,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:28,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:28,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:51:28,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:28,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:28,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:28,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:28,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:28,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:28,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:28,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:28,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:28,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:51:28,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:28,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:28,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:28,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:28,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:28,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:28,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:28,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:28,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:28,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:51:28,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:28,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:28,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:28,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:28,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:28,918 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:28,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:28,961 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:28,980 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 06:51:28,986 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 06:51:29,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:51:29,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 06:51:29,381 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:51:29,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 06:51:29,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 06:51:29,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 06:51:29,426 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 06:51:29,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 06:51:29,459 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 06:51:29,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 06:51:29,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:51:29,516 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 06:51:29,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:51:29,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:29,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:51:29,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:29,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:29,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:51:29,635 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 06:51:29,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:51:29,650 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:29,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:51:29,683 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:29,714 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:51:29,749 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:29,778 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:29,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:29,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:29,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:29,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:29,941 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:29,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:30,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:30,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:51:30,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:30,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:30,081 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:30,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:30,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:30,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:30,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:30,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:30,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:30,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:51:30,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:30,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:30,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:30,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:30,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:30,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:30,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:30,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:30,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:30,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:51:30,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:30,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:30,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:30,484 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:30,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:30,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:30,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:30,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:30,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:30,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:51:30,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:30,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:30,632 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:30,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:30,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:30,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:30,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:30,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:30,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:30,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:51:30,795 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:30,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:30,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:30,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:30,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:30,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:30,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:30,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:30,941 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 06:51:30,944 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:51:31,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:51:31,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:51:31,621 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:51:31,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:51:31,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:51:31,650 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:51:31,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:51:31,681 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:51:31,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:51:31,716 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:51:31,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:51:31,756 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 06:51:31,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:51:31,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:35,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:51:36,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:36,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:37,028 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:51:37,151 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 06:51:37,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:51:37,178 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:37,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:51:37,276 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 06:51:37,276 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:51:37,279 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:51:37,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:51:37,642 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:51:37,726 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:51:37,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:51:37,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:51:37,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:51:37,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:51:37,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:51:37,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:51:37,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:51:37,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:51:37,898 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 06:51:37,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:51:37,948 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:38,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:51:39,155 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:39,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:39,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:51:39,743 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 06:51:39,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:51:39,760 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:39,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:51:39,825 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 06:51:39,825 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:51:39,828 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:51:40,410 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:51:40,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:51:40,910 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:51:40,910 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:51:40,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:51:40,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:51:40,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:51:40,997 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:51:41,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:51:41,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:51:41,066 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:51:41,099 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 06:51:41,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:51:41,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:41,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:51:42,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:42,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:42,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:51:42,895 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 06:51:42,895 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:51:42,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:42,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:51:42,970 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 06:51:42,970 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:51:42,973 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:51:43,071 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:51:43,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:51:43,368 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:51:43,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:51:43,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:51:43,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:51:43,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:51:43,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:51:43,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:51:43,464 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:51:43,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:51:43,496 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 06:51:43,525 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:51:43,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:45,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:51:46,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:46,862 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:47,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:51:47,217 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 06:51:47,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:51:47,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:47,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:51:47,276 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 06:51:47,276 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:51:47,278 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 06:51:47,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:51:47,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 06:51:47,671 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:51:47,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 06:51:47,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 06:51:47,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 06:51:47,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 06:51:47,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 06:51:47,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 06:51:47,762 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 06:51:47,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:51:47,792 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 06:51:47,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:51:47,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:47,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:51:47,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:47,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:51:47,895 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:51:47,911 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 06:51:47,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:51:47,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:51:47,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:51:47,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:47,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:51:48,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:48,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:48,084 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:48,119 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:48,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:48,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:48,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:48,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:51:48,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:48,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:51:48,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:48,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:48,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:48,377 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:48,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:48,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:48,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:48,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:51:48,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:48,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:51:48,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:48,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:48,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:48,571 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:48,586 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:48,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:48,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:48,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:51:48,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:48,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:51:48,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:48,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:48,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:48,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:48,752 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:48,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:48,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:48,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:51:48,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:48,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:51:48,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:48,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:48,890 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:48,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:48,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:48,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:48,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:48,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:51:48,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:49,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:51:49,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:49,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:49,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:49,091 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:49,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:49,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:49,142 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:49,160 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:51:49,177 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 06:51:49,177 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:51:49,180 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:51:49,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 9129 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:51:49,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:51:49,585 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:51:49,586 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:51:49,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:51:49,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:51:49,630 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:51:49,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:51:49,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:51:49,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:51:49,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:51:50,180 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:51:50,188 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.01099991798. 2022-02-08 06:51:50,188 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:51:50,190 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:51:50,303 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:51:50,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:51:50,661 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:51:50,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:51:50,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:51:50,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:51:50,714 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:51:50,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:51:50,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:51:50,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:51:50,798 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:51:51,040 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:51:51,040 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:51:53,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:51:53,223 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:51:53,223 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:51:55,224 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:51:55,381 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:51:55,381 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:51:57,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:51:57,578 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:51:57,578 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:51:59,579 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:51:59,793 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:51:59,793 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:52:01,795 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:02,030 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:52:02,030 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:52:04,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:04,198 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:52:04,200 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:52:06,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:06,486 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:52:06,486 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:52:08,487 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:08,789 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:52:08,789 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:52:10,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:10,953 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:52:10,953 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:52:12,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:13,194 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:52:13,194 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:52:15,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:15,374 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:52:15,374 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:52:17,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:17,529 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:52:17,529 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:52:19,530 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:19,696 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:52:19,696 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:52:21,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:21,878 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:52:21,878 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:52:23,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:24,038 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:52:24,038 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:52:26,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:26,193 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:52:26,193 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:52:28,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:28,476 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:52:28,476 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:52:30,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:30,628 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:52:30,628 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:52:32,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:32,790 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:52:32,790 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:52:34,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:34,967 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:52:34,967 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 44.7790000439. 2022-02-08 06:52:34,969 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:52:34,970 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:52:35,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:52:35,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:52:36,003 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:52:36,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:52:36,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:52:36,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:52:36,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:52:36,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:52:36,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:52:36,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:52:36,196 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:36,786 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:52:36,788 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 06:52:38,788 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:38,950 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:52:38,950 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.98199987411. 2022-02-08 06:52:38,950 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:52:38,953 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:52:40,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:52:40,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:52:40,957 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:52:40,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:52:40,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:52:41,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:52:41,017 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:52:41,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:52:41,058 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:52:41,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:52:41,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:41,901 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:52:41,901 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.95000004768. 2022-02-08 06:52:41,901 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:52:41,903 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:52:42,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:52:43,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:52:43,302 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:52:43,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:52:43,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:52:43,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:52:43,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:52:43,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:52:43,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:52:43,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:52:43,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:44,147 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:52:44,147 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.24500012398. 2022-02-08 06:52:44,148 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:52:44,151 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:52:44,918 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:52:45,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:52:45,335 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:52:45,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:52:45,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:52:45,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:52:45,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:52:45,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:52:45,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:52:45,448 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:52:45,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:46,038 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:52:46,049 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.90100002289. 2022-02-08 06:52:46,049 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:52:46,051 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:52:46,630 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:52:46,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:52:47,026 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:52:47,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:52:47,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:52:47,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:52:47,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:52:47,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:52:47,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:52:47,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:52:47,145 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:47,607 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:52:47,608 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.55800008774. 2022-02-08 06:52:47,608 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:52:47,609 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:52:48,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:52:48,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:52:48,417 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:52:48,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:52:48,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:52:48,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:52:48,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:52:48,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:52:48,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:52:48,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:52:48,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:48,855 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:52:48,855 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.24699997902. 2022-02-08 06:52:48,855 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:52:48,857 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:52:49,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 9130 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:52:49,642 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:52:49,729 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:52:49,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:52:49,744 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:52:49,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:52:49,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:52:49,801 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:52:49,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:52:49,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:52:49,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:50,404 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:52:50,404 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.54999995232. 2022-02-08 06:52:50,404 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:52:50,407 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:52:51,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:52:51,573 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:52:51,727 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:52:51,727 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:52:51,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:52:51,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:52:51,798 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:52:51,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:52:51,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:52:51,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:52:51,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:52:52,578 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:52:52,579 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.1740000248. ############################################################ # # RAM FootPrint: 4380.08984375 # ############################################################ 2022-02-08 06:52:52,851 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 06:52:52,852 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 110 seconds. Reason: Iteration:11 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 9156 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:54:42,858 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 06:54:42,858 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 06:54:47,861 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:54:47,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:54:48,221 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:54:48,338 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:54:48,339 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:54:48,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:54:48,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:54:48,410 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:54:48,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:54:48,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:54:48,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:54:48,503 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:54:48,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:54:48,565 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['17205 memcached\n'], []) 2022-02-08 06:54:48,566 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 06:54:48,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:54:48,615 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['17205 memcached\n'], []) 2022-02-08 06:54:48,615 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:54:48,618 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:54:49,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:54:49,408 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 5 queries submitted, 0 failed, 9181 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:54:49,494 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:54:49,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:54:49,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:54:49,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:54:49,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:54:49,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:54:49,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:54:49,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:54:49,618 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:54:49,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:54:49,673 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['65206 memcached\n'], []) 2022-02-08 06:54:49,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 06:54:49,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:54:49,719 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['65206 memcached\n'], []) 2022-02-08 06:54:49,719 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:54:49,723 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:54:49,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:54:50,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:54:50,115 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:54:50,117 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:54:50,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:54:50,147 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:54:50,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:54:50,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:54:50,196 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:54:50,211 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:54:50,226 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:54:50,227 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:54:50,279 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['50251 memcached\n'], []) 2022-02-08 06:54:50,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 06:54:50,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:54:50,332 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['50251 memcached\n'], []) 2022-02-08 06:54:50,332 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:54:50,335 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:54:50,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:54:50,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:54:50,723 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:54:50,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:54:50,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:54:50,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:54:50,767 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:54:50,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:54:50,798 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:54:50,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:54:50,825 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:54:50,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:54:50,875 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['81766 memcached\n'], []) 2022-02-08 06:54:50,877 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 06:54:50,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:54:50,924 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['81766 memcached\n'], []) 2022-02-08 06:54:50,924 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:54:50,927 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:54:51,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:54:51,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:54:51,302 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:54:51,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:54:51,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:54:51,329 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:54:51,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:54:51,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:54:51,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:54:51,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:54:51,403 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:54:51,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:54:51,466 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['75609 memcached\n'], []) 2022-02-08 06:54:51,466 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 06:54:51,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:54:51,530 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['75609 memcached\n'], []) 2022-02-08 06:54:51,530 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:54:51,533 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:54:51,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:54:51,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:54:51,923 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:54:51,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:54:51,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:54:51,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:54:51,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:54:51,989 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:54:52,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:54:52,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:54:52,026 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:54:52,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:54:52,069 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['110589 memcached\n'], []) 2022-02-08 06:54:52,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 06:54:52,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:54:52,118 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['110589 memcached\n'], []) 2022-02-08 06:54:52,118 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:54:52,121 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:54:52,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:54:52,418 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:54:52,494 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:54:52,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:54:52,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:54:52,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:54:52,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:54:52,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:54:52,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:54:52,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:54:52,592 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:54:52,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:54:52,638 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['44443 memcached\n'], []) 2022-02-08 06:54:52,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 06:54:52,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:54:52,691 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['44443 memcached\n'], []) 2022-02-08 06:54:52,691 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:54:52,694 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:54:52,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:54:53,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:54:53,075 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:54:53,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:54:53,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:54:53,104 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:54:53,119 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:54:53,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:54:53,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:54:53,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:54:53,177 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:54:53,178 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:54:53,269 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['59392 memcached\n'], []) 2022-02-08 06:54:53,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 06:54:53,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:54:53,322 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['59392 memcached\n'], []) 2022-02-08 06:54:53,322 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:54:53,326 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:54:53,474 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:54:53,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:54:53,726 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:54:53,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:54:53,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:54:53,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:54:53,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:54:53,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:54:53,795 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:54:53,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:54:53,823 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:54:53,823 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:54:53,867 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['108737 memcached\n'], []) 2022-02-08 06:54:53,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 06:54:53,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:54:53,911 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['108737 memcached\n'], []) 2022-02-08 06:54:53,911 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:54:53,914 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:54:53,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:54:54,219 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:54:54,282 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:54:54,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:54:54,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:54:54,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:54:54,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:54:54,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:54:54,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:54:54,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:54:54,384 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:54:54,385 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:54:54,469 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['125399 memcached\n'], []) 2022-02-08 06:54:54,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 06:54:54,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:54:54,525 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['125399 memcached\n'], []) 2022-02-08 06:54:54,525 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:54:54,533 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:54:54,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:54:54,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:54:54,940 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:54:54,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:54:54,956 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:54:54,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:54:54,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:54:55,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:54:55,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:54:55,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:54:55,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:54:55,078 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 06:54:55,119 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:54:55,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:54:57,778 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:54:58,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:54:58,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:54:59,068 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:54:59,084 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 06:54:59,084 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:54:59,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:54:59,163 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:54:59,180 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 06:54:59,180 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:54:59,186 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:54:59,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:54:59,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:54:59,568 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:54:59,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:54:59,584 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:54:59,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:54:59,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:54:59,630 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:54:59,642 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:54:59,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:54:59,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:54:59,687 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 06:54:59,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:54:59,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:01,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:55:02,362 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:02,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:02,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:55:03,019 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 06:55:03,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:55:03,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:03,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:55:03,112 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 06:55:03,112 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:55:03,121 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:55:03,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:55:03,413 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:55:03,476 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:55:03,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:55:03,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:55:03,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:55:03,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:55:03,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:55:03,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:55:03,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:55:03,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:55:03,595 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 06:55:03,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:55:03,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:04,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:55:04,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:04,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:05,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:55:05,092 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 06:55:05,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:55:05,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:05,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:55:05,210 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 06:55:05,211 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:55:05,214 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 06:55:05,349 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:55:05,589 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 06:55:05,660 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:55:05,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 06:55:05,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 06:55:05,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 06:55:05,706 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 06:55:05,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 06:55:05,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 06:55:05,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 06:55:05,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:55:05,786 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 06:55:05,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:55:05,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:05,859 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:55:05,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:05,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:05,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:55:05,921 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 06:55:05,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:55:05,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:05,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:55:05,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:06,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:55:06,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:06,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:06,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:06,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:06,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:06,196 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:06,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:06,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:06,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:06,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:55:06,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:06,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:06,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:06,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:06,405 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:06,423 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:06,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:06,471 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:06,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:06,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:55:06,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:06,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:06,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:06,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:06,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:06,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:06,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:06,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:06,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:06,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:55:06,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:06,721 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:06,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:06,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:06,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:06,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:06,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:06,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:06,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:06,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:55:06,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:06,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:06,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:06,933 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:06,948 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:06,964 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:06,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:06,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:07,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:07,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:55:07,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:07,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:07,081 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:07,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:07,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:07,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:07,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:07,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:07,180 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 06:55:07,184 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:55:07,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:55:07,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:55:08,000 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:55:08,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:55:08,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:55:08,109 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:55:08,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:55:08,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:55:08,224 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:55:08,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:55:08,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:55:08,361 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 06:55:08,397 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:55:08,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:08,885 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:55:09,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:09,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:09,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:55:09,938 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 06:55:09,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:55:09,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:10,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:55:10,121 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 06:55:10,122 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:55:10,125 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 06:55:10,227 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:55:10,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 06:55:10,528 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:55:10,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 06:55:10,543 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 06:55:10,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 06:55:10,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 06:55:10,589 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 06:55:10,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 06:55:10,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 06:55:10,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:55:10,651 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 06:55:10,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:55:10,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:10,716 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:55:10,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:10,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:10,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:55:10,773 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 06:55:10,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:55:10,788 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:10,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:55:10,816 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:10,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:55:10,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:10,913 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:10,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:10,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:11,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:11,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:11,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:11,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:11,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:11,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:55:11,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:11,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:11,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:11,221 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:11,239 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:11,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:11,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:11,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:11,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:11,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:55:11,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:11,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:11,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:11,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:11,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:11,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:11,438 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:11,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:11,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:11,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:55:11,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:11,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:11,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:11,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:11,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:11,607 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:11,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:11,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:11,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:11,670 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:55:11,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:11,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:11,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:11,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:11,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:11,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:11,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:11,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:11,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:11,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:55:11,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:11,862 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:11,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:11,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:11,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:11,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:12,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:12,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:12,051 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 06:55:12,053 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:55:12,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:55:12,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:55:12,404 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:55:12,405 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:55:12,418 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:55:12,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:55:12,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:55:12,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:55:12,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:55:12,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:55:12,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:55:12,546 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 06:55:12,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:55:12,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:16,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:55:17,466 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:17,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:18,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:55:18,171 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 06:55:18,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:55:18,186 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:18,234 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:55:18,259 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 06:55:18,259 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:55:18,263 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:55:18,341 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:55:18,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:55:18,658 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:55:18,660 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:55:18,678 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:55:18,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:55:18,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:55:18,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:55:18,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:55:18,765 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:55:18,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:55:18,799 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 06:55:18,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:55:18,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:19,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:55:20,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:20,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:20,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:55:20,831 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 06:55:20,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:55:20,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:20,948 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:55:20,973 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 06:55:20,974 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:55:20,977 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 06:55:21,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:55:21,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 06:55:21,414 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:55:21,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 06:55:21,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 06:55:21,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 06:55:21,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 06:55:21,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 06:55:21,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 06:55:21,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 06:55:21,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:55:21,575 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 06:55:21,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:55:21,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:21,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:55:21,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:21,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:21,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:55:21,729 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 06:55:21,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:55:21,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:21,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:55:21,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:21,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:55:21,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:21,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:21,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:21,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:22,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:22,068 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:22,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:22,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:22,183 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:22,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:55:22,227 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:22,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:22,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:22,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:22,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:22,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:22,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:22,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:22,466 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:22,487 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:55:22,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:22,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:22,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:22,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:22,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:22,624 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:22,645 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:22,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:22,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:22,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:55:22,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:22,744 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:22,765 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:22,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:22,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:22,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:22,849 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:22,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:22,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:22,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:55:22,936 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:22,956 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:22,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:22,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:23,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:23,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:23,056 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:23,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:23,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:23,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:55:23,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:23,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:23,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:23,219 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:23,239 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:23,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:23,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:23,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:23,318 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 06:55:23,322 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 06:55:23,410 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:55:23,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 06:55:23,687 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:55:23,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 06:55:23,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 06:55:23,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 06:55:23,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 06:55:23,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 06:55:23,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 06:55:23,770 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 06:55:23,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:55:23,801 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 06:55:23,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:55:23,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:23,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:55:23,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:23,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:23,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:55:23,915 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 06:55:23,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:55:23,936 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:23,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:55:23,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:24,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:55:24,038 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:24,066 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:24,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:24,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:24,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:24,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:24,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:24,290 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:24,321 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:24,339 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:55:24,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:24,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:24,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:24,405 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:24,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:24,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:24,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:24,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:24,484 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:24,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:55:24,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:24,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:24,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:24,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:24,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:24,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:24,607 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:24,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:24,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:24,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:55:24,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:24,681 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:24,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:24,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:24,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:24,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:24,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:24,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:24,780 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:24,795 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:55:24,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:24,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:24,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:24,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:24,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:24,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:24,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:24,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:24,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:24,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:55:24,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:24,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:25,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:25,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:25,043 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:25,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:25,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:25,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:25,111 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 06:55:25,114 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:55:25,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:55:25,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:55:25,789 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:55:25,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:55:25,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:55:25,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:55:25,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:55:25,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:55:26,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:55:26,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:55:26,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:55:26,382 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 06:55:26,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:55:26,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:28,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:55:28,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:29,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:29,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:55:29,457 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 06:55:29,459 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:55:29,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:29,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:55:29,551 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 06:55:29,552 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:55:29,555 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:55:29,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:55:29,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:55:30,048 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:55:30,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:55:30,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:55:30,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:55:30,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:55:30,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:55:30,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:55:30,150 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:55:30,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:55:30,196 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 06:55:30,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:55:30,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:32,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:55:32,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:32,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:33,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:55:33,072 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 06:55:33,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:55:33,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:33,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:55:33,147 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 06:55:33,148 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:55:33,150 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:55:33,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:55:33,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:55:34,063 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:55:34,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:55:34,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:55:34,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:55:34,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:55:34,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:55:34,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:55:34,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:55:34,196 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:55:34,210 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 06:55:34,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:55:34,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:37,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:55:38,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:38,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:38,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:55:38,622 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 06:55:38,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:55:38,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:38,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:55:38,687 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 06:55:38,687 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:55:38,690 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:55:38,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:55:39,005 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:55:39,068 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:55:39,068 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:55:39,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:55:39,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:55:39,109 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:55:39,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:55:39,147 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:55:39,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:55:39,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:55:39,210 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 06:55:39,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:55:39,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:40,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:55:40,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:41,193 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:41,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:55:41,528 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 06:55:41,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:55:41,543 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:41,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:55:41,607 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 06:55:41,608 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:55:41,609 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 06:55:41,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:55:41,918 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 06:55:42,019 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:55:42,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 06:55:42,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 06:55:42,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 06:55:42,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 06:55:42,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 06:55:42,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 06:55:42,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 06:55:42,117 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:55:42,132 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 06:55:42,170 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:55:42,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:42,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:55:42,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:42,232 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:55:42,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:55:42,263 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 06:55:42,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:55:42,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:55:42,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:55:42,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:42,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:55:42,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:42,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:42,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:42,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:42,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:42,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:42,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:42,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:55:42,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:42,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:55:42,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:42,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:42,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:42,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:42,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:42,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:42,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:42,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:55:42,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:42,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:55:42,849 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:42,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:42,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:42,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:42,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:42,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:42,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:42,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:55:42,979 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:42,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:55:43,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:43,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:43,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:43,056 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:43,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:43,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:43,104 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:43,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:55:43,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:43,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:55:43,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:43,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:43,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:43,221 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:43,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:43,255 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:43,270 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:43,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:55:43,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:43,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:55:43,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:43,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:43,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:43,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:43,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:43,448 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:43,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:43,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:55:43,500 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 06:55:43,502 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:55:43,505 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:55:43,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:55:43,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:55:43,898 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:55:43,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:55:43,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:55:43,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:55:43,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:55:43,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:55:44,022 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:55:44,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:55:44,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:55:44,545 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:55:44,545 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.04299998283. 2022-02-08 06:55:44,545 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:55:44,546 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:55:44,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:55:44,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:55:44,920 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:55:44,920 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:55:44,936 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:55:44,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:55:44,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:55:44,989 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:55:45,005 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:55:45,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:55:45,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:55:45,183 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:55:45,183 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:55:47,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:55:47,345 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:55:47,345 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:55:49,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 9198 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:55:49,512 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:55:49,512 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:55:51,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:55:51,661 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:55:51,661 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:55:53,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:55:53,811 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:55:53,811 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:55:55,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:55:55,956 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:55:55,956 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:55:57,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:55:58,105 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:55:58,105 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:56:00,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:00,255 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:56:00,255 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:56:02,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:02,470 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:56:02,471 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:56:04,471 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:04,637 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:56:04,637 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:56:06,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:06,894 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:56:06,894 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:56:08,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:09,088 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:56:09,088 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:56:11,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:11,441 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:56:11,441 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:56:13,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:13,759 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:56:13,760 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:56:15,760 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:16,187 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:56:16,188 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:56:18,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:18,471 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:56:18,471 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:56:20,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:20,819 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:56:20,819 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:56:22,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:23,030 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:56:23,032 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:56:25,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:25,280 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:56:25,280 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:56:27,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:27,914 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:56:27,914 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:56:29,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:30,168 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:56:30,168 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:56:32,170 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:32,368 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:56:32,368 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:56:34,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:34,671 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:56:34,671 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 50.126999855. 2022-02-08 06:56:34,673 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:56:34,677 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:56:35,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:56:36,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:56:36,834 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:56:36,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:56:36,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:56:37,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:56:37,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:56:37,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:56:37,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:56:37,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:56:37,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:37,980 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:56:37,982 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.30900001526. 2022-02-08 06:56:37,982 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:56:37,983 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:56:38,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:56:39,201 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:56:39,309 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:56:39,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:56:39,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:56:39,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:56:39,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:56:39,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:56:39,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:56:39,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:56:39,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:40,394 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:56:40,394 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.41299986839. 2022-02-08 06:56:40,395 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:56:40,398 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:56:41,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:56:41,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:56:41,836 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:56:41,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:56:41,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:56:41,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:56:41,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:56:41,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:56:41,956 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:56:41,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:56:42,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:43,125 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:56:43,127 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.73200011253. 2022-02-08 06:56:43,127 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:56:43,128 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:56:43,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:56:44,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:56:44,565 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:56:44,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:56:44,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:56:44,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:56:44,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:56:44,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:56:44,895 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:56:44,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:56:44,989 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:45,960 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:56:45,960 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.83399987221. 2022-02-08 06:56:45,960 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:56:45,963 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:56:46,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:56:46,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:56:47,122 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:56:47,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:56:47,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:56:47,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:56:47,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:56:47,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:56:47,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:56:47,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:56:47,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:47,696 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:56:47,697 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.73500013351. 2022-02-08 06:56:47,697 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:56:47,698 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:56:49,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 9205 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:56:49,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:56:49,812 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:56:49,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:56:49,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:56:49,862 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:56:49,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:56:49,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:56:49,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:56:49,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:56:50,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:50,740 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:56:50,740 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.04399991035. 2022-02-08 06:56:50,742 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:56:50,743 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:56:51,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:56:52,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:56:52,388 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:56:52,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:56:52,415 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:56:52,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:56:52,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:56:52,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:56:52,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:56:52,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:56:52,571 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:54,703 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:56:54,703 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.96300005913. 2022-02-08 06:56:54,703 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:56:54,706 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:56:55,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:56:56,045 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:56:56,226 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:56:56,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:56:56,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:56:56,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:56:56,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:56:56,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:56:56,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:56:56,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:56:56,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:56:57,318 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:56:57,319 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.61500000954. ############################################################ # # RAM FootPrint: 4384.4140625 # ############################################################ 2022-02-08 06:56:57,372 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 06:56:57,372 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 88 seconds. Reason: Iteration:12 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 9233 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:58:25,378 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 06:58:25,378 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 06:58:30,381 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:58:30,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:58:31,112 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:58:31,255 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:58:31,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:58:31,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:58:31,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:58:31,331 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:58:31,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:58:31,362 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:58:31,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:58:31,407 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:58:31,408 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:58:31,476 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['17927 memcached\n'], []) 2022-02-08 06:58:31,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 06:58:31,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 06:58:31,533 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['17927 memcached\n'], []) 2022-02-08 06:58:31,533 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:58:31,536 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:58:32,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:58:32,423 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:58:32,525 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:58:32,525 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:58:32,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:58:32,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:58:32,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:58:32,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:58:32,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:58:32,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:58:32,638 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:58:32,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:58:32,694 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['66169 memcached\n'], []) 2022-02-08 06:58:32,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 06:58:32,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 06:58:32,743 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['66169 memcached\n'], []) 2022-02-08 06:58:32,743 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:58:32,746 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:58:32,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:58:33,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:58:33,112 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:58:33,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:58:33,130 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:58:33,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:58:33,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:58:33,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:58:33,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:58:33,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:58:33,230 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:58:33,232 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:58:33,295 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['51083 memcached\n'], []) 2022-02-08 06:58:33,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 06:58:33,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 06:58:33,348 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['51083 memcached\n'], []) 2022-02-08 06:58:33,348 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:58:33,349 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:58:33,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:58:33,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:58:33,721 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:58:33,721 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:58:33,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:58:33,752 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:58:33,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:58:33,783 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:58:33,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:58:33,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:58:33,832 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:58:33,834 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:58:33,875 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['82491 memcached\n'], []) 2022-02-08 06:58:33,877 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 06:58:33,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 06:58:33,940 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['82491 memcached\n'], []) 2022-02-08 06:58:33,940 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:58:33,951 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:58:34,028 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:58:34,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:58:34,299 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:58:34,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:58:34,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:58:34,326 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:58:34,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:58:34,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:58:34,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:58:34,405 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:58:34,421 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:58:34,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:58:34,473 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['76332 memcached\n'], []) 2022-02-08 06:58:34,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 06:58:34,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 06:58:34,532 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['76332 memcached\n'], []) 2022-02-08 06:58:34,532 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:58:34,535 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:58:34,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:58:34,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:58:34,907 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:58:34,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:58:34,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:58:34,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:58:34,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:58:34,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:58:34,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:58:34,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:58:35,000 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:58:35,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:58:35,049 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['111312 memcached\n'], []) 2022-02-08 06:58:35,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 06:58:35,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 06:58:35,098 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['111312 memcached\n'], []) 2022-02-08 06:58:35,098 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:58:35,101 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:58:35,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:58:35,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:58:35,476 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:58:35,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:58:35,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:58:35,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:58:35,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:58:35,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:58:35,545 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:58:35,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:58:35,569 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:58:35,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:58:35,614 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['45139 memcached\n'], []) 2022-02-08 06:58:35,614 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 06:58:35,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 06:58:35,660 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['45139 memcached\n'], []) 2022-02-08 06:58:35,661 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:58:35,663 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:58:35,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:58:35,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:58:36,062 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:58:36,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:58:36,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:58:36,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:58:36,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:58:36,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:58:36,201 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:58:36,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:58:36,234 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:58:36,234 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:58:36,301 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['60116 memcached\n'], []) 2022-02-08 06:58:36,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 06:58:36,326 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 06:58:36,355 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['60116 memcached\n'], []) 2022-02-08 06:58:36,355 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:58:36,358 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:58:36,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:58:36,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:58:36,730 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:58:36,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:58:36,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:58:36,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:58:36,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:58:36,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:58:36,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:58:36,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:58:36,832 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:58:36,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:58:36,878 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['109461 memcached\n'], []) 2022-02-08 06:58:36,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 06:58:36,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 06:58:36,923 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['109461 memcached\n'], []) 2022-02-08 06:58:36,923 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:58:36,924 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:58:37,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:58:37,244 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:58:37,323 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:58:37,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:58:37,341 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:58:37,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:58:37,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:58:37,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:58:37,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:58:37,415 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:58:37,434 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 06:58:37,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:58:37,496 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['126123 memcached\n'], []) 2022-02-08 06:58:37,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 06:58:37,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 06:58:37,543 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['126123 memcached\n'], []) 2022-02-08 06:58:37,543 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:58:37,546 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 06:58:37,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:58:37,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 06:58:37,918 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:58:37,920 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 06:58:37,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 06:58:37,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 06:58:37,961 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 06:58:37,979 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 06:58:37,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 06:58:38,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 06:58:38,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:58:38,036 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 06:58:38,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:58:38,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:58:40,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:58:41,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:58:41,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:58:41,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:58:41,790 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 06:58:41,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:58:41,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:58:41,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:58:41,885 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 06:58:41,885 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 06:58:41,888 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 06:58:41,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:58:42,198 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 06:58:42,272 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:58:42,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 06:58:42,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 06:58:42,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 06:58:42,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 06:58:42,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 06:58:42,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 06:58:42,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 06:58:42,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:58:42,385 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 06:58:42,426 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:58:42,438 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:58:45,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:58:45,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:58:45,964 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:58:46,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:58:46,312 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 06:58:46,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:58:46,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:58:46,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:58:46,398 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 06:58:46,398 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 06:58:46,401 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 06:58:46,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:58:46,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 06:58:46,753 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:58:46,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 06:58:46,767 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 06:58:46,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 06:58:46,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 06:58:46,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 06:58:46,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 06:58:46,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 06:58:46,854 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:58:46,869 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 06:58:46,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:58:46,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:58:48,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:58:48,530 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:58:48,788 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:58:49,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:58:49,045 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 06:58:49,045 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:58:49,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:58:49,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:58:49,144 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 06:58:49,144 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 06:58:49,148 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 06:58:49,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:58:49,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 5 queries submitted, 0 failed, 9257 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:58:49,542 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:58:49,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 06:58:49,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 06:58:49,571 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 06:58:49,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 06:58:49,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 06:58:49,619 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 06:58:49,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 06:58:49,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:58:49,665 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 06:58:49,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:58:49,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:58:49,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:58:49,744 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:58:49,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:58:49,780 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:58:49,801 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 06:58:49,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:58:49,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:58:49,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:58:49,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:49,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:58:49,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:49,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:49,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:50,017 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:50,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:50,084 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:50,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:50,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:50,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:50,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:58:50,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:50,239 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:50,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:50,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:50,293 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:50,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:50,339 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:50,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:50,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:50,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:58:50,410 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:50,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:50,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:50,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:50,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:50,497 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:50,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:50,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:50,548 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:50,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:58:50,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:50,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:50,619 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:50,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:50,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:50,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:50,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:50,706 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:50,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:50,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:58:50,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:50,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:50,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:50,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:50,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:50,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:50,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:50,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:50,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:50,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:58:50,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:50,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:50,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:50,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:51,005 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:51,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:51,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:51,058 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:51,075 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 06:58:51,078 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 06:58:51,321 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:58:51,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 06:58:52,121 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:58:52,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 06:58:52,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 06:58:52,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 06:58:52,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 06:58:52,454 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 06:58:52,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 06:58:52,586 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 06:58:52,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:58:52,700 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 06:58:52,729 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:58:52,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:58:54,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:58:54,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:58:54,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:58:55,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:58:55,046 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 06:58:55,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:58:55,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:58:55,119 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:58:55,134 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 06:58:55,134 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 06:58:55,137 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 06:58:55,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:58:55,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 06:58:55,599 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:58:55,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 06:58:55,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 06:58:55,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 06:58:55,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 06:58:55,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 06:58:55,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 06:58:55,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 06:58:55,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:58:55,773 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 06:58:55,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:58:55,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:58:55,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:58:55,857 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:58:55,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:58:55,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:58:55,901 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 06:58:55,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:58:55,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:58:55,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:58:55,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:55,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:58:56,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:56,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:56,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:56,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:56,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:56,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:56,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:56,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:58:56,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:56,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:58:56,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:56,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:56,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:56,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:56,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:56,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:56,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:56,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:58:56,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:56,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:58:56,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:56,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:56,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:56,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:56,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:56,528 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:56,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:56,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:58:56,584 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:56,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:58:56,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:56,630 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:56,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:56,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:56,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:56,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:56,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:56,727 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:58:56,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:56,755 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:58:56,770 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:56,783 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:56,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:56,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:56,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:56,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:56,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:56,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:58:56,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:56,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:58:56,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:56,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:56,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:56,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:56,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:56,982 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:56,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:57,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:58:57,026 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 06:58:57,029 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 06:58:57,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:58:57,308 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 06:58:57,375 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:58:57,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 06:58:57,392 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 06:58:57,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 06:58:57,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 06:58:57,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 06:58:57,446 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 06:58:57,459 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 06:58:57,471 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:58:57,486 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 06:58:57,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:58:57,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:58:58,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:58:59,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:58:59,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:59:00,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:59:00,128 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 06:59:00,130 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:59:00,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:59:00,191 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:59:00,207 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 06:59:00,207 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 06:59:00,210 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 06:59:00,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:59:00,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 06:59:00,584 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:59:00,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 06:59:00,599 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 06:59:00,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 06:59:00,640 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 06:59:00,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 06:59:00,678 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 06:59:00,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 06:59:00,711 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:59:00,729 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 06:59:00,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:59:00,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:59:05,155 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:59:05,706 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:59:06,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:59:06,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:59:06,315 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 06:59:06,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:59:06,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:59:06,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:59:06,417 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 06:59:06,417 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 06:59:06,420 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 06:59:06,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:59:06,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 06:59:06,825 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:59:06,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 06:59:06,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 06:59:06,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 06:59:06,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 06:59:06,890 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 06:59:06,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 06:59:06,923 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 06:59:06,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:59:06,957 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 06:59:06,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:59:07,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:59:07,022 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:59:07,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:59:07,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:59:07,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:59:07,073 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 06:59:07,073 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:59:07,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:59:07,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:59:07,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:07,160 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:59:07,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:07,224 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:07,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:07,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:07,321 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:07,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:07,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:07,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:07,456 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:07,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:59:07,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:07,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:07,528 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:07,545 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:07,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:07,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:07,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:07,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:07,630 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:07,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:59:07,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:07,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:07,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:07,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:07,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:07,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:07,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:07,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:07,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:07,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:59:07,857 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:07,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:07,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:07,910 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:07,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:07,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:07,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:07,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:08,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:08,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:59:08,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:08,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:08,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:08,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:08,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:08,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:08,140 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:08,155 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:08,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:08,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:59:08,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:08,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:08,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:08,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:08,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:08,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:08,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:08,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:08,352 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 06:59:08,355 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 06:59:08,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:59:08,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 06:59:08,724 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:59:08,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 06:59:08,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 06:59:08,752 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 06:59:08,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 06:59:08,783 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 06:59:08,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 06:59:08,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 06:59:08,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:59:08,844 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 06:59:08,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:59:08,885 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:59:08,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:59:08,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:59:08,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:59:08,943 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:59:08,957 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 06:59:08,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:59:08,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:59:08,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:59:08,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:09,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:59:09,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:09,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:09,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:09,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:09,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:09,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:09,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:09,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:09,341 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:09,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:59:09,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:09,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:09,413 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:09,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:09,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:09,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:09,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:09,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:09,525 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:09,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:59:09,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:09,573 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:09,589 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:09,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:09,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:09,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:09,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:09,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:09,693 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:09,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:59:09,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:09,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:09,783 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:09,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:09,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:09,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:09,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:09,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:09,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:09,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:59:09,931 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:09,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:09,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:09,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:09,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:10,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:10,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:10,043 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:10,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:10,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:59:10,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:10,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:10,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:10,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:10,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:10,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:10,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:10,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:10,236 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 06:59:10,239 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:59:10,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:59:10,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:59:10,697 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:59:10,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:59:10,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:59:10,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:59:10,749 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:59:10,767 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:59:10,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:59:10,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:59:10,823 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:59:10,848 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 06:59:10,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:59:10,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:59:12,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:59:13,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:59:13,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:59:14,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:59:14,230 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 06:59:14,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:59:14,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:59:14,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:59:14,332 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 06:59:14,332 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 06:59:14,335 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 06:59:14,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:59:14,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 06:59:14,818 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:59:14,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 06:59:14,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 06:59:14,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 06:59:14,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 06:59:14,903 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 06:59:14,923 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 06:59:14,941 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 06:59:14,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:59:14,994 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 06:59:15,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:59:15,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:59:18,660 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:59:19,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:59:19,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:59:19,762 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:59:19,782 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 06:59:19,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:59:19,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:59:19,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:59:19,878 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 06:59:19,880 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 06:59:19,881 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:59:20,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:59:21,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:59:21,289 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:59:21,290 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:59:21,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:59:21,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:59:21,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:59:21,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:59:21,418 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:59:21,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:59:21,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:59:21,483 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 06:59:21,515 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:59:21,530 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:59:25,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:59:25,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:59:25,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:59:26,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:59:26,197 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 06:59:26,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:59:26,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:59:26,285 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:59:26,302 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 06:59:26,303 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:59:26,305 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 06:59:26,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:59:26,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 06:59:26,930 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:59:26,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 06:59:26,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 06:59:26,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 06:59:27,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 06:59:27,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 06:59:27,084 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 06:59:27,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 06:59:27,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:59:27,158 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 06:59:27,186 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:59:27,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:59:32,229 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:59:32,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:59:33,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:59:33,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:59:33,740 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 06:59:33,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:59:33,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:59:33,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:59:33,973 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 06:59:33,973 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 06:59:33,976 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 06:59:34,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:59:34,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 06:59:34,384 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:59:34,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 06:59:34,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 06:59:34,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 06:59:34,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 06:59:34,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 06:59:34,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 06:59:34,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 06:59:34,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 06:59:34,520 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 06:59:34,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 06:59:34,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:59:34,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 06:59:34,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:59:34,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 06:59:34,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 06:59:34,651 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 06:59:34,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 06:59:34,670 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 06:59:34,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 06:59:34,709 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:34,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 06:59:34,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:34,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:34,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:34,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:34,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:34,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:34,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:35,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 06:59:35,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:35,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 06:59:35,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:35,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:35,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:35,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:35,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:35,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:35,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:35,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 06:59:35,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:35,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 06:59:35,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:35,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:35,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:35,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:35,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:35,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:35,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:35,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 06:59:35,441 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:35,459 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 06:59:35,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:35,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:35,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:35,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:35,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:35,566 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:35,584 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:35,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 06:59:35,619 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:35,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 06:59:35,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:35,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:35,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:35,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:35,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:35,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:35,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:35,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 06:59:35,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:35,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 06:59:35,841 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:35,859 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:35,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:35,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:35,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:35,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:35,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:35,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 06:59:35,989 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 06:59:35,990 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:59:35,990 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 06:59:36,081 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:59:36,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 06:59:36,436 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:59:36,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 06:59:36,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 06:59:36,464 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 06:59:36,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 06:59:36,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 06:59:36,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 06:59:36,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 06:59:36,548 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:59:37,072 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 06:59:37,072 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.08200001717. 2022-02-08 06:59:37,072 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 06:59:37,073 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 06:59:37,183 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 06:59:37,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 06:59:37,651 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 06:59:37,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 06:59:37,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 06:59:37,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 06:59:37,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 06:59:37,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 06:59:37,778 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 06:59:37,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 06:59:37,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:59:38,075 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:59:38,076 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:59:40,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:59:40,267 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:59:40,267 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:59:42,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:59:42,556 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:59:42,558 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:59:44,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:59:44,799 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:59:44,801 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:59:46,801 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:59:47,016 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:59:47,016 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:59:49,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:59:49,232 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:59:49,232 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 9262 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 06:59:51,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:59:51,509 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:59:51,509 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:59:53,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:59:54,154 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:59:54,154 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:59:56,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:59:56,634 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:59:56,634 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 06:59:58,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 06:59:58,915 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 06:59:58,915 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:00:00,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:01,154 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:00:01,154 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:00:03,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:03,592 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:00:03,594 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:00:05,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:05,769 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:00:05,769 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:00:07,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:08,124 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:00:08,125 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:00:10,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:10,401 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:00:10,403 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:00:12,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:12,601 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:00:12,601 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:00:14,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:14,823 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:00:14,825 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:00:16,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:17,000 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:00:17,000 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:00:19,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:19,158 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:00:19,160 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:00:21,160 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:21,302 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:00:21,302 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 44.2300000191. 2022-02-08 07:00:21,302 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:00:21,305 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:00:21,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:00:22,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:00:22,483 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:00:22,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:00:22,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:00:22,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:00:22,548 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:00:22,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:00:22,640 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:00:22,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:00:22,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:23,200 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:00:23,201 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.89999985695. 2022-02-08 07:00:23,201 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:00:23,204 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:00:23,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:00:24,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:00:24,206 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:00:24,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:00:24,221 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:00:24,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:00:24,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:00:24,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:00:24,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:00:24,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:00:24,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:24,921 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:00:24,923 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.72100019455. 2022-02-08 07:00:24,923 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:00:24,924 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:00:25,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:00:25,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:00:25,865 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:00:25,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:00:25,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:00:25,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:00:25,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:00:25,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:00:25,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:00:25,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:00:25,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:26,457 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:00:26,457 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.53399991989. 2022-02-08 07:00:26,457 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:00:26,459 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:00:27,145 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:00:27,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:00:27,502 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:00:27,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:00:27,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:00:27,535 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:00:27,548 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:00:27,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:00:27,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:00:27,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:00:27,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:28,061 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:00:28,062 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.60400009155. 2022-02-08 07:00:28,062 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:00:28,063 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:00:28,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:00:28,910 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:00:29,009 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:00:29,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:00:29,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:00:29,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:00:29,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:00:29,066 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:00:29,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:00:29,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:00:29,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:29,572 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:00:29,573 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.5119998455. 2022-02-08 07:00:29,573 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:00:29,575 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:00:30,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:00:30,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:00:30,732 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:00:30,732 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:00:30,762 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:00:30,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:00:30,798 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:00:30,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:00:30,854 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:00:30,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:00:30,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:31,323 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:00:31,323 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.75. 2022-02-08 07:00:31,325 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:00:31,326 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:00:31,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:00:32,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:00:32,290 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:00:32,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:00:32,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:00:32,331 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:00:32,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:00:32,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:00:32,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:00:32,397 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:00:32,418 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:33,198 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:00:33,198 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.87399983406. 2022-02-08 07:00:33,198 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:00:33,200 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:00:33,658 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:00:33,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:00:34,065 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:00:34,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:00:34,084 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:00:34,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:00:34,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:00:34,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:00:34,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:00:34,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:00:34,211 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:00:34,996 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:00:34,996 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.79700016975. ############################################################ # # RAM FootPrint: 4391.87109375 # ############################################################ 2022-02-08 07:00:35,036 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 07:00:35,036 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 105 seconds. Reason: Iteration:13 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 9268 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 9294 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:02:20,046 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 07:02:20,046 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 07:02:25,049 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:02:25,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:25,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:02:25,825 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:25,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:02:25,859 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:02:25,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:02:25,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:02:25,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:02:25,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:02:25,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:02:25,980 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:02:25,982 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 07:02:26,046 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['18605 memcached\n'], []) 2022-02-08 07:02:26,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 07:02:26,084 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 07:02:26,105 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['18605 memcached\n'], []) 2022-02-08 07:02:26,105 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:02:26,108 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:02:26,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:26,729 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:02:26,808 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:26,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:02:26,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:02:26,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:02:26,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:02:26,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:02:26,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:02:26,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:02:26,937 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:02:26,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 07:02:27,042 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['67140 memcached\n'], []) 2022-02-08 07:02:27,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 07:02:27,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 07:02:27,104 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['67140 memcached\n'], []) 2022-02-08 07:02:27,104 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:02:27,107 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:02:27,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:27,655 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:02:27,720 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:27,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:02:27,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:02:27,752 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:02:27,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:02:27,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:02:27,801 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:02:27,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:02:27,831 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:02:27,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 07:02:27,913 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['51884 memcached\n'], []) 2022-02-08 07:02:27,913 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 07:02:27,941 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 07:02:27,966 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['51884 memcached\n'], []) 2022-02-08 07:02:27,967 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:02:27,969 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:02:28,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:28,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:02:28,349 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:28,349 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:02:28,362 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:02:28,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:02:28,387 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:02:28,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:02:28,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:02:28,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:02:28,444 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:02:28,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 07:02:28,489 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['83156 memcached\n'], []) 2022-02-08 07:02:28,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 07:02:28,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 07:02:28,533 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['83156 memcached\n'], []) 2022-02-08 07:02:28,533 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:02:28,536 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:02:28,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:28,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:02:28,882 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:28,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:02:28,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:02:28,910 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:02:28,923 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:02:28,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:02:28,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:02:28,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:02:28,980 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:02:28,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 07:02:29,019 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['77039 memcached\n'], []) 2022-02-08 07:02:29,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 07:02:29,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 07:02:29,069 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['77039 memcached\n'], []) 2022-02-08 07:02:29,071 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:02:29,072 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:02:29,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:29,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:02:29,427 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:29,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:02:29,441 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:02:29,454 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:02:29,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:02:29,484 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:02:29,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:02:29,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:02:29,523 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:02:29,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 07:02:29,595 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['112022 memcached\n'], []) 2022-02-08 07:02:29,596 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 07:02:29,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 07:02:29,641 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['112022 memcached\n'], []) 2022-02-08 07:02:29,641 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:02:29,644 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:02:29,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:29,943 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:02:30,049 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:30,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:02:30,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:02:30,073 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:02:30,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:02:30,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:02:30,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:02:30,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:02:30,138 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:02:30,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 07:02:30,187 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['45851 memcached\n'], []) 2022-02-08 07:02:30,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 07:02:30,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 07:02:30,230 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['45851 memcached\n'], []) 2022-02-08 07:02:30,230 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:02:30,233 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:02:30,331 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:30,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:02:30,611 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:30,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:02:30,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:02:30,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:02:30,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:02:30,670 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:02:30,683 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:02:30,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:02:30,709 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:02:30,709 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 07:02:30,760 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['60782 memcached\n'], []) 2022-02-08 07:02:30,760 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 07:02:30,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 07:02:30,812 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['60782 memcached\n'], []) 2022-02-08 07:02:30,812 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:02:30,822 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:02:30,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:31,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:02:31,183 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:31,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:02:31,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:02:31,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:02:31,221 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:02:31,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:02:31,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:02:31,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:02:31,275 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:02:31,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 07:02:31,319 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['110129 memcached\n'], []) 2022-02-08 07:02:31,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 07:02:31,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 07:02:31,365 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['110129 memcached\n'], []) 2022-02-08 07:02:31,365 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:02:31,367 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:02:31,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:31,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:02:31,752 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:31,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:02:31,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:02:31,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:02:31,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:02:31,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:02:31,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:02:31,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:02:31,884 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:02:31,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 07:02:31,951 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['126833 memcached\n'], []) 2022-02-08 07:02:31,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 07:02:31,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 07:02:32,007 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['126833 memcached\n'], []) 2022-02-08 07:02:32,007 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:02:32,009 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:02:32,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:32,329 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:02:32,394 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:32,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:02:32,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:02:32,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:02:32,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:02:32,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:02:32,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:02:32,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:02:32,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:02:32,517 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 07:02:32,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:02:32,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:34,997 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:02:35,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:35,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:36,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:02:36,301 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 07:02:36,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:02:36,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:36,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:02:36,391 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 07:02:36,391 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:02:36,394 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:02:36,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:36,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:02:36,746 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:36,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:02:36,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:02:36,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:02:36,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:02:36,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:02:36,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:02:36,829 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:02:36,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:02:36,867 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 07:02:36,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:02:36,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:38,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:02:38,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:38,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:39,198 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:02:39,213 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 07:02:39,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:02:39,227 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:39,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:02:39,309 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 07:02:39,311 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:02:39,312 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:02:39,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:39,607 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:02:39,674 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:39,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:02:39,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:02:39,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:02:39,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:02:39,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:02:39,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:02:39,765 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:02:39,780 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:02:39,795 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 07:02:39,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:02:39,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:40,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:02:40,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:40,969 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:41,198 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:02:41,213 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 07:02:41,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:02:41,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:41,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:02:41,332 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 07:02:41,332 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:02:41,336 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 07:02:41,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:41,650 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 07:02:41,721 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:41,721 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 07:02:41,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 07:02:41,752 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 07:02:41,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 07:02:41,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 07:02:41,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 07:02:41,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 07:02:41,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:02:41,844 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 07:02:41,869 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:02:41,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:41,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:02:41,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:41,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:41,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:02:41,960 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 07:02:41,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:02:41,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:41,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:02:42,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:42,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:02:42,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:42,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:42,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:42,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:42,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:42,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:42,293 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:42,331 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:42,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:42,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:02:42,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:42,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:42,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:42,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:42,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:42,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:42,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:42,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:42,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:42,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:02:42,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:42,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:42,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:42,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:42,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:42,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:42,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:42,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:42,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:42,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:02:42,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:42,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:42,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:42,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:42,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:42,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:42,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:42,869 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:42,885 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:42,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:02:42,920 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:42,936 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:42,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:42,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:42,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:43,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:43,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:43,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:43,058 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:43,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:02:43,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:43,112 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:43,130 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:43,147 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:43,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:43,183 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:43,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:43,219 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:43,239 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 07:02:43,240 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:02:43,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:43,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:02:44,239 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:44,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:02:44,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:02:44,438 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:02:44,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:02:44,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:02:44,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:02:44,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:02:44,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:02:44,885 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 07:02:44,979 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:02:45,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:45,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:02:45,961 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:46,221 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:46,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:02:46,569 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 07:02:46,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:02:46,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:46,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:02:46,864 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 07:02:46,864 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:02:46,867 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 07:02:46,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:47,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 07:02:47,263 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:47,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 07:02:47,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 07:02:47,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 07:02:47,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 07:02:47,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 07:02:47,336 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 07:02:47,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 07:02:47,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:02:47,381 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 07:02:47,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:02:47,426 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:47,441 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:02:47,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:47,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:47,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:02:47,523 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 07:02:47,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:02:47,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:47,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:02:47,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:47,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:02:47,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:47,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:47,709 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:47,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:47,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:47,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:47,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:47,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:47,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:47,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:02:47,956 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:47,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:47,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:48,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:48,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:48,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:48,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:48,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:48,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:48,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:02:48,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:48,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:48,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:48,186 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:48,201 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:48,219 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:48,234 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:48,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:48,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:48,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:02:48,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:48,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:48,326 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:48,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:48,357 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:48,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:48,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:48,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:48,423 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:48,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:02:48,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:48,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:48,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:48,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:48,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:48,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:48,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:48,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:48,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:48,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:02:48,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:48,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:48,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:48,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:48,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:48,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:48,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:48,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:48,782 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 07:02:48,786 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:02:48,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:49,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:02:49,128 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:49,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:02:49,142 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:02:49,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:02:49,170 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:02:49,186 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:02:49,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:02:49,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:02:49,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:02:49,239 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 07:02:49,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:02:49,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 9314 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:02:53,512 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:02:54,117 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:54,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:54,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:02:54,838 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 07:02:54,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:02:54,854 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:54,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:02:54,913 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 07:02:54,914 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:02:54,914 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:02:55,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:55,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:02:55,286 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:55,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:02:55,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:02:55,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:02:55,329 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:02:55,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:02:55,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:02:55,377 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:02:55,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:02:55,407 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 07:02:55,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:02:55,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:56,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:02:56,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:56,918 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:57,216 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:02:57,236 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 07:02:57,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:02:57,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:57,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:02:57,342 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 07:02:57,342 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:02:57,345 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 07:02:57,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:57,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 07:02:57,763 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:57,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 07:02:57,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 07:02:57,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 07:02:57,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 07:02:57,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 07:02:57,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 07:02:57,854 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 07:02:57,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:02:57,907 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 07:02:57,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:02:57,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:57,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:02:57,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:58,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:58,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:02:58,036 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 07:02:58,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:02:58,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:58,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:02:58,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:58,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:02:58,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:58,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:58,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:58,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:58,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:58,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:58,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:58,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:02:58,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:58,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:02:58,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:58,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:58,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:58,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:58,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:58,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:58,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:58,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:02:58,589 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:58,607 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:02:58,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:58,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:58,658 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:58,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:58,693 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:58,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:58,727 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:58,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:02:58,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:58,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:02:58,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:58,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:58,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:58,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:58,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:58,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:58,903 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:58,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:02:58,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:58,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:02:58,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:58,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:59,015 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:59,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:59,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:59,068 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:59,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:59,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:02:59,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:59,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:02:59,178 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:59,204 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:59,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:59,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:59,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:59,283 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:59,303 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:59,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:02:59,341 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 07:02:59,344 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 07:02:59,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:02:59,642 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 07:02:59,700 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:02:59,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 07:02:59,714 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 07:02:59,727 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 07:02:59,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 07:02:59,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 07:02:59,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 07:02:59,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 07:02:59,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:02:59,819 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 07:02:59,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:02:59,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:59,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:02:59,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:59,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:02:59,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:02:59,930 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 07:02:59,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:02:59,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:02:59,961 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:02:59,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:00,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:03:00,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:00,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:00,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:00,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:00,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:00,204 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:00,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:00,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:00,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:00,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:03:00,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:00,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:00,385 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:00,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:00,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:00,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:00,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:00,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:00,482 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:00,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:03:00,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:00,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:00,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:00,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:00,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:00,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:00,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:00,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:00,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:00,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:03:00,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:00,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:00,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:00,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:00,732 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:00,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:00,762 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:00,783 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:00,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:00,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:03:00,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:00,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:00,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:00,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:00,890 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:00,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:00,918 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:00,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:00,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:00,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:03:00,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:00,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:01,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:01,025 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:01,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:01,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:01,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:01,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:01,101 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 07:03:01,104 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:03:01,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:03:01,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:03:01,720 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:03:01,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:03:01,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:03:01,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:03:01,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:03:01,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:03:01,890 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:03:01,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:03:01,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:03:01,984 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 07:03:02,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:03:02,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:03:03,017 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:03:03,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:03:04,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:03:04,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:03:04,538 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 07:03:04,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:03:04,650 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:03:04,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:03:04,953 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 07:03:04,953 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:03:04,957 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:03:05,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:03:05,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:03:05,594 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:03:05,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:03:05,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:03:05,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:03:05,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:03:05,727 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:03:05,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:03:05,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:03:05,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:03:06,002 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 07:03:06,028 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:03:06,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:03:06,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:03:07,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:03:07,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:03:08,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:03:08,286 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 07:03:08,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:03:08,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:03:08,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:03:08,375 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 07:03:08,377 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:03:08,378 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:03:08,464 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:03:08,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:03:08,744 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:03:08,744 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:03:08,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:03:08,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:03:08,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:03:08,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:03:08,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:03:08,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:03:08,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:03:08,861 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 07:03:08,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:03:08,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:03:09,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:03:09,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:03:10,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:03:10,285 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:03:10,299 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 07:03:10,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:03:10,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:03:10,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:03:10,371 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 07:03:10,371 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:03:10,374 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:03:10,474 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:03:10,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:03:10,789 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:03:10,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:03:10,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:03:10,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:03:10,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:03:10,854 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:03:10,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:03:10,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:03:10,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:03:10,934 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 07:03:10,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:03:10,979 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:03:11,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:03:12,117 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:03:12,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:03:12,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:03:12,796 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 07:03:12,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:03:12,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:03:12,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:03:12,872 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 07:03:12,872 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:03:12,875 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 07:03:12,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:03:13,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 07:03:13,236 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:03:13,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 07:03:13,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 07:03:13,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 07:03:13,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 07:03:13,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 07:03:13,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 07:03:13,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 07:03:13,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:03:13,346 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 07:03:13,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:03:13,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:03:13,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:03:13,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:03:13,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:03:13,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:03:13,463 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 07:03:13,464 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:03:13,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:03:13,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:03:13,505 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:13,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:03:13,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:13,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:13,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:13,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:13,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:13,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:13,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:13,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:03:13,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:13,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:03:13,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:13,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:13,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:13,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:13,931 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:13,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:13,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:13,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:03:13,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:14,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:03:14,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:14,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:14,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:14,071 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:14,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:14,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:14,117 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:14,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:03:14,145 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:14,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:03:14,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:14,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:14,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:14,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:14,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:14,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:14,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:14,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:03:14,293 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:14,308 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:03:14,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:14,336 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:14,349 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:14,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:14,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:14,392 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:14,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:14,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:03:14,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:14,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:03:14,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:14,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:14,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:14,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:14,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:14,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:14,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:14,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:03:14,602 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 07:03:14,602 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:03:14,605 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:03:14,681 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:03:14,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:03:14,954 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:03:14,956 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:03:14,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:03:14,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:03:14,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:03:15,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:03:15,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:03:15,045 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:03:15,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:15,523 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:03:15,523 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.920000076294. 2022-02-08 07:03:15,523 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:03:15,526 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:03:15,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:03:15,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:03:15,891 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:03:15,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:03:15,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:03:15,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:03:15,928 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:03:15,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:03:15,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:03:15,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:03:15,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:16,125 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:16,127 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:18,127 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:18,365 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:18,367 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:20,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:20,519 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:20,520 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:22,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:22,680 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:22,680 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:24,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:24,835 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:24,835 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:26,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:26,994 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:26,994 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:28,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:29,142 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:29,144 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:31,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:31,286 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:31,286 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:33,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:33,757 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:33,757 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:35,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:36,025 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:36,026 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:38,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:38,716 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:38,717 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:40,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:40,914 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:40,914 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:42,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:43,144 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:43,144 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:45,147 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:45,354 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:45,354 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:47,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:47,576 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:47,578 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 9315 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:03:49,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:49,769 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:49,769 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:51,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:51,957 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:51,957 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:53,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:54,233 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:54,233 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:56,234 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:56,517 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:56,519 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:03:58,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:03:58,818 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:03:58,818 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:04:00,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:01,072 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:04:01,073 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:04:03,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:03,312 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:04:03,312 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:04:05,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:05,578 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:04:05,578 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:04:07,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:07,786 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:04:07,786 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:04:09,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:10,032 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:04:10,032 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 54.5090000629. 2022-02-08 07:04:10,032 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:04:10,035 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:04:10,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:04:10,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:04:11,450 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:04:11,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:04:11,579 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:04:11,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:04:11,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:04:12,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:04:12,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:04:12,326 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:04:12,410 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:13,007 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:04:13,009 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 07:04:15,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:15,509 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:04:15,509 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 07:04:17,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:18,302 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:04:18,302 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 07:04:20,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:20,678 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:04:20,678 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 07:04:22,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:23,296 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:04:23,296 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 07:04:25,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:25,819 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:04:25,819 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 07:04:27,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:28,418 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:04:28,418 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 18.385999918. 2022-02-08 07:04:28,420 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:04:28,421 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:04:29,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:04:30,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:04:30,516 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:04:30,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:04:30,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:04:30,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:04:30,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:04:30,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:04:30,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:04:30,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:04:30,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:31,871 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:04:31,871 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.45199990273. 2022-02-08 07:04:31,871 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:04:31,874 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:04:32,760 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:04:33,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:04:33,276 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:04:33,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:04:33,308 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:04:33,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:04:33,349 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:04:33,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:04:33,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:04:33,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:04:33,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:34,269 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:04:34,269 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.39800000191. 2022-02-08 07:04:34,269 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:04:34,272 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:04:35,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:04:35,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:04:35,611 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:04:35,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:04:35,647 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:04:35,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:04:35,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:04:35,732 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:04:35,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:04:35,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:04:35,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:36,755 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:04:36,756 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.48600006104. 2022-02-08 07:04:36,756 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:04:36,759 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:04:37,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:04:37,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:04:37,776 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:04:37,778 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:04:37,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:04:37,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:04:37,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:04:37,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:04:37,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:04:37,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:04:37,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:38,336 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:04:38,336 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.58099985123. 2022-02-08 07:04:38,336 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:04:38,338 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:04:39,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:04:39,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:04:39,835 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:04:39,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:04:39,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:04:39,885 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:04:39,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:04:39,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:04:39,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:04:39,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:04:40,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:40,815 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:04:40,815 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.47900009155. 2022-02-08 07:04:40,816 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:04:40,818 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:04:41,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:04:41,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:04:41,839 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:04:41,841 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:04:41,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:04:41,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:04:41,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:04:41,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:04:41,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:04:41,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:04:41,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:43,688 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:04:43,688 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.87199997902. 2022-02-08 07:04:43,688 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:04:43,690 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:04:44,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:04:44,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:04:44,506 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:04:44,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:04:44,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:04:44,545 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:04:44,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:04:44,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:04:44,589 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:04:44,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:04:44,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:04:45,391 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:04:45,391 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.70300006866. ############################################################ # # RAM FootPrint: 4381.16015625 # ############################################################ 2022-02-08 07:04:45,430 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 07:04:45,430 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 109 seconds. Reason: Iteration:14 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 9334 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 9360 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:06:34,437 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 07:06:34,438 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 07:06:39,440 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:06:40,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:06:40,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:06:40,665 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:06:40,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:06:40,681 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:06:40,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:06:40,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:06:40,767 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:06:40,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:06:40,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:06:40,822 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:06:40,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 07:06:40,874 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['19348 memcached\n'], []) 2022-02-08 07:06:40,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 07:06:40,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 07:06:40,937 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['19348 memcached\n'], []) 2022-02-08 07:06:40,937 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:06:40,940 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:06:41,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:06:41,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:06:41,852 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:06:41,854 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:06:41,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:06:41,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:06:41,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:06:41,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:06:41,928 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:06:41,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:06:41,976 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:06:41,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 07:06:42,029 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['68104 memcached\n'], []) 2022-02-08 07:06:42,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 07:06:42,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 07:06:42,079 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['68104 memcached\n'], []) 2022-02-08 07:06:42,081 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:06:42,084 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:06:42,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:06:42,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:06:42,755 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:06:42,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:06:42,780 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:06:42,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:06:42,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:06:42,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:06:42,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:06:42,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:06:42,900 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:06:42,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 07:06:43,042 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['52686 memcached\n'], []) 2022-02-08 07:06:43,043 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 07:06:43,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 07:06:43,092 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['52686 memcached\n'], []) 2022-02-08 07:06:43,092 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:06:43,095 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:06:43,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:06:43,413 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:06:43,470 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:06:43,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:06:43,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:06:43,497 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:06:43,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:06:43,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:06:43,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:06:43,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:06:43,565 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:06:43,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 07:06:43,638 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['83890 memcached\n'], []) 2022-02-08 07:06:43,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 07:06:43,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 07:06:43,687 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['83890 memcached\n'], []) 2022-02-08 07:06:43,688 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:06:43,690 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:06:43,780 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:06:43,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:06:44,049 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:06:44,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:06:44,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:06:44,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:06:44,091 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:06:44,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:06:44,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:06:44,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:06:44,148 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:06:44,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 07:06:44,203 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['77746 memcached\n'], []) 2022-02-08 07:06:44,204 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 07:06:44,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 07:06:44,250 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['77746 memcached\n'], []) 2022-02-08 07:06:44,250 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:06:44,253 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:06:44,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:06:44,543 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:06:44,611 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:06:44,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:06:44,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:06:44,640 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:06:44,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:06:44,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:06:44,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:06:44,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:06:44,714 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:06:44,714 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 07:06:44,763 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['112730 memcached\n'], []) 2022-02-08 07:06:44,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 07:06:44,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 07:06:44,813 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['112730 memcached\n'], []) 2022-02-08 07:06:44,813 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:06:44,815 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:06:44,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:06:45,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:06:45,163 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:06:45,163 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:06:45,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:06:45,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:06:45,201 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:06:45,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:06:45,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:06:45,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:06:45,256 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:06:45,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 07:06:45,302 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['46595 memcached\n'], []) 2022-02-08 07:06:45,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 07:06:45,326 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 07:06:45,348 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['46595 memcached\n'], []) 2022-02-08 07:06:45,348 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:06:45,351 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:06:45,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:06:45,665 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:06:45,733 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:06:45,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:06:45,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:06:45,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:06:45,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:06:45,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:06:45,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:06:45,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:06:45,832 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:06:45,834 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 07:06:45,898 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['61520 memcached\n'], []) 2022-02-08 07:06:45,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 07:06:45,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 07:06:46,006 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['61520 memcached\n'], []) 2022-02-08 07:06:46,006 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:06:46,009 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:06:46,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:06:46,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:06:46,351 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:06:46,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:06:46,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:06:46,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:06:46,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:06:46,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:06:46,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:06:46,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:06:46,457 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:06:46,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 07:06:46,506 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['110872 memcached\n'], []) 2022-02-08 07:06:46,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 07:06:46,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 07:06:46,561 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['110872 memcached\n'], []) 2022-02-08 07:06:46,562 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:06:46,565 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:06:46,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:06:46,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:06:46,924 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:06:46,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:06:46,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:06:46,956 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:06:46,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:06:46,989 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:06:47,005 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:06:47,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:06:47,038 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:06:47,038 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 07:06:47,099 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['127577 memcached\n'], []) 2022-02-08 07:06:47,099 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 07:06:47,127 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 07:06:47,148 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['127577 memcached\n'], []) 2022-02-08 07:06:47,148 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:06:47,151 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:06:47,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:06:47,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:06:47,561 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:06:47,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:06:47,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:06:47,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:06:47,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:06:47,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:06:47,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:06:47,650 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:06:47,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:06:47,680 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 07:06:47,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:06:47,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:06:48,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:06:48,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:06:49,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 9384 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:06:49,709 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:06:49,723 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 07:06:49,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:06:49,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:06:49,823 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:06:49,839 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 07:06:49,839 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:06:49,844 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:06:49,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:06:50,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:06:50,226 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:06:50,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:06:50,242 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:06:50,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:06:50,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:06:50,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:06:50,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:06:50,316 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:06:50,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:06:50,348 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 07:06:50,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:06:50,387 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:06:55,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:06:55,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:06:56,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:06:56,410 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:06:56,424 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 07:06:56,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:06:56,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:06:56,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:06:56,532 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 07:06:56,532 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:06:56,540 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:06:56,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:06:56,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:06:56,897 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:06:56,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:06:56,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:06:56,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:06:56,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:06:56,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:06:56,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:06:56,982 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:06:56,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:06:57,009 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 07:06:57,038 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:06:57,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:06:57,474 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:06:57,903 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:06:58,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:06:58,446 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:06:58,463 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 07:06:58,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:06:58,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:06:58,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:06:58,568 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 07:06:58,568 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:06:58,571 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 07:06:58,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:06:58,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 07:06:58,938 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:06:58,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 07:06:58,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 07:06:58,964 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 07:06:58,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 07:06:58,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 07:06:59,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 07:06:59,022 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 07:06:59,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:06:59,048 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 07:06:59,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:06:59,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:06:59,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:06:59,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:06:59,127 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:06:59,140 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:06:59,151 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 07:06:59,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:06:59,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:06:59,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:06:59,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:06:59,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:06:59,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:06:59,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:06:59,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:06:59,349 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:06:59,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:06:59,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:06:59,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:06:59,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:06:59,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:06:59,545 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:06:59,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:06:59,581 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:06:59,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:06:59,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:06:59,632 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:06:59,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:06:59,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:06:59,688 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:06:59,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:06:59,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:06:59,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:06:59,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:06:59,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:06:59,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:06:59,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:06:59,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:06:59,849 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:06:59,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:06:59,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:06:59,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:06:59,920 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:06:59,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:06:59,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:06:59,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:06:59,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:00,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:00,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:00,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:00,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:00,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:07:00,099 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:00,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:00,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:00,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:00,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:00,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:00,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:00,219 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:00,234 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:00,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:07:00,270 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:00,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:00,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:00,326 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:00,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:00,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:00,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:00,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:00,414 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 07:07:00,417 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:07:00,548 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:07:00,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:07:01,000 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:07:01,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:07:01,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:07:01,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:07:01,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:07:01,119 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:07:01,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:07:01,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:07:01,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:07:01,288 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 07:07:01,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:07:01,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:04,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:07:04,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:04,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:04,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:07:04,964 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 07:07:04,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:07:04,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:05,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:07:05,055 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 07:07:05,055 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:07:05,058 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 07:07:05,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:07:05,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 07:07:05,421 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:07:05,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 07:07:05,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 07:07:05,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 07:07:05,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 07:07:05,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 07:07:05,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 07:07:05,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 07:07:05,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:07:05,532 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 07:07:05,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:07:05,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:05,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:07:05,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:05,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:05,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:07:05,641 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 07:07:05,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:07:05,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:05,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:07:05,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:05,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:07:05,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:05,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:05,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:05,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:05,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:05,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:05,948 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:05,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:06,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:06,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:07:06,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:06,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:06,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:06,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:06,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:06,127 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:06,142 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:06,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:06,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:06,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:07:06,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:06,229 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:06,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:06,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:06,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:06,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:06,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:06,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:06,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:06,357 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:07:06,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:06,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:06,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:06,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:06,446 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:06,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:06,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:06,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:06,505 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:06,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:07:06,535 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:06,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:06,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:06,581 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:06,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:06,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:06,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:06,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:06,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:06,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:07:06,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:06,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:06,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:06,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:06,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:06,780 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:06,798 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:06,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:06,832 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 07:07:06,832 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:07:06,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:07:07,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:07:07,210 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:07:07,211 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:07:07,224 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:07:07,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:07:07,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:07:07,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:07:07,283 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:07:07,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:07:07,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:07:07,328 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 07:07:07,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:07:07,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:09,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:07:09,655 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:10,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:10,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:07:10,371 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 07:07:10,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:07:10,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:10,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:07:10,460 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 07:07:10,460 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:07:10,463 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:07:10,614 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:07:11,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:07:11,914 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:07:11,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:07:12,022 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:07:12,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:07:12,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:07:12,280 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:07:12,303 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:07:12,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:07:12,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:07:12,414 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 07:07:12,441 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:07:12,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:13,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:07:13,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:14,295 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:14,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:07:14,611 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 07:07:14,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:07:14,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:14,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:07:14,717 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 07:07:14,717 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:07:14,720 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 07:07:14,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:07:15,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 07:07:15,095 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:07:15,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 07:07:15,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 07:07:15,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 07:07:15,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 07:07:15,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 07:07:15,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 07:07:15,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 07:07:15,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:07:15,204 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 07:07:15,229 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:07:15,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:15,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:07:15,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:15,290 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:15,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:07:15,325 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 07:07:15,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:07:15,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:15,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:07:15,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:15,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:07:15,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:15,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:15,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:15,584 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:15,619 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:15,655 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:15,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:15,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:15,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:15,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:07:15,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:15,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:15,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:15,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:15,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:15,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:15,928 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:15,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:15,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:15,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:07:16,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:16,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:16,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:16,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:16,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:16,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:16,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:16,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:16,155 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:16,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:07:16,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:16,219 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:16,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:16,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:16,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:16,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:16,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:16,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:16,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:16,362 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:07:16,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:16,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:16,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:16,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:16,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:16,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:16,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:16,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:16,512 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:16,528 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:07:16,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:16,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:16,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:16,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:16,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:16,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:16,647 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:16,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:16,678 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 07:07:16,681 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 07:07:16,755 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:07:16,979 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 07:07:17,049 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:07:17,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 07:07:17,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 07:07:17,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 07:07:17,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 07:07:17,109 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 07:07:17,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 07:07:17,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 07:07:17,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:07:17,171 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 07:07:17,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:07:17,214 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:17,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:07:17,244 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:17,257 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:17,272 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:07:17,286 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 07:07:17,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:07:17,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:17,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:07:17,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:17,362 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:07:17,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:17,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:17,471 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:17,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:17,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:17,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:17,614 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:17,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:17,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:17,729 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:07:17,749 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:17,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:17,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:17,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:17,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:17,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:17,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:17,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:17,890 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:17,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:07:17,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:17,941 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:17,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:17,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:17,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:18,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:18,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:18,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:18,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:18,081 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:07:18,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:18,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:18,130 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:18,147 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:18,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:18,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:18,193 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:18,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:18,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:18,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:07:18,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:18,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:18,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:18,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:18,316 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:18,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:18,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:18,362 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:18,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:18,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:07:18,418 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:18,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:18,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:18,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:18,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:18,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:18,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:18,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:18,548 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 07:07:18,549 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:07:18,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:07:18,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:07:19,061 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:07:19,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:07:19,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:07:19,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:07:19,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:07:19,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:07:19,150 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:07:19,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:07:19,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:07:19,213 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 07:07:19,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:07:19,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:21,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:07:22,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:22,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:23,306 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:07:23,375 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 07:07:23,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:07:23,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:23,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:07:23,565 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 07:07:23,565 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:07:23,568 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:07:23,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:07:23,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:07:24,061 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:07:24,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:07:24,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:07:24,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:07:24,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:07:24,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:07:24,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:07:24,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:07:24,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:07:24,213 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 07:07:24,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:07:24,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:24,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:07:25,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:25,530 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:25,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:07:25,844 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 07:07:25,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:07:25,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:25,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:07:25,923 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 07:07:25,923 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:07:25,926 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:07:26,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:07:26,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:07:26,454 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:07:26,456 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:07:26,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:07:26,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:07:26,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:07:26,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:07:26,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:07:26,545 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:07:26,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:07:26,576 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 07:07:26,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:07:26,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:30,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:07:30,683 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:30,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:31,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:07:31,220 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 07:07:31,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:07:31,232 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:31,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:07:31,293 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 07:07:31,293 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:07:31,296 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:07:31,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:07:31,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:07:31,700 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:07:31,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:07:31,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:07:31,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:07:31,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:07:31,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:07:31,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:07:31,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:07:31,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:07:31,855 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 07:07:31,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:07:31,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:33,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:07:34,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:34,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:35,053 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:07:35,072 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 07:07:35,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:07:35,091 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:35,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:07:35,144 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 07:07:35,144 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:07:35,147 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 07:07:35,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:07:35,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 07:07:35,532 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:07:35,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 07:07:35,548 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 07:07:35,563 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 07:07:35,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 07:07:35,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 07:07:35,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 07:07:35,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 07:07:35,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:07:35,651 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 07:07:35,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:07:35,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:35,709 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:07:35,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:35,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:07:35,760 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:07:35,786 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 07:07:35,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:07:35,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:07:35,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:07:35,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:35,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:07:35,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:35,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:35,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:36,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:36,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:36,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:36,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:36,163 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:07:36,196 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:36,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:07:36,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:36,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:36,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:36,283 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:36,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:36,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:36,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:36,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:07:36,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:36,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:07:36,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:36,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:36,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:36,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:36,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:36,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:36,505 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:36,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:07:36,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:36,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:07:36,573 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:36,589 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:36,607 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:36,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:36,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:36,658 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:36,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:36,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:07:36,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:36,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:07:36,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:36,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:36,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:36,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:36,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:36,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:36,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:36,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:07:36,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:36,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:07:36,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:36,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:36,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:36,969 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:36,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:37,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:37,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:37,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:07:37,055 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 07:07:37,055 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:07:37,056 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:07:37,142 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:07:37,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:07:37,461 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:07:37,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:07:37,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:07:37,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:07:37,507 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:07:37,525 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:07:37,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:07:37,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:07:37,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:07:38,131 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:07:38,131 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.0759999752. 2022-02-08 07:07:38,131 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:07:38,134 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:07:38,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:07:38,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:07:38,624 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:07:38,624 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:07:38,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:07:38,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:07:38,665 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:07:38,688 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:07:38,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:07:38,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:07:38,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:07:38,938 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:07:38,940 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:07:40,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:07:41,101 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:07:41,102 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:07:43,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:07:43,269 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:07:43,269 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:07:45,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:07:45,486 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:07:45,486 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:07:47,487 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:07:47,838 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:07:47,838 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 9390 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:07:49,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:07:50,039 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:07:50,039 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:07:52,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:07:52,242 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:07:52,243 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:07:54,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:07:54,575 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:07:54,575 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:07:56,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:07:56,898 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:07:56,900 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:07:58,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:07:59,336 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:07:59,336 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:08:01,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:01,601 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:08:01,601 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:08:03,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:03,895 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:08:03,897 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:08:05,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:06,203 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:08:06,203 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:08:08,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:08,437 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:08:08,437 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:08:10,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:10,654 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:08:10,654 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:08:12,655 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:12,895 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:08:12,895 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:08:14,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:15,127 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:08:15,127 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:08:17,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:17,349 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:08:17,349 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:08:19,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:19,546 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:08:19,546 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:08:21,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:21,752 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:08:21,752 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:08:23,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:23,934 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:08:23,934 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:08:25,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:26,095 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:08:26,096 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 47.9649999142. 2022-02-08 07:08:26,096 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:08:26,098 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:08:26,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:08:27,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:08:27,378 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:08:27,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:08:27,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:08:27,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:08:27,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:08:27,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:08:27,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:08:27,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:08:27,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:27,806 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:08:27,806 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.71000003815. 2022-02-08 07:08:27,808 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:08:27,809 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:08:28,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:08:28,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:08:29,006 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:08:29,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:08:29,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:08:29,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:08:29,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:08:29,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:08:29,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:08:29,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:08:29,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:29,832 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:08:29,832 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.02499985695. 2022-02-08 07:08:29,832 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:08:29,835 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:08:30,642 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:08:31,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:08:31,285 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:08:31,285 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:08:31,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:08:31,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:08:31,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:08:31,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:08:31,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:08:31,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:08:31,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:32,529 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:08:32,530 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.69799995422. 2022-02-08 07:08:32,530 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:08:32,532 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:08:33,484 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:08:33,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:08:33,959 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:08:33,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:08:33,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:08:34,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:08:34,091 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:08:34,117 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:08:34,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:08:34,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:08:34,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:35,601 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:08:35,601 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.07100009918. 2022-02-08 07:08:35,601 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:08:35,604 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:08:36,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:08:36,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:08:36,815 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:08:36,816 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:08:36,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:08:36,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:08:36,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:08:36,895 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:08:36,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:08:36,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:08:36,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:37,651 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:08:37,651 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.04999995232. 2022-02-08 07:08:37,651 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:08:37,654 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:08:38,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:08:38,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:08:38,818 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:08:38,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:08:38,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:08:38,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:08:38,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:08:38,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:08:38,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:08:39,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:08:39,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:39,696 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:08:39,697 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.04500007629. 2022-02-08 07:08:39,697 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:08:39,698 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:08:40,658 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:08:40,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:08:41,154 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:08:41,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:08:41,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:08:41,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:08:41,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:08:41,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:08:41,249 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:08:41,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:08:41,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:42,809 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:08:42,812 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.11500000954. 2022-02-08 07:08:42,812 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:08:42,813 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:08:43,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:08:43,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:08:43,993 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:08:43,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:08:44,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:08:44,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:08:44,066 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:08:44,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:08:44,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:08:44,163 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:08:44,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:08:45,671 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:08:45,671 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.85899996758. ############################################################ # # RAM FootPrint: 4392.125 # ############################################################ 2022-02-08 07:08:45,706 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 07:08:45,706 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 66 seconds. Reason: Iteration:15 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 9399 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 9422 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:09:51,713 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 07:09:51,713 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 07:09:56,716 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:09:56,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:09:57,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:09:57,085 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:09:57,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:09:57,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:09:57,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:09:57,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:09:57,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:09:57,196 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:09:57,227 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:09:57,253 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:09:57,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 07:09:57,323 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['20089 memcached\n'], []) 2022-02-08 07:09:57,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 07:09:57,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 07:09:57,388 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['20089 memcached\n'], []) 2022-02-08 07:09:57,388 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:09:57,390 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:09:57,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:09:58,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:09:58,355 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:09:58,357 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:09:58,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:09:58,387 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:09:58,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:09:58,415 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:09:58,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:09:58,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:09:58,473 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:09:58,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 07:09:58,529 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['69131 memcached\n'], []) 2022-02-08 07:09:58,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 07:09:58,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 07:09:58,569 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['69131 memcached\n'], []) 2022-02-08 07:09:58,571 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:09:58,572 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:09:58,645 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:09:58,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:09:58,938 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:09:58,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:09:58,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:09:58,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:09:59,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:09:59,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:09:59,038 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:09:59,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:09:59,075 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:09:59,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 07:09:59,121 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['53636 memcached\n'], []) 2022-02-08 07:09:59,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 07:09:59,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 07:09:59,171 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['53636 memcached\n'], []) 2022-02-08 07:09:59,173 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:09:59,174 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:09:59,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:09:59,482 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:09:59,539 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:09:59,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:09:59,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:09:59,563 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:09:59,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:09:59,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:09:59,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:09:59,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:09:59,634 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:09:59,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 07:09:59,676 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['84629 memcached\n'], []) 2022-02-08 07:09:59,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 07:09:59,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 07:09:59,723 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['84629 memcached\n'], []) 2022-02-08 07:09:59,723 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:09:59,726 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:09:59,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:00,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:10:00,109 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:00,109 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:10:00,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:10:00,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:10:00,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:10:00,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:10:00,183 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:10:00,201 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:10:00,214 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:10:00,216 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 07:10:00,265 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['78515 memcached\n'], []) 2022-02-08 07:10:00,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 07:10:00,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 07:10:00,315 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['78515 memcached\n'], []) 2022-02-08 07:10:00,315 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:10:00,318 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:10:00,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:00,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:10:00,694 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:00,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:10:00,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:10:00,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:10:00,732 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:10:00,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:10:00,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:10:00,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:10:00,788 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:10:00,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 07:10:00,859 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['113500 memcached\n'], []) 2022-02-08 07:10:00,859 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 07:10:00,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 07:10:00,903 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['113500 memcached\n'], []) 2022-02-08 07:10:00,904 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:10:00,905 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:10:00,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:01,191 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:10:01,246 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:01,247 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:10:01,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:10:01,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:10:01,295 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:10:01,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:10:01,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:10:01,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:10:01,351 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:10:01,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 07:10:01,418 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['47336 memcached\n'], []) 2022-02-08 07:10:01,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 07:10:01,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 07:10:01,473 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['47336 memcached\n'], []) 2022-02-08 07:10:01,473 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:10:01,476 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:10:01,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:01,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:10:01,846 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:01,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:10:01,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:10:01,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:10:01,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:10:01,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:10:01,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:10:01,928 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:10:01,943 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:10:01,943 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 07:10:01,997 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['62303 memcached\n'], []) 2022-02-08 07:10:01,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 07:10:02,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 07:10:02,043 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['62303 memcached\n'], []) 2022-02-08 07:10:02,043 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:10:02,046 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:10:02,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:02,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:10:02,398 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:02,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:10:02,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:10:02,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:10:02,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:10:02,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:10:02,464 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:10:02,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:10:02,492 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:10:02,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 07:10:02,536 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['111610 memcached\n'], []) 2022-02-08 07:10:02,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 07:10:02,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 07:10:02,582 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['111610 memcached\n'], []) 2022-02-08 07:10:02,582 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:10:02,584 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:10:02,693 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:02,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:10:02,980 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:02,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:10:02,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:10:03,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:10:03,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:10:03,043 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:10:03,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:10:03,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:10:03,092 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:10:03,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 07:10:03,170 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['128316 memcached\n'], []) 2022-02-08 07:10:03,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 07:10:03,198 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 07:10:03,221 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['128316 memcached\n'], []) 2022-02-08 07:10:03,221 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:10:03,223 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:10:03,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:03,589 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:10:03,647 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:03,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:10:03,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:10:03,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:10:03,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:10:03,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:10:03,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:10:03,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:10:03,752 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:10:03,770 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 07:10:03,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:10:03,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:06,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:10:07,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:07,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:07,859 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:10:07,874 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 07:10:07,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:10:07,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:07,956 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:10:07,970 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 07:10:07,971 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:10:07,973 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:10:08,056 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:08,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:10:08,345 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:08,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:10:08,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:10:08,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:10:08,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:10:08,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:10:08,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:10:08,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:10:08,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:10:08,456 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 07:10:08,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:10:08,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:09,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:10:09,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:09,941 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:10,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:10:10,262 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 07:10:10,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:10:10,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:10,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:10:10,384 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 07:10:10,384 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:10:10,387 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:10:10,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:10,683 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:10:10,743 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:10,744 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:10:10,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:10:10,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:10:10,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:10:10,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:10:10,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:10:10,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:10:10,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:10:10,855 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 07:10:10,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:10:10,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:11,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:10:11,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:12,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:12,272 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:10:12,293 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 07:10:12,293 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:10:12,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:12,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:10:12,388 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 07:10:12,388 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:10:12,391 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 07:10:12,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:12,760 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 07:10:12,828 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:12,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 07:10:12,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 07:10:12,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 07:10:12,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 07:10:12,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 07:10:12,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 07:10:12,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 07:10:12,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:10:12,954 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 07:10:12,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:10:12,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:13,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:10:13,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:13,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:13,056 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:10:13,072 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 07:10:13,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:10:13,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:13,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:10:13,117 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:13,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:10:13,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:13,221 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:13,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:13,293 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:13,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:13,362 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:13,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:13,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:13,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:13,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:10:13,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:13,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:13,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:13,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:13,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:13,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:13,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:13,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:13,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:13,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:10:13,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:13,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:13,721 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:13,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:13,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:13,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:13,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:13,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:13,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:13,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:10:13,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:13,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:13,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:13,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:13,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:13,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:13,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:13,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:14,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:14,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:10:14,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:14,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:14,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:14,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:14,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:14,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:14,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:14,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:14,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:14,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:10:14,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:14,244 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:14,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:14,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:14,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:14,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:14,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:14,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:14,380 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 07:10:14,381 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:10:14,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:15,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:10:15,599 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:15,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:10:15,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:10:15,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:10:15,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:10:15,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:10:15,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:10:15,688 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:10:15,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:10:15,719 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 07:10:15,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:10:15,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:16,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:10:17,071 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:17,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:17,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:10:17,697 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 07:10:17,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:10:17,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:17,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:10:18,016 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 07:10:18,017 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:10:18,019 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 07:10:18,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:18,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 07:10:18,421 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:18,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 07:10:18,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 07:10:18,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 07:10:18,464 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 07:10:18,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 07:10:18,497 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 07:10:18,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 07:10:18,528 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:10:18,545 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 07:10:18,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:10:18,589 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:18,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:10:18,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:18,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:18,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:10:18,661 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 07:10:18,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:10:18,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:18,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:10:18,706 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:18,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:10:18,780 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:18,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:18,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:18,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:18,918 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:18,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:18,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:19,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:19,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:19,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:10:19,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:19,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:19,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:19,170 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:19,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:19,204 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:19,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:19,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:19,257 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:19,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:10:19,295 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:19,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:19,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:19,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:19,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:19,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:19,397 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:19,413 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:19,428 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:19,446 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:10:19,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:19,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:19,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:19,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:19,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:19,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:19,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:19,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:19,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:19,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:10:19,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:19,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:19,660 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:19,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:19,693 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:19,709 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:19,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:19,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:19,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:19,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:10:19,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:19,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:19,829 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:19,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:19,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:19,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:19,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:19,918 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:19,936 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 07:10:19,938 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:10:20,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:20,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:10:20,263 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:20,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:10:20,280 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:10:20,295 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:10:20,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:10:20,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:10:20,336 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:10:20,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:10:20,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:10:20,380 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 07:10:20,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:10:20,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:23,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:10:23,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:24,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:24,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:10:24,630 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 07:10:24,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:10:24,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:24,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:10:24,703 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 07:10:24,703 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:10:24,707 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:10:24,780 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:25,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:10:25,069 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:25,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:10:25,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:10:25,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:10:25,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:10:25,140 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:10:25,160 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:10:25,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:10:25,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:10:25,221 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 07:10:25,255 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:10:25,272 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:28,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:10:28,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:29,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:29,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:10:29,441 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 07:10:29,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:10:29,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:29,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:10:29,535 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 07:10:29,536 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:10:29,536 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 07:10:29,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:29,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 07:10:29,904 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:29,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 07:10:29,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 07:10:29,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 07:10:29,943 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 07:10:29,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 07:10:29,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 07:10:29,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 07:10:30,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:10:30,019 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 07:10:30,045 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:10:30,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:30,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:10:30,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:30,099 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:30,112 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:10:30,125 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 07:10:30,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:10:30,140 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:30,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:10:30,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:30,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:10:30,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:30,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:30,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:30,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:30,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:30,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:30,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:30,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:30,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:30,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:10:30,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:30,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:30,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:30,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:30,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:30,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:30,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:30,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:30,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:30,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:10:30,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:30,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:30,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:30,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:30,798 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:30,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:30,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:30,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:30,862 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:30,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:10:30,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:30,910 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:30,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:30,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:30,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:30,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:30,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:31,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:31,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:31,035 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:10:31,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:31,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:31,081 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:31,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:31,109 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:31,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:31,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:31,155 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:31,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:31,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:10:31,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:31,227 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:31,244 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:31,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:31,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:31,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:31,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:31,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:31,345 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 07:10:31,348 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 07:10:31,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:31,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 07:10:31,726 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:31,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 07:10:31,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 07:10:31,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 07:10:31,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 07:10:31,788 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 07:10:31,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 07:10:31,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 07:10:31,834 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:10:31,851 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 07:10:31,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:10:31,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:31,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:10:31,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:31,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:31,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:10:31,976 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 07:10:31,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:10:31,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:32,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:10:32,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:32,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:10:32,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:32,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:32,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:32,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:32,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:32,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:32,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:32,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:32,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:32,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:10:32,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:32,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:32,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:32,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:32,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:32,535 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:32,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:32,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:32,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:32,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:10:32,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:32,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:32,650 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:32,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:32,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:32,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:32,711 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:32,727 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:32,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:32,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:10:32,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:32,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:32,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:32,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:32,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:32,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:32,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:32,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:32,903 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:32,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:10:32,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:32,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:32,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:32,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:33,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:33,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:33,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:33,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:33,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:33,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:10:33,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:33,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:33,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:33,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:33,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:33,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:33,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:33,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:33,234 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 07:10:33,236 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:10:33,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:33,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:10:33,625 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:33,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:10:33,640 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:10:33,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:10:33,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:10:33,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:10:33,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:10:33,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:10:33,729 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:10:33,747 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 07:10:33,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:10:33,788 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:34,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:10:35,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:35,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:35,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:10:36,000 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 07:10:36,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:10:36,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:36,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:10:36,088 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 07:10:36,088 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:10:36,092 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:10:36,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:36,459 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:10:36,533 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:36,535 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:10:36,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:10:36,566 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:10:36,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:10:36,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:10:36,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:10:36,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:10:36,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:10:36,667 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 07:10:36,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:10:36,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:37,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:10:37,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:37,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:38,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:10:38,168 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 07:10:38,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:10:38,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:38,227 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:10:38,246 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 07:10:38,246 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:10:38,249 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:10:38,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:39,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:10:39,272 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:39,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:10:39,290 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:10:39,308 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:10:39,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:10:39,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:10:39,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:10:39,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:10:39,415 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:10:39,431 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 07:10:39,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:10:39,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:39,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:10:40,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:40,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:40,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:10:41,000 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 07:10:41,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:10:41,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:41,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:10:41,109 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 07:10:41,109 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:10:41,112 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:10:41,221 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:41,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:10:41,753 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:41,755 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:10:41,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:10:41,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:10:41,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:10:41,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:10:41,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:10:41,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:10:41,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:10:42,086 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 07:10:42,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:10:42,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:43,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:10:43,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:44,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:44,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:10:44,822 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 07:10:44,823 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:10:44,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:45,163 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:10:45,323 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 07:10:45,325 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:10:45,328 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 07:10:45,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:45,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 07:10:45,759 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:45,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 07:10:45,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 07:10:45,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 07:10:45,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 07:10:45,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 07:10:45,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 07:10:45,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 07:10:45,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:10:45,882 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 07:10:45,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:10:45,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:45,943 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:10:45,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:45,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:10:45,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:10:46,000 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 07:10:46,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:10:46,015 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:10:46,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:10:46,045 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:46,081 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:10:46,117 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:46,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:46,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:46,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:46,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:46,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:46,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:46,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:10:46,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:46,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:10:46,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:46,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:46,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:46,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:46,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:46,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:46,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:46,573 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:10:46,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:46,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:10:46,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:46,645 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:46,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:46,678 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:46,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:46,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:46,729 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:46,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:10:46,760 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:46,778 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:10:46,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:46,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:46,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:46,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:46,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:46,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:46,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:46,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:10:46,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:46,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:10:46,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:46,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:46,982 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:46,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:47,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:47,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:47,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:47,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:10:47,073 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:47,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:10:47,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:47,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:47,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:47,170 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:47,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:47,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:47,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:47,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:10:47,275 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 07:10:47,276 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:10:47,278 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:10:47,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:47,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:10:47,691 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:47,693 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:10:47,706 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:10:47,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:10:47,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:10:47,767 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:10:47,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:10:47,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:10:47,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:10:48,369 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:10:48,371 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.09500002861. 2022-02-08 07:10:48,371 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:10:48,374 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:10:48,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:10:48,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:10:48,815 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:10:48,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:10:48,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:10:48,859 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:10:48,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:10:48,903 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:10:48,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:10:48,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:10:48,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:10:49,124 | test | WARNING | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 07:10:49,128 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:10:49,128 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 9437 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:10:51,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:10:51,259 | test | WARNING | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 07:10:51,265 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:10:51,265 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:10:53,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:10:53,401 | test | WARNING | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 07:10:53,410 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:10:53,410 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:10:55,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:10:55,546 | test | WARNING | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 07:10:55,552 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:10:55,553 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:10:57,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:10:57,698 | test | WARNING | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 07:10:57,706 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:10:57,706 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:10:59,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:10:59,878 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:10:59,878 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:01,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:02,053 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:02,053 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:04,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:04,230 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:04,230 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:06,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:06,401 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:06,401 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:08,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:08,579 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:08,581 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:10,581 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:10,779 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:10,779 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:12,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:12,982 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:12,982 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:14,982 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:15,164 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:15,164 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:17,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:17,382 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:17,382 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:19,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:19,578 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:19,578 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:21,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:21,776 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:21,778 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:23,778 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:23,957 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:23,957 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:25,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:26,160 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:26,161 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:28,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:28,334 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:28,335 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:30,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:30,538 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:30,538 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:32,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:32,752 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:32,752 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:34,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:35,072 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:35,072 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:37,073 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:37,316 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:37,318 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:39,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:39,621 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:39,622 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:41,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:41,937 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:41,937 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:43,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:44,155 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:44,157 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:46,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:46,348 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:46,349 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:48,349 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:48,716 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:48,717 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 9437 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:11:50,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:51,565 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:51,565 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:53,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:54,082 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:54,082 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:56,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:56,391 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:56,392 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:11:58,392 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:11:58,615 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:11:58,615 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:12:00,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:12:00,821 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:12:00,821 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:12:02,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:12:03,221 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:12:03,223 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 74.8519999981. 2022-02-08 07:12:03,223 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:12:03,224 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:12:04,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:12:04,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:12:04,733 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:12:04,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:12:04,752 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:12:04,783 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:12:04,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:12:04,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:12:04,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:12:04,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:12:04,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:12:05,615 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:12:05,615 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.39299988747. 2022-02-08 07:12:05,615 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:12:05,618 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:12:06,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:12:07,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:12:07,226 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:12:07,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:12:07,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:12:07,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:12:07,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:12:07,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:12:07,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:12:07,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:12:07,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:12:08,559 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:12:08,561 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.94500017166. 2022-02-08 07:12:08,561 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:12:08,562 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:12:09,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:12:09,482 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:12:09,617 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:12:09,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:12:09,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:12:09,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:12:09,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:12:09,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:12:09,706 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:12:09,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:12:09,767 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:12:10,461 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:12:10,463 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.90199995041. 2022-02-08 07:12:10,463 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:12:10,464 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:12:11,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:12:11,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:12:11,447 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:12:11,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:12:11,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:12:11,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:12:11,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:12:11,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:12:11,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:12:11,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:12:11,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:12:12,163 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:12:12,163 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.70000004768. 2022-02-08 07:12:12,163 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:12:12,164 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:12:12,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:12:13,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:12:13,101 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:12:13,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:12:13,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:12:13,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:12:13,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:12:13,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:12:13,178 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:12:13,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:12:13,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:12:13,670 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:12:13,671 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.507999897. 2022-02-08 07:12:13,671 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:12:13,673 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:12:14,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:12:14,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:12:14,959 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:12:14,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:12:14,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:12:14,989 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:12:15,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:12:15,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:12:15,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:12:15,056 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:12:15,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:12:15,930 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:12:15,931 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.26099991798. 2022-02-08 07:12:15,931 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:12:15,934 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:12:16,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:12:17,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:12:17,421 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:12:17,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:12:17,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:12:17,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:12:17,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:12:17,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:12:17,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:12:17,535 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:12:17,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:12:18,789 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:12:18,789 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.85800004005. 2022-02-08 07:12:18,789 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:12:18,792 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:12:19,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:12:20,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:12:20,148 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:12:20,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:12:20,165 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:12:20,186 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:12:20,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:12:20,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:12:20,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:12:20,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:12:20,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:12:20,862 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:12:20,862 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.07299995422. ############################################################ # # RAM FootPrint: 4383.28515625 # ############################################################ 2022-02-08 07:12:20,895 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 07:12:20,895 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 77 seconds. Reason: Iteration:16 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 9460 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:13:37,904 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 07:13:37,904 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 07:13:42,907 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:13:43,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:13:43,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:13:43,358 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:13:43,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:13:43,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:13:43,387 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:13:43,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:13:43,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:13:43,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:13:43,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:13:43,467 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:13:43,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 07:13:43,532 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['20776 memcached\n'], []) 2022-02-08 07:13:43,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 07:13:43,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 07:13:43,582 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['20776 memcached\n'], []) 2022-02-08 07:13:43,582 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:13:43,585 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:13:44,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:13:44,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:13:44,448 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:13:44,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:13:44,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:13:44,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:13:44,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:13:44,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:13:44,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:13:44,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:13:44,598 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:13:44,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 07:13:44,657 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['70041 memcached\n'], []) 2022-02-08 07:13:44,658 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 07:13:44,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 07:13:44,707 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['70041 memcached\n'], []) 2022-02-08 07:13:44,707 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:13:44,710 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:13:44,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:13:45,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:13:45,072 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:13:45,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:13:45,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:13:45,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:13:45,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:13:45,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:13:45,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:13:45,163 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:13:45,178 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:13:45,178 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 07:13:45,236 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['54389 memcached\n'], []) 2022-02-08 07:13:45,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 07:13:45,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 07:13:45,285 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['54389 memcached\n'], []) 2022-02-08 07:13:45,286 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:13:45,288 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:13:45,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:13:45,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:13:45,651 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:13:45,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:13:45,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:13:45,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:13:45,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:13:45,706 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:13:45,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:13:45,732 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:13:45,746 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:13:45,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 07:13:45,805 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['85317 memcached\n'], []) 2022-02-08 07:13:45,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 07:13:45,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 07:13:45,851 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['85317 memcached\n'], []) 2022-02-08 07:13:45,851 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:13:45,855 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:13:45,948 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:13:46,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:13:46,210 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:13:46,211 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:13:46,224 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:13:46,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:13:46,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:13:46,267 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:13:46,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:13:46,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:13:46,318 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:13:46,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 07:13:46,375 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['79174 memcached\n'], []) 2022-02-08 07:13:46,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 07:13:46,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 07:13:46,424 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['79174 memcached\n'], []) 2022-02-08 07:13:46,426 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:13:46,430 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:13:46,528 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:13:46,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:13:46,818 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:13:46,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:13:46,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:13:46,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:13:46,862 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:13:46,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:13:46,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:13:46,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:13:46,927 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:13:46,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 07:13:46,976 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['114157 memcached\n'], []) 2022-02-08 07:13:46,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 07:13:47,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 07:13:47,023 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['114157 memcached\n'], []) 2022-02-08 07:13:47,023 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:13:47,025 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:13:47,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:13:47,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:13:47,536 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:13:47,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:13:47,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:13:47,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:13:47,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:13:47,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:13:47,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:13:47,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:13:47,637 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:13:47,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 07:13:47,681 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['48033 memcached\n'], []) 2022-02-08 07:13:47,681 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 07:13:47,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 07:13:47,726 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['48033 memcached\n'], []) 2022-02-08 07:13:47,726 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:13:47,729 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:13:47,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:13:48,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:13:48,085 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:13:48,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:13:48,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:13:48,112 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:13:48,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:13:48,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:13:48,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:13:48,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:13:48,180 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:13:48,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 07:13:48,243 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['62947 memcached\n'], []) 2022-02-08 07:13:48,244 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 07:13:48,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 07:13:48,296 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['62947 memcached\n'], []) 2022-02-08 07:13:48,298 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:13:48,299 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:13:48,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:13:48,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:13:48,638 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:13:48,640 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:13:48,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:13:48,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:13:48,681 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:13:48,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:13:48,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:13:48,721 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:13:48,734 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:13:48,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 07:13:48,778 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['112307 memcached\n'], []) 2022-02-08 07:13:48,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 07:13:48,801 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 07:13:48,821 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['112307 memcached\n'], []) 2022-02-08 07:13:48,822 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:13:48,823 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:13:48,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:13:49,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:13:49,203 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:13:49,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:13:49,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:13:49,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:13:49,242 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:13:49,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:13:49,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:13:49,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:13:49,303 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:13:49,303 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 07:13:49,361 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['128966 memcached\n'], []) 2022-02-08 07:13:49,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 07:13:49,387 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 07:13:49,408 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['128966 memcached\n'], []) 2022-02-08 07:13:49,408 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:13:49,411 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:13:49,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 9485 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:13:49,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:13:49,796 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:13:49,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:13:49,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:13:49,823 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:13:49,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:13:49,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:13:49,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:13:49,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:13:49,895 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:13:49,911 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 07:13:49,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:13:49,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:13:52,232 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:13:52,823 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:13:53,211 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:13:53,535 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:13:53,549 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 07:13:53,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:13:53,563 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:13:53,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:13:53,645 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 07:13:53,645 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:13:53,648 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:13:53,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:13:53,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:13:54,055 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:13:54,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:13:54,071 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:13:54,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:13:54,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:13:54,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:13:54,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:13:54,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:13:54,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:13:54,167 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 07:13:54,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:13:54,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:13:59,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:13:59,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:00,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:00,586 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:14:00,601 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 07:14:00,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:14:00,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:00,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:14:00,701 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 07:14:00,701 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:14:00,703 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:14:00,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:14:00,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:14:01,043 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:14:01,045 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:14:01,058 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:14:01,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:14:01,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:14:01,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:14:01,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:14:01,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:14:01,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:14:01,167 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 07:14:01,193 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:14:01,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:05,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:14:05,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:05,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:06,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:14:06,025 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 07:14:06,025 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:14:06,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:06,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:14:06,121 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 07:14:06,121 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:14:06,124 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 07:14:06,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:14:06,441 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 07:14:06,505 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:14:06,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 07:14:06,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 07:14:06,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 07:14:06,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 07:14:06,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 07:14:06,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 07:14:06,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 07:14:06,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:14:06,637 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 07:14:06,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:14:06,678 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:06,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:14:06,711 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:06,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:06,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:14:06,755 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 07:14:06,755 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:14:06,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:06,788 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:14:06,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:06,841 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:14:06,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:06,923 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:06,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:06,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:07,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:07,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:07,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:07,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:07,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:07,183 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:14:07,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:07,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:07,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:07,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:07,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:07,280 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:07,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:07,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:07,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:07,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:14:07,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:07,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:07,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:07,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:07,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:07,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:07,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:07,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:07,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:07,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:14:07,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:07,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:07,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:07,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:07,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:07,596 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:07,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:07,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:07,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:07,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:14:07,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:07,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:07,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:07,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:07,727 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:07,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:07,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:07,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:07,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:07,801 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:14:07,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:07,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:07,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:07,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:07,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:07,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:07,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:07,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:07,950 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 07:14:07,953 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:14:08,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:14:08,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:14:08,357 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:14:08,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:14:08,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:14:08,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:14:08,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:14:08,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:14:08,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:14:08,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:14:08,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:14:08,486 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 07:14:08,515 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:14:08,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:09,665 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:14:10,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:10,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:10,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:14:10,563 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 07:14:10,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:14:10,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:10,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:14:10,642 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 07:14:10,642 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:14:10,644 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 07:14:10,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:14:10,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 07:14:11,007 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:14:11,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 07:14:11,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 07:14:11,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 07:14:11,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 07:14:11,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 07:14:11,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 07:14:11,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 07:14:11,109 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:14:11,125 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 07:14:11,150 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:14:11,165 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:11,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:14:11,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:11,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:11,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:14:11,236 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 07:14:11,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:14:11,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:11,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:14:11,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:11,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:14:11,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:11,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:11,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:11,456 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:11,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:11,530 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:11,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:11,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:11,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:11,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:14:11,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:11,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:11,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:11,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:11,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:11,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:11,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:11,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:11,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:11,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:14:11,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:11,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:11,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:11,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:11,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:11,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:11,941 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:11,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:11,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:11,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:14:12,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:12,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:12,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:12,058 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:12,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:12,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:12,104 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:12,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:12,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:12,150 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:14:12,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:12,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:12,196 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:12,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:12,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:12,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:12,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:12,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:12,285 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:12,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:14:12,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:12,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:12,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:12,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:12,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:12,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:12,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:12,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:12,460 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 07:14:12,463 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:14:12,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:14:12,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:14:12,802 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:14:12,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:14:12,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:14:12,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:14:12,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:14:12,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:14:12,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:14:12,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:14:12,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:14:12,914 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 07:14:12,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:14:12,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:15,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:14:15,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:16,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:16,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:14:16,450 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 07:14:16,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:14:16,466 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:16,512 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:14:16,568 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 07:14:16,569 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:14:16,572 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:14:16,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:14:16,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:14:16,973 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:14:16,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:14:16,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:14:17,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:14:17,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:14:17,043 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:14:17,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:14:17,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:14:17,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:14:17,122 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 07:14:17,150 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:14:17,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:18,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:14:19,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:19,665 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:19,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:14:19,994 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 07:14:19,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:14:20,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:20,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:14:20,105 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 07:14:20,107 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:14:20,109 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 07:14:20,191 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:14:20,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 07:14:20,529 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:14:20,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 07:14:20,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 07:14:20,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 07:14:20,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 07:14:20,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 07:14:20,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 07:14:20,647 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 07:14:20,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:14:20,683 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 07:14:20,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:14:20,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:20,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:14:20,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:20,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:20,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:14:20,805 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 07:14:20,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:14:20,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:20,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:14:20,862 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:20,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:14:20,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:20,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:21,025 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:21,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:21,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:21,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:21,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:21,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:21,244 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:21,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:14:21,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:21,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:21,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:21,336 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:21,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:21,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:21,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:21,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:21,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:21,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:14:21,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:21,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:21,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:21,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:21,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:21,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:21,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:21,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:21,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:21,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:14:21,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:21,670 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:21,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:21,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:21,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:21,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:21,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:21,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:21,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:21,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:14:21,829 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:21,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:21,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:21,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:21,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:21,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:21,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:21,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:21,964 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:21,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:14:22,005 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:22,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:22,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:22,066 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:22,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:22,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:22,127 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:22,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:22,167 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 07:14:22,170 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 07:14:22,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:14:22,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 07:14:22,555 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:14:22,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 07:14:22,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 07:14:22,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 07:14:22,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 07:14:22,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 07:14:22,630 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 07:14:22,642 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 07:14:22,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:14:22,671 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 07:14:22,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:14:22,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:22,727 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:14:22,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:22,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:22,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:14:22,786 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 07:14:22,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:14:22,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:22,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:14:22,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:22,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:14:22,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:22,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:22,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:23,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:23,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:23,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:23,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:23,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:23,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:23,227 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:14:23,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:23,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:23,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:23,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:23,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:23,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:23,349 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:23,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:23,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:23,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:14:23,418 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:23,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:23,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:23,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:23,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:23,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:23,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:23,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:23,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:23,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:14:23,586 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:23,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:23,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:23,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:23,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:23,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:23,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:23,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:23,716 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:23,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:14:23,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:23,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:23,778 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:23,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:23,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:23,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:23,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:23,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:23,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:23,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:14:23,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:23,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:23,956 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:23,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:23,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:24,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:24,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:24,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:24,059 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 07:14:24,062 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:14:24,160 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:14:24,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:14:24,538 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:14:24,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:14:24,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:14:24,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:14:24,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:14:24,607 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:14:24,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:14:24,642 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:14:24,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:14:24,680 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 07:14:24,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:14:24,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:27,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:14:27,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:28,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:28,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:14:28,535 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 07:14:28,535 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:14:28,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:28,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:14:28,634 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 07:14:28,634 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:14:28,635 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:14:28,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:14:28,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:14:29,085 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:14:29,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:14:29,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:14:29,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:14:29,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:14:29,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:14:29,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:14:29,211 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:14:29,232 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:14:29,253 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 07:14:29,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:14:29,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:33,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:14:33,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:34,165 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:34,454 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:14:34,473 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 07:14:34,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:14:34,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:34,545 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:14:34,562 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 07:14:34,562 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:14:34,565 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:14:35,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:14:35,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:14:35,545 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:14:35,545 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:14:35,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:14:35,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:14:35,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:14:35,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:14:35,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:14:35,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:14:35,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:14:35,733 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 07:14:35,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:14:35,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:36,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:14:37,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:37,303 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:37,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:14:37,595 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 07:14:37,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:14:37,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:37,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:14:37,676 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 07:14:37,676 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:14:37,677 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:14:37,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:14:38,058 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:14:38,207 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:14:38,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:14:38,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:14:38,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:14:38,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:14:38,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:14:38,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:14:38,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:14:38,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:14:38,438 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 07:14:38,466 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:14:38,507 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:42,201 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:14:42,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:43,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:43,849 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:14:43,878 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 07:14:43,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:14:43,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:43,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:14:44,002 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 07:14:44,003 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:14:44,005 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 07:14:44,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:14:44,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 07:14:44,391 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:14:44,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 07:14:44,405 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 07:14:44,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 07:14:44,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 07:14:44,456 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 07:14:44,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 07:14:44,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 07:14:44,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:14:44,516 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 07:14:44,548 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:14:44,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:44,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:14:44,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:44,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:14:44,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:14:44,635 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 07:14:44,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:14:44,650 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:14:44,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:14:44,678 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:44,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:14:44,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:44,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:44,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:44,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:44,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:44,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:44,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:45,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:14:45,053 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:45,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:14:45,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:45,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:45,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:45,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:45,170 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:45,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:45,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:45,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:14:45,242 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:45,257 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:14:45,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:45,295 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:45,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:45,329 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:45,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:45,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:45,385 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:45,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:14:45,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:45,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:14:45,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:45,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:45,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:45,515 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:45,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:45,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:45,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:45,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:14:45,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:45,619 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:14:45,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:45,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:45,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:45,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:45,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:45,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:45,765 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:45,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:14:45,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:45,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:14:45,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:45,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:45,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:45,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:45,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:45,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:45,956 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:45,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:14:45,990 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 07:14:45,992 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:14:45,993 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:14:46,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:14:46,341 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:14:46,421 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:14:46,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:14:46,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:14:46,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:14:46,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:14:46,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:14:46,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:14:46,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:14:46,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:14:47,088 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:14:47,088 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.09599995613. 2022-02-08 07:14:47,088 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:14:47,089 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:14:47,214 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:14:47,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:14:47,701 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:14:47,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:14:47,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:14:47,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:14:47,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:14:47,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:14:47,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:14:47,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:14:47,920 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:14:48,137 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:14:48,138 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 9490 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:14:50,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:14:50,482 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:14:50,482 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:14:52,487 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:14:52,865 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:14:52,865 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:14:54,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:14:55,073 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:14:55,073 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:14:57,073 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:14:57,311 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:14:57,312 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:14:59,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:14:59,688 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:14:59,688 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:15:01,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:01,996 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:15:01,996 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:15:03,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:04,253 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:15:04,253 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:15:06,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:06,543 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:15:06,545 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:15:08,545 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:08,822 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:15:08,823 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:15:10,823 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:11,009 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:15:11,009 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:15:13,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:13,220 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:15:13,220 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:15:15,221 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:15,506 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:15:15,507 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:15:17,507 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:17,785 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:15:17,785 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:15:19,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:20,121 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:15:20,121 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:15:22,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:22,381 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:15:22,381 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:15:24,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:24,576 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:15:24,576 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:15:26,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:26,861 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:15:26,861 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:15:28,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:29,084 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:15:29,084 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 41.995000124. 2022-02-08 07:15:29,084 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:15:29,085 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:15:30,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:15:30,739 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:15:30,969 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:15:30,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:15:31,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:15:31,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:15:31,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:15:31,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:15:31,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:15:31,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:15:31,545 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:32,407 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:15:32,407 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 07:15:34,408 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:34,608 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:15:34,609 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 07:15:36,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:36,802 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:15:36,802 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 07:15:38,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:38,960 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:15:38,961 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 9.87799978256. 2022-02-08 07:15:38,961 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:15:38,963 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:15:39,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:15:40,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:15:40,180 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:15:40,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:15:40,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:15:40,219 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:15:40,234 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:15:40,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:15:40,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:15:40,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:15:40,349 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:41,072 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:15:41,072 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.11000013351. 2022-02-08 07:15:41,072 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:15:41,073 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:15:41,780 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:15:42,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:15:42,178 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:15:42,178 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:15:42,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:15:42,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:15:42,234 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:15:42,255 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:15:42,272 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:15:42,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:15:42,326 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:42,987 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:15:42,989 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.91700005531. 2022-02-08 07:15:42,989 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:15:42,990 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:15:43,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:15:44,441 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:15:44,651 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:15:44,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:15:44,670 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:15:44,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:15:44,709 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:15:44,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:15:44,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:15:44,780 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:15:44,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:45,351 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:15:45,351 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.36299991608. 2022-02-08 07:15:45,351 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:15:45,354 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:15:45,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:15:46,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:15:46,372 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:15:46,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:15:46,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:15:46,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:15:46,418 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:15:46,436 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:15:46,448 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:15:46,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:15:46,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:46,937 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:15:46,937 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.58599996567. 2022-02-08 07:15:46,937 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:15:46,940 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:15:47,581 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:15:47,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:15:48,000 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:15:48,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:15:48,022 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:15:48,068 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:15:48,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:15:48,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:15:48,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:15:48,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:15:48,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:49,309 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:15:49,309 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.37199997902. 2022-02-08 07:15:49,311 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:15:49,313 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 5 queries submitted, 0 failed, 9499 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:15:50,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:15:50,446 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:15:50,618 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:15:50,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:15:50,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:15:50,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:15:50,681 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:15:50,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:15:50,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:15:50,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:15:50,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:51,612 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:15:51,612 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.30199980736. 2022-02-08 07:15:51,612 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:15:51,615 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:15:52,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:15:52,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:15:52,862 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:15:52,862 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:15:52,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:15:52,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:15:52,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:15:53,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:15:53,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:15:53,117 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:15:53,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:15:53,930 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:15:53,930 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.3180000782. ############################################################ # # RAM FootPrint: 4392.0859375 # ############################################################ 2022-02-08 07:15:53,963 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 07:15:53,964 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 92 seconds. Reason: Iteration:17 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 9521 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:17:25,970 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 07:17:25,971 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 07:17:30,973 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:17:31,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:31,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:17:31,371 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:31,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:17:31,387 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:17:31,415 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:17:31,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:17:31,464 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:17:31,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:17:31,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:17:31,519 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:17:31,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 07:17:31,601 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['21454 memcached\n'], []) 2022-02-08 07:17:31,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 07:17:31,658 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 07:17:31,687 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['21454 memcached\n'], []) 2022-02-08 07:17:31,687 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:17:31,691 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:17:32,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:32,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:17:32,753 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:32,755 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:17:32,767 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:17:32,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:17:32,798 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:17:32,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:17:32,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:17:32,859 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:17:32,874 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:17:32,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 07:17:32,927 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['71154 memcached\n'], []) 2022-02-08 07:17:32,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 07:17:32,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 07:17:32,974 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['71154 memcached\n'], []) 2022-02-08 07:17:32,974 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:17:32,976 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:17:33,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:33,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:17:33,332 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:33,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:17:33,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:17:33,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:17:33,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:17:33,392 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:17:33,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:17:33,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:17:33,434 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:17:33,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 07:17:33,476 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['55204 memcached\n'], []) 2022-02-08 07:17:33,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 07:17:33,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 07:17:33,523 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['55204 memcached\n'], []) 2022-02-08 07:17:33,525 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:17:33,526 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:17:33,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:33,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:17:33,910 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:33,910 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:17:33,923 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:17:33,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:17:33,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:17:33,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:17:33,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:17:33,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:17:34,006 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:17:34,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 07:17:34,049 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['86002 memcached\n'], []) 2022-02-08 07:17:34,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 07:17:34,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 07:17:34,095 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['86002 memcached\n'], []) 2022-02-08 07:17:34,095 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:17:34,096 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:17:34,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:34,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:17:34,456 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:34,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:17:34,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:17:34,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:17:34,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:17:34,512 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:17:34,525 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:17:34,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:17:34,552 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:17:34,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 07:17:34,598 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['79889 memcached\n'], []) 2022-02-08 07:17:34,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 07:17:34,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 07:17:34,644 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['79889 memcached\n'], []) 2022-02-08 07:17:34,644 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:17:34,648 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:17:34,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:34,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:17:35,036 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:35,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:17:35,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:17:35,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:17:35,073 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:17:35,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:17:35,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:17:35,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:17:35,128 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:17:35,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 07:17:35,173 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['114873 memcached\n'], []) 2022-02-08 07:17:35,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 07:17:35,198 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 07:17:35,220 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['114873 memcached\n'], []) 2022-02-08 07:17:35,220 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:17:35,223 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:17:35,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:35,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:17:35,628 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:35,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:17:35,642 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:17:35,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:17:35,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:17:35,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:17:35,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:17:35,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:17:35,733 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:17:35,734 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 07:17:35,786 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['48709 memcached\n'], []) 2022-02-08 07:17:35,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 07:17:35,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 07:17:35,855 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['48709 memcached\n'], []) 2022-02-08 07:17:35,857 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:17:35,859 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:17:35,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:36,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:17:36,227 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:36,229 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:17:36,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:17:36,255 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:17:36,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:17:36,283 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:17:36,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:17:36,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:17:36,322 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:17:36,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 07:17:36,390 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['63664 memcached\n'], []) 2022-02-08 07:17:36,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 07:17:36,415 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 07:17:36,437 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['63664 memcached\n'], []) 2022-02-08 07:17:36,437 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:17:36,440 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:17:36,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:36,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:17:36,813 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:36,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:17:36,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:17:36,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:17:36,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:17:36,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:17:36,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:17:36,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:17:36,911 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:17:36,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 07:17:36,957 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['112994 memcached\n'], []) 2022-02-08 07:17:36,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 07:17:36,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 07:17:37,009 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['112994 memcached\n'], []) 2022-02-08 07:17:37,009 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:17:37,013 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:17:37,091 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:37,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:17:37,394 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:37,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:17:37,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:17:37,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:17:37,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:17:37,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:17:37,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:17:37,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:17:37,493 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:17:37,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 07:17:37,553 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['129683 memcached\n'], []) 2022-02-08 07:17:37,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 07:17:37,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 07:17:37,601 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['129683 memcached\n'], []) 2022-02-08 07:17:37,601 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:17:37,604 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:17:37,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:37,923 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:17:38,002 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:38,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:17:38,017 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:17:38,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:17:38,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:17:38,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:17:38,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:17:38,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:17:38,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:17:38,115 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 07:17:38,140 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:17:38,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:17:38,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:17:39,413 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:17:39,765 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:17:40,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:17:40,134 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 07:17:40,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:17:40,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:17:40,214 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:17:40,230 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 07:17:40,230 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:17:40,232 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:17:40,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:40,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:17:40,607 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:40,607 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:17:40,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:17:40,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:17:40,647 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:17:40,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:17:40,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:17:40,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:17:40,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:17:40,737 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 07:17:40,767 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:17:40,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:17:41,341 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:17:41,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:17:42,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:17:42,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:17:42,591 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 07:17:42,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:17:42,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:17:42,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:17:42,674 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 07:17:42,674 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:17:42,677 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:17:42,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:42,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:17:43,026 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:43,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:17:43,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:17:43,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:17:43,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:17:43,081 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:17:43,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:17:43,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:17:43,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:17:43,138 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 07:17:43,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:17:43,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:17:43,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:17:44,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:17:44,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:17:44,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:17:44,615 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 07:17:44,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:17:44,630 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:17:44,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:17:44,714 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 07:17:44,714 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:17:44,717 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 07:17:44,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:45,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 07:17:45,164 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:45,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 07:17:45,178 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 07:17:45,193 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 07:17:45,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 07:17:45,224 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 07:17:45,239 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 07:17:45,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 07:17:45,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:17:45,285 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 07:17:45,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:17:45,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:17:45,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:17:45,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:17:45,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:17:45,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:17:45,408 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 07:17:45,408 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:17:45,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:17:45,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:17:45,456 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:45,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:17:45,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:45,566 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:45,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:45,640 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:45,681 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:45,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:45,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:45,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:45,829 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:45,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:17:45,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:45,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:45,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:45,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:45,943 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:45,961 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:45,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:45,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:46,015 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:46,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:17:46,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:46,068 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:46,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:46,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:46,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:46,142 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:46,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:46,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:46,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:46,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:17:46,229 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:46,249 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:46,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:46,280 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:46,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:46,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:46,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:46,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:46,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:46,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:17:46,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:46,415 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:46,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:46,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:46,466 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:46,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:46,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:46,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:46,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:46,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:17:46,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:46,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:46,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:46,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:46,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:46,658 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:46,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:46,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:46,720 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 07:17:46,721 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:17:47,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:47,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:17:47,953 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:47,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:17:48,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:17:48,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:17:48,155 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:17:48,290 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:17:48,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:17:48,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:17:48,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:17:48,851 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 07:17:48,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:17:48,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 9546 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:17:50,548 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:17:51,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:17:51,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:17:51,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:17:51,696 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 07:17:51,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:17:51,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:17:51,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:17:51,821 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 07:17:51,822 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:17:51,823 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 07:17:51,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:52,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 07:17:52,197 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:52,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 07:17:52,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 07:17:52,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 07:17:52,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 07:17:52,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 07:17:52,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 07:17:52,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 07:17:52,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:17:52,318 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 07:17:52,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:17:52,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:17:52,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:17:52,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:17:52,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:17:52,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:17:52,433 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 07:17:52,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:17:52,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:17:52,464 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:17:52,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:52,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:17:52,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:52,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:52,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:52,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:52,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:52,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:52,778 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:52,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:17:52,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:52,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:17:52,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:52,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:52,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:52,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:52,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:52,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:52,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:53,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:17:53,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:53,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:17:53,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:53,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:53,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:53,119 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:53,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:53,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:53,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:53,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:17:53,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:53,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:17:53,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:53,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:53,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:53,285 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:53,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:53,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:53,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:53,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:17:53,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:53,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:17:53,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:53,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:53,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:53,446 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:53,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:53,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:53,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:53,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:17:53,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:53,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:17:53,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:53,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:53,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:53,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:53,624 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:53,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:53,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:53,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:17:53,691 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 07:17:53,694 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:17:53,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:53,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:17:54,051 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:54,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:17:54,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:17:54,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:17:54,091 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:17:54,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:17:54,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:17:54,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:17:54,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:17:54,158 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 07:17:54,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:17:54,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:17:58,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:17:58,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:17:59,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:17:59,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:17:59,437 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 07:17:59,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:17:59,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:17:59,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:17:59,516 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 07:17:59,516 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:17:59,519 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:17:59,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:17:59,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:17:59,907 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:17:59,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:17:59,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:17:59,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:17:59,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:17:59,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:17:59,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:18:00,005 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:18:00,022 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:18:00,039 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 07:18:00,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:18:00,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:18:02,377 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:18:02,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:18:03,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:18:03,602 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:18:03,638 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 07:18:03,640 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:18:03,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:18:03,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:18:03,773 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 07:18:03,773 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:18:03,776 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 07:18:03,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:18:04,099 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 07:18:04,163 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:18:04,163 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 07:18:04,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 07:18:04,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 07:18:04,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 07:18:04,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 07:18:04,239 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 07:18:04,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 07:18:04,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:18:04,289 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 07:18:04,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:18:04,331 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:18:04,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:18:04,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:18:04,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:18:04,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:18:04,408 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 07:18:04,410 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:18:04,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:18:04,438 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:18:04,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:04,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:18:04,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:04,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:04,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:04,647 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:04,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:04,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:04,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:04,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:04,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:04,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:18:04,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:04,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:04,928 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:04,948 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:04,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:04,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:05,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:05,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:05,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:05,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:18:05,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:05,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:05,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:05,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:05,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:05,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:05,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:05,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:05,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:05,244 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:18:05,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:05,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:05,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:05,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:05,336 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:05,357 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:05,377 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:05,397 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:05,415 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:05,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:18:05,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:05,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:05,487 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:05,505 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:05,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:05,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:05,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:05,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:05,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:05,614 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:18:05,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:05,665 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:05,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:05,711 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:05,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:05,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:05,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:05,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:05,812 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 07:18:05,815 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 07:18:05,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:18:06,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 07:18:06,194 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:18:06,196 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 07:18:06,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 07:18:06,224 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 07:18:06,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 07:18:06,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 07:18:06,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 07:18:06,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 07:18:06,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:18:06,315 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 07:18:06,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:18:06,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:18:06,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:18:06,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:18:06,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:18:06,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:18:06,431 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 07:18:06,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:18:06,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:18:06,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:18:06,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:06,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:18:06,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:06,525 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:06,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:06,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:06,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:06,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:06,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:06,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:06,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:06,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:18:06,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:06,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:06,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:06,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:06,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:06,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:06,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:06,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:06,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:06,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:18:06,877 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:06,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:06,908 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:06,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:06,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:06,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:06,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:06,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:07,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:07,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:18:07,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:07,056 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:07,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:07,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:07,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:07,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:07,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:07,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:07,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:07,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:18:07,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:07,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:07,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:07,247 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:07,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:07,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:07,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:07,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:07,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:07,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:18:07,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:07,377 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:07,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:07,408 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:07,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:07,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:07,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:07,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:07,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:18:07,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 07:18:07,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:18:07,548 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:18:07,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:18:07,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:18:07,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:18:07,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:18:07,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:18:07,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:18:07,684 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 07:18:07,687 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:18:07,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:18:08,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:18:08,092 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:18:08,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:18:08,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:18:08,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:18:08,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:18:08,155 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:18:08,170 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:18:08,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:18:08,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:18:08,227 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 07:18:08,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:18:08,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:18:10,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:18:11,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:18:11,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:18:11,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:18:11,878 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 07:18:11,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:18:11,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:18:11,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:18:11,986 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 07:18:11,986 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:18:11,989 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:18:12,127 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:18:12,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:18:12,786 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:18:12,788 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:18:12,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:18:12,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:18:12,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:18:12,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:18:12,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:18:12,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:18:12,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:18:13,019 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 07:18:13,045 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:18:13,219 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:18:13,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:18:14,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:18:15,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:18:16,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:18:16,147 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 07:18:16,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:18:16,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:18:16,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:18:16,224 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 07:18:16,224 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:18:16,226 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:18:16,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:18:17,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:18:17,269 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:18:17,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:18:17,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:18:17,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:18:17,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:18:17,357 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:18:17,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:18:17,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:18:17,413 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:18:17,437 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 07:18:17,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:18:17,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:18:20,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:18:21,257 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:18:21,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:18:21,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:18:21,845 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 07:18:21,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:18:21,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:18:21,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:18:21,948 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 07:18:21,950 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:18:21,951 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:18:22,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:18:22,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:18:22,355 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:18:22,357 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:18:22,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:18:22,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:18:22,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:18:22,426 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:18:22,441 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:18:22,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:18:22,474 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:18:22,493 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 07:18:22,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:18:22,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:18:23,448 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:18:24,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:18:24,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:18:24,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:18:24,973 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 07:18:24,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:18:24,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:18:25,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:18:25,052 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 07:18:25,053 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:18:25,055 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 07:18:25,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:18:25,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 07:18:25,467 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:18:25,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 07:18:25,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 07:18:25,497 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 07:18:25,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 07:18:25,528 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 07:18:25,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 07:18:25,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 07:18:25,571 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:18:25,586 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 07:18:25,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:18:25,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:18:25,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:18:25,660 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:18:25,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:18:25,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:18:25,700 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 07:18:25,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:18:25,714 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:18:25,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:18:25,744 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:25,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:18:25,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:25,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:25,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:25,928 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:25,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:26,028 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:26,068 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:26,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:18:26,150 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:26,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:18:26,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:26,204 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:26,221 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:26,239 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:26,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:26,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:26,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:26,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:18:26,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:26,341 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:18:26,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:26,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:26,392 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:26,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:26,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:26,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:26,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:26,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:18:26,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:26,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:18:26,528 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:26,543 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:26,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:26,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:26,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:26,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:26,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:26,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:18:26,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:26,678 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:18:26,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:26,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:26,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:26,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:26,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:26,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:26,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:26,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:18:26,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:26,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:18:26,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:26,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:26,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:26,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:26,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:26,979 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:27,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:27,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:18:27,038 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 07:18:27,038 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:18:27,039 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:18:27,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:18:27,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:18:27,443 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:18:27,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:18:27,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:18:27,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:18:27,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:18:27,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:18:27,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:18:27,543 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:18:27,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:18:28,086 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:18:28,086 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.04899978638. 2022-02-08 07:18:28,088 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:18:28,088 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:18:28,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:18:28,586 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:18:28,697 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:18:28,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:18:28,721 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:18:28,749 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:18:28,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:18:28,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:18:28,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:18:28,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:18:28,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:18:29,092 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:18:29,092 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:18:31,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:18:31,269 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:18:31,269 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:18:33,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:18:33,530 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:18:33,530 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:18:35,530 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:18:35,713 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:18:35,713 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:18:37,714 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:18:37,905 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:18:37,905 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:18:39,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:18:40,096 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:18:40,098 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:18:42,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:18:42,338 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:18:42,338 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:18:44,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:18:44,509 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:18:44,509 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:18:46,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:18:46,691 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:18:46,691 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:18:48,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:18:48,907 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:18:48,907 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 9549 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:18:50,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:18:51,263 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:18:51,263 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:18:53,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:18:53,684 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:18:53,684 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:18:55,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:18:56,046 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:18:56,046 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:18:58,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:18:58,345 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:18:58,345 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:19:00,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:00,592 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:19:00,592 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:19:02,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:02,828 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:19:02,828 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:19:04,829 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:05,075 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:19:05,075 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:19:07,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:07,289 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:19:07,289 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:19:09,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:09,555 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:19:09,555 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:19:11,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:11,798 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:19:11,799 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:19:13,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:14,092 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:19:14,092 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:19:16,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:16,371 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:19:16,371 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:19:18,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:18,588 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:19:18,588 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:19:20,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:20,789 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:19:20,789 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:19:22,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:22,983 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:19:22,983 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:19:24,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:25,168 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:19:25,170 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 57.0820000172. 2022-02-08 07:19:25,170 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:19:25,171 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:19:26,308 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:19:26,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:19:27,030 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:19:27,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:19:27,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:19:27,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:19:27,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:19:27,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:19:27,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:19:27,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:19:27,211 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:27,651 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:19:27,651 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.48199987411. 2022-02-08 07:19:27,651 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:19:27,654 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:19:28,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:19:29,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:19:29,286 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:19:29,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:19:29,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:19:29,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:19:29,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:19:29,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:19:29,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:19:29,428 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:19:29,446 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:30,112 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:19:30,114 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.46200013161. 2022-02-08 07:19:30,114 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:19:30,115 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:19:31,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:19:31,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:19:31,776 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:19:31,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:19:31,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:19:31,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:19:31,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:19:31,859 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:19:31,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:19:31,923 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:19:31,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:32,549 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:19:32,549 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.4359998703. 2022-02-08 07:19:32,549 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:19:32,552 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:19:33,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:19:33,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:19:34,026 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:19:34,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:19:34,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:19:34,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:19:34,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:19:34,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:19:34,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:19:34,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:19:34,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:35,042 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:19:35,042 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.49300003052. 2022-02-08 07:19:35,043 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:19:35,045 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:19:35,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:19:35,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:19:35,993 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:19:35,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:19:36,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:19:36,035 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:19:36,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:19:36,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:19:36,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:19:36,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:19:36,119 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:36,948 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:19:36,948 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.90499997139. 2022-02-08 07:19:36,950 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:19:36,951 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:19:37,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:19:37,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:19:37,857 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:19:37,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:19:37,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:19:37,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:19:37,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:19:37,948 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:19:37,969 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:19:38,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:19:38,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:39,115 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:19:39,115 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.16700005531. 2022-02-08 07:19:39,115 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:19:39,118 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:19:39,647 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:19:39,933 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:19:40,030 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:19:40,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:19:40,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:19:40,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:19:40,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:19:40,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:19:40,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:19:40,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:19:40,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:41,250 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:19:41,252 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.1360001564. 2022-02-08 07:19:41,252 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:19:41,253 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:19:41,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:19:42,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:19:42,407 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:19:42,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:19:42,428 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:19:42,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:19:42,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:19:42,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:19:42,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:19:42,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:19:42,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:19:44,503 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:19:44,505 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.25300002098. ############################################################ # # RAM FootPrint: 4385.4375 # ############################################################ 2022-02-08 07:19:44,535 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 07:19:44,536 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 81 seconds. Reason: Iteration:18 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 9559 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 9585 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:21:05,540 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 07:21:05,542 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 07:21:10,545 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:21:10,660 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:10,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:21:10,967 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:10,969 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:21:10,989 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:21:11,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:21:11,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:21:11,056 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:21:11,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:21:11,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:21:11,112 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:21:11,112 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 07:21:11,177 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['22168 memcached\n'], []) 2022-02-08 07:21:11,178 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 07:21:11,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 07:21:11,236 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['22168 memcached\n'], []) 2022-02-08 07:21:11,236 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:21:11,239 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:21:11,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:11,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:21:12,013 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:12,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:21:12,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:21:12,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:21:12,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:21:12,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:21:12,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:21:12,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:21:12,150 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:21:12,150 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 07:21:12,203 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['72085 memcached\n'], []) 2022-02-08 07:21:12,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 07:21:12,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 07:21:12,246 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['72085 memcached\n'], []) 2022-02-08 07:21:12,247 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:21:12,249 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:21:12,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:12,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:21:12,638 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:12,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:21:12,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:21:12,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:21:12,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:21:12,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:21:12,719 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:21:12,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:21:12,750 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:21:12,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 07:21:12,799 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['56055 memcached\n'], []) 2022-02-08 07:21:12,801 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 07:21:12,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 07:21:12,852 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['56055 memcached\n'], []) 2022-02-08 07:21:12,852 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:21:12,855 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:21:12,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:13,174 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:21:13,233 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:13,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:21:13,249 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:21:13,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:21:13,280 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:21:13,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:21:13,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:21:13,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:21:13,352 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:21:13,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 07:21:13,405 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['86719 memcached\n'], []) 2022-02-08 07:21:13,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 07:21:13,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 07:21:13,469 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['86719 memcached\n'], []) 2022-02-08 07:21:13,469 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:21:13,471 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:21:13,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:13,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:21:13,825 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:13,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:21:13,841 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:21:13,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:21:13,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:21:13,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:21:13,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:21:13,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:21:13,923 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:21:13,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 07:21:13,990 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['80606 memcached\n'], []) 2022-02-08 07:21:13,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 07:21:14,015 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 07:21:14,042 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['80606 memcached\n'], []) 2022-02-08 07:21:14,043 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:21:14,046 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:21:14,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:14,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:21:14,441 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:14,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:21:14,456 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:21:14,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:21:14,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:21:14,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:21:14,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:21:14,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:21:14,535 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:21:14,535 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 07:21:14,581 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['115590 memcached\n'], []) 2022-02-08 07:21:14,581 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 07:21:14,607 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 07:21:14,628 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['115590 memcached\n'], []) 2022-02-08 07:21:14,628 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:21:14,630 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:21:14,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:14,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:21:15,072 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:15,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:21:15,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:21:15,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:21:15,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:21:15,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:21:15,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:21:15,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:21:15,167 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:21:15,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 07:21:15,213 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['49422 memcached\n'], []) 2022-02-08 07:21:15,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 07:21:15,239 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 07:21:15,259 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['49422 memcached\n'], []) 2022-02-08 07:21:15,260 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:21:15,263 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:21:15,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:15,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:21:15,648 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:15,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:21:15,663 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:21:15,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:21:15,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:21:15,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:21:15,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:21:15,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:21:15,750 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:21:15,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 07:21:15,806 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['64347 memcached\n'], []) 2022-02-08 07:21:15,806 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 07:21:15,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 07:21:15,857 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['64347 memcached\n'], []) 2022-02-08 07:21:15,857 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:21:15,859 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:21:15,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:16,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:21:16,220 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:16,221 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:21:16,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:21:16,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:21:16,272 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:21:16,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:21:16,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:21:16,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:21:16,325 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:21:16,326 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 07:21:16,365 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['113712 memcached\n'], []) 2022-02-08 07:21:16,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 07:21:16,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 07:21:16,408 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['113712 memcached\n'], []) 2022-02-08 07:21:16,408 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:21:16,411 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:21:16,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:16,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:21:16,782 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:16,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:21:16,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:21:16,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:21:16,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:21:16,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:21:16,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:21:16,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:21:16,891 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:21:16,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 07:21:16,947 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['130400 memcached\n'], []) 2022-02-08 07:21:16,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 07:21:16,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 07:21:16,993 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['130400 memcached\n'], []) 2022-02-08 07:21:16,993 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:21:16,994 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:21:17,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:17,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:21:17,374 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:17,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:21:17,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:21:17,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:21:17,415 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:21:17,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:21:17,446 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:21:17,459 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:21:17,471 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:21:17,486 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 07:21:17,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:21:17,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:18,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:21:19,362 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:19,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:20,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:21:20,086 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 07:21:20,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:21:20,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:20,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:21:20,177 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 07:21:20,178 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:21:20,181 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:21:20,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:20,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:21:20,559 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:20,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:21:20,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:21:20,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:21:20,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:21:20,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:21:20,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:21:20,645 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:21:20,660 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:21:20,677 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 07:21:20,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:21:20,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:25,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:21:26,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:26,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:26,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:21:26,992 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 07:21:26,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:21:27,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:27,073 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:21:27,088 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 07:21:27,088 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:21:27,091 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:21:27,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:27,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:21:27,433 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:27,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:21:27,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:21:27,459 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:21:27,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:21:27,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:21:27,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:21:27,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:21:27,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:21:27,549 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 07:21:27,579 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:21:27,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:28,002 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:21:28,466 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:28,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:28,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:21:28,994 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 07:21:28,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:21:29,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:29,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:21:29,091 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 07:21:29,092 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:21:29,094 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 07:21:29,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:29,408 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 07:21:29,469 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:29,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 07:21:29,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 07:21:29,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 07:21:29,512 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 07:21:29,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 07:21:29,540 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 07:21:29,553 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 07:21:29,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:21:29,582 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 07:21:29,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:21:29,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:29,655 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:21:29,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:29,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:29,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:21:29,714 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 07:21:29,714 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:21:29,729 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:29,744 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:21:29,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:29,795 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:21:29,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:29,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:29,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:29,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:29,969 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:30,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:30,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:30,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:30,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:30,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:21:30,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:30,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:30,186 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:30,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:30,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:30,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:30,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:30,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:30,293 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:30,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:21:30,331 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:30,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:30,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:30,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:30,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:30,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:30,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:30,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:30,466 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:30,487 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:21:30,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:30,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:30,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:30,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:30,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:30,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:30,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:30,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:30,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:30,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:21:30,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:30,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:30,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:30,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:30,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:30,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:30,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:30,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:30,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:30,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:21:30,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:30,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:30,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:30,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:30,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:30,936 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:30,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:30,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:30,986 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 07:21:30,989 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:21:31,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:31,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:21:31,608 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:31,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:21:31,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:21:31,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:21:31,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:21:31,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:21:32,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:21:32,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:21:32,150 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:21:32,229 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 07:21:32,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:21:32,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:33,005 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:21:33,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:33,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:34,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:21:34,137 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 07:21:34,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:21:34,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:34,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:21:34,336 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 07:21:34,338 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:21:34,339 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 07:21:34,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:34,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 07:21:34,730 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:34,732 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 07:21:34,744 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 07:21:34,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 07:21:34,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 07:21:34,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 07:21:34,798 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 07:21:34,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 07:21:34,823 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:21:34,838 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 07:21:34,862 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:21:34,877 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:34,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:21:34,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:34,918 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:34,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:21:34,944 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 07:21:34,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:21:34,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:34,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:21:34,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:34,997 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:21:35,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:35,025 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:35,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:35,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:35,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:35,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:35,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:35,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:35,119 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:35,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:21:35,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:35,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:35,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:35,198 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:35,214 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:35,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:35,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:35,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:35,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:35,293 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:21:35,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:35,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:35,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:35,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:35,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:35,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:35,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:35,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:35,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:35,454 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:21:35,471 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:35,487 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:35,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:35,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:35,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:35,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:35,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:35,581 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:35,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:35,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:21:35,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:35,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:35,660 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:35,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:35,688 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:35,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:35,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:35,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:35,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:35,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:21:35,783 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:35,798 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:35,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:35,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:35,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:35,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:35,877 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:35,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:35,913 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:35,948 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 07:21:35,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:35,987 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:36,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:36,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:36,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:36,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:36,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:36,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:36,096 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 07:21:36,098 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:21:36,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:36,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:21:36,476 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:36,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:21:36,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:21:36,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:21:36,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:21:36,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:21:36,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:21:36,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:21:36,579 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:21:36,598 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 07:21:36,624 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:21:36,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:39,190 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:21:39,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:40,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:40,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:21:40,548 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 07:21:40,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:21:40,563 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:40,619 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:21:40,635 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 07:21:40,635 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:21:40,640 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:21:40,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:40,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:21:41,023 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:41,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:21:41,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:21:41,053 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:21:41,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:21:41,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:21:41,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:21:41,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:21:41,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:21:41,157 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 07:21:41,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:21:41,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:43,693 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:21:44,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:44,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:44,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:21:44,898 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 07:21:44,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:21:44,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:44,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:21:45,000 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 07:21:45,000 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:21:45,002 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 07:21:45,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:45,336 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 07:21:45,404 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:45,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 07:21:45,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 07:21:45,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 07:21:45,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 07:21:45,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 07:21:45,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 07:21:45,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 07:21:45,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:21:45,532 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 07:21:45,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:21:45,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:45,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:21:45,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:45,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:45,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:21:45,664 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 07:21:45,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:21:45,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:45,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:21:45,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:45,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:21:45,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:45,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:45,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:45,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:45,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:46,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:46,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:46,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:46,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:46,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:21:46,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:46,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:46,201 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:46,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:46,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:46,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:46,270 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:46,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:46,303 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:46,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:21:46,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:46,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:46,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:46,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:46,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:46,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:46,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:46,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:46,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:46,507 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:21:46,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:46,543 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:46,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:46,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:46,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:46,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:46,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:46,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:46,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:46,681 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:21:46,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:46,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:46,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:46,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:46,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:46,795 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:46,813 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:46,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:46,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:46,877 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:21:46,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:46,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:46,943 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:46,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:46,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:47,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:47,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:47,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:47,059 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 07:21:47,062 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 07:21:47,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:47,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 07:21:47,417 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:47,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 07:21:47,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 07:21:47,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 07:21:47,459 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 07:21:47,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 07:21:47,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 07:21:47,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 07:21:47,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:21:47,538 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 07:21:47,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:21:47,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:47,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:21:47,607 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:47,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:47,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:21:47,653 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 07:21:47,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:21:47,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:47,683 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:21:47,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:47,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:21:47,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:47,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:47,760 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:47,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:47,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:47,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:47,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:47,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:21:47,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:47,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:21:47,895 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:47,913 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:47,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:47,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:47,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:47,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:48,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:48,022 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:21:48,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:48,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:21:48,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:48,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:48,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:48,145 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:48,163 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:48,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:48,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:48,214 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:21:48,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:48,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:21:48,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:48,280 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:48,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:48,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:48,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:48,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:48,359 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:48,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:21:48,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:48,405 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:21:48,423 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:48,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:48,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:48,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:48,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:48,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:48,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:48,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:21:48,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:48,573 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:21:48,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:48,607 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:48,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:48,637 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:48,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:48,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:48,683 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:48,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:21:48,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:48,732 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 07:21:48,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:48,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:48,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:48,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:48,818 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:48,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:48,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:48,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:21:48,894 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 07:21:48,895 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:21:48,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:49,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:21:49,328 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:49,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:21:49,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:21:49,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:21:49,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:21:49,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:21:49,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:21:49,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:21:49,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:21:49,463 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 07:21:49,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:21:49,507 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 9603 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:21:52,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:21:52,765 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:53,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:53,454 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:21:53,473 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 07:21:53,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:21:53,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:53,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:21:53,573 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 07:21:53,575 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:21:53,576 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:21:53,795 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:54,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:21:55,000 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:55,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:21:55,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:21:55,339 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:21:55,466 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:21:55,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:21:55,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:21:55,920 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:21:56,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:21:56,142 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 07:21:56,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:21:56,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:56,798 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:21:57,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:57,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:21:58,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:21:58,122 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 07:21:58,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:21:58,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:21:58,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:21:58,223 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 07:21:58,224 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:21:58,226 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:21:58,747 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:21:59,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:21:59,138 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:21:59,140 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:21:59,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:21:59,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:21:59,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:21:59,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:21:59,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:21:59,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:21:59,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:21:59,329 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 07:21:59,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:21:59,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:22:00,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:22:00,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:22:01,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:22:01,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:22:01,510 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 07:22:01,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:22:01,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:22:01,579 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:22:01,599 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 07:22:01,601 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:22:01,602 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:22:01,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:22:02,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:22:02,230 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:22:02,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:22:02,249 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:22:02,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:22:02,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:22:02,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:22:02,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:22:02,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:22:02,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:22:02,392 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 07:22:02,428 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:22:02,471 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:22:07,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:22:08,614 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:22:09,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:22:09,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:22:09,454 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 07:22:09,456 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:22:09,528 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:22:09,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:22:09,612 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 07:22:09,614 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:22:09,615 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 07:22:09,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:22:09,918 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 07:22:09,982 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:22:09,982 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 07:22:09,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 07:22:10,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 07:22:10,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 07:22:10,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 07:22:10,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 07:22:10,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 07:22:10,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:22:10,107 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 07:22:10,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:22:10,150 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:22:10,165 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:22:10,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:22:10,196 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:22:10,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:22:10,223 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 07:22:10,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:22:10,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:22:10,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:22:10,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:22:10,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:22:10,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:22:10,329 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:22:10,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:22:10,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:22:10,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:22:10,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:22:10,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:22:10,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:22:10,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:22:10,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:22:10,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:22:10,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:22:10,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:22:10,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:22:10,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:22:10,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:22:10,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:22:10,614 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:22:10,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:22:10,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:22:10,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:22:10,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:22:10,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:22:10,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:22:10,742 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:22:10,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:22:10,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:22:10,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:22:10,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:22:10,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:22:10,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:22:10,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:22:10,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:22:10,903 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:22:10,920 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:22:10,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:22:10,954 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:22:10,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:22:10,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:22:11,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:22:11,022 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:22:11,038 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:22:11,053 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:22:11,071 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:22:11,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:22:11,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:22:11,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:22:11,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:22:11,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:22:11,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:22:11,188 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:22:11,204 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:22:11,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:22:11,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:22:11,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:22:11,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:22:11,285 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:22:11,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:22:11,316 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:22:11,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 07:22:11,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:22:11,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:22:11,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:22:11,405 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:22:11,424 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:22:11,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:22:11,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:22:11,482 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:22:11,500 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 07:22:11,500 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:22:11,502 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:22:11,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:22:11,829 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:22:11,898 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:22:11,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:22:11,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:22:11,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:22:11,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:22:11,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:22:12,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:22:12,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:22:12,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:12,520 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:22:12,520 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.02099990845. 2022-02-08 07:22:12,520 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:22:12,523 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:22:12,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:22:12,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:22:12,976 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:22:12,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:22:12,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:22:13,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:22:13,035 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:22:13,053 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:22:13,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:22:13,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:22:13,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:13,263 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:13,265 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:15,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:15,415 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:15,417 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:17,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:17,594 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:17,594 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:19,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:19,756 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:19,756 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:21,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:21,976 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:21,976 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:23,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:24,244 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:24,244 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:26,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:26,466 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:26,467 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:28,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:28,773 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:28,773 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:30,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:31,065 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:31,065 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:33,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:33,319 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:33,319 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:35,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:35,586 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:35,586 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:37,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:37,946 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:37,947 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:39,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:40,236 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:40,236 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:42,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:42,480 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:42,480 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:44,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:44,786 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:44,786 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:46,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:47,131 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:47,131 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:49,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 9611 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:22:49,551 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:49,551 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:51,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:51,802 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:51,802 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:53,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:54,138 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:54,138 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:56,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:56,598 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:56,598 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:22:58,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:22:59,003 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:22:59,005 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:23:01,005 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:23:01,184 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:23:01,184 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:23:03,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:23:03,398 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:23:03,400 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:23:05,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:23:05,585 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:23:05,585 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 53.0640001297. 2022-02-08 07:23:05,585 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:23:05,588 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:23:06,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:23:07,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:23:07,655 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:23:07,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:23:07,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:23:07,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:23:07,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:23:08,015 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:23:08,038 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:23:08,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:23:08,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:23:08,720 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:23:08,721 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 07:23:10,721 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:23:11,108 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:23:11,108 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 07:23:13,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:23:13,362 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:23:13,362 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 07:23:15,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:23:16,457 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:23:16,457 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 10.871999979. 2022-02-08 07:23:16,457 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:23:16,460 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:23:17,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:23:18,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:23:18,315 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:23:18,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:23:18,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:23:18,387 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:23:18,440 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:23:18,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:23:18,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:23:18,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:23:18,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:23:19,605 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:23:19,605 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.14700007439. 2022-02-08 07:23:19,605 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:23:19,607 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:23:20,298 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:23:20,678 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:23:20,790 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:23:20,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:23:20,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:23:20,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:23:20,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:23:20,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:23:21,005 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:23:21,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:23:21,056 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:23:21,976 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:23:21,976 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.37199997902. 2022-02-08 07:23:21,976 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:23:21,979 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:23:22,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:23:22,928 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:23:23,029 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:23:23,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:23:23,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:23:23,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:23:23,076 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:23:23,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:23:23,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:23:23,145 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:23:23,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:23:23,716 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:23:23,717 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.74000000954. 2022-02-08 07:23:23,717 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:23:23,719 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:23:24,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:23:24,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:23:25,072 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:23:25,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:23:25,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:23:25,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:23:25,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:23:25,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:23:25,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:23:25,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:23:25,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:23:25,690 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:23:25,690 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.97399997711. 2022-02-08 07:23:25,691 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:23:25,694 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:23:26,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:23:26,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:23:26,536 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:23:26,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:23:26,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:23:26,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:23:26,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:23:26,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:23:26,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:23:26,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:23:26,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:23:27,187 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:23:27,188 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.49699997902. 2022-02-08 07:23:27,188 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:23:27,190 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:23:27,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:23:28,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:23:28,118 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:23:28,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:23:28,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:23:28,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:23:28,165 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:23:28,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:23:28,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:23:28,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:23:28,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:23:29,006 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:23:29,006 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.8180000782. 2022-02-08 07:23:29,006 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:23:29,009 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:23:29,729 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:23:30,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:23:30,203 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:23:30,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:23:30,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:23:30,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:23:30,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:23:30,397 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:23:30,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:23:30,464 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:23:30,482 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:23:31,072 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:23:31,072 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.06599998474. ############################################################ # # RAM FootPrint: 4392.3828125 # ############################################################ 2022-02-08 07:23:31,105 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 07:23:31,105 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 108 seconds. Reason: Iteration:19 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 9625 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 9652 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:25:19,111 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 07:25:19,112 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 07:25:24,117 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:25:24,237 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:24,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:25:24,546 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:24,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:25:24,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:25:24,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:25:24,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:25:24,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:25:24,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:25:24,640 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:25:24,654 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:25:24,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 07:25:24,717 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['22873 memcached\n'], []) 2022-02-08 07:25:24,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 07:25:24,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 07:25:24,792 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['22873 memcached\n'], []) 2022-02-08 07:25:24,792 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:25:24,795 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:25:25,459 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:25,749 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:25:25,859 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:25,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:25:25,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:25:25,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:25:25,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:25:25,933 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:25:25,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:25:25,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:25:25,993 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:25:25,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 07:25:26,075 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['73049 memcached\n'], []) 2022-02-08 07:25:26,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 07:25:26,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 07:25:26,128 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['73049 memcached\n'], []) 2022-02-08 07:25:26,128 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:25:26,131 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:25:26,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:26,420 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:25:26,480 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:26,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:25:26,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:25:26,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:25:26,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:25:26,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:25:26,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:25:26,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:25:26,584 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:25:26,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 07:25:26,637 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['56822 memcached\n'], []) 2022-02-08 07:25:26,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 07:25:26,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 07:25:26,687 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['56822 memcached\n'], []) 2022-02-08 07:25:26,687 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:25:26,688 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:25:26,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:26,979 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:25:27,039 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:27,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:25:27,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:25:27,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:25:27,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:25:27,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:25:27,109 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:25:27,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:25:27,137 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:25:27,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 07:25:27,213 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['87424 memcached\n'], []) 2022-02-08 07:25:27,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 07:25:27,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 07:25:27,267 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['87424 memcached\n'], []) 2022-02-08 07:25:27,269 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:25:27,273 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:25:27,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:27,579 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:25:27,644 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:27,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:25:27,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:25:27,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:25:27,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:25:27,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:25:27,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:25:27,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:25:27,753 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:25:27,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 07:25:27,808 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['81269 memcached\n'], []) 2022-02-08 07:25:27,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 07:25:27,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 07:25:27,859 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['81269 memcached\n'], []) 2022-02-08 07:25:27,859 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:25:27,862 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:25:27,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:28,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:25:28,223 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:28,224 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:25:28,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:25:28,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:25:28,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:25:28,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:25:28,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:25:28,308 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:25:28,322 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:25:28,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 07:25:28,414 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['116252 memcached\n'], []) 2022-02-08 07:25:28,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 07:25:28,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 07:25:28,457 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['116252 memcached\n'], []) 2022-02-08 07:25:28,459 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:25:28,461 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:25:28,538 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:28,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:25:28,815 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:28,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:25:28,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:25:28,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:25:28,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:25:28,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:25:28,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:25:28,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:25:28,914 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:25:28,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 07:25:28,973 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['50137 memcached\n'], []) 2022-02-08 07:25:28,974 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 07:25:29,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 07:25:29,023 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['50137 memcached\n'], []) 2022-02-08 07:25:29,023 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:25:29,026 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:25:29,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:29,326 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:25:29,407 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:29,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:25:29,423 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:25:29,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:25:29,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:25:29,463 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:25:29,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:25:29,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:25:29,506 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:25:29,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 07:25:29,569 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['65053 memcached\n'], []) 2022-02-08 07:25:29,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 07:25:29,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 07:25:29,614 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['65053 memcached\n'], []) 2022-02-08 07:25:29,614 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:25:29,615 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:25:29,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:29,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:25:29,967 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:29,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:25:29,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:25:29,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:25:30,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:25:30,025 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:25:30,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:25:30,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:25:30,065 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:25:30,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 07:25:30,111 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['114424 memcached\n'], []) 2022-02-08 07:25:30,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 07:25:30,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 07:25:30,155 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['114424 memcached\n'], []) 2022-02-08 07:25:30,155 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:25:30,157 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:25:30,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:30,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:25:30,566 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:30,568 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:25:30,584 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:25:30,596 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:25:30,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:25:30,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:25:30,640 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:25:30,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:25:30,667 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:25:30,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 07:25:30,716 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['313 memcached\n'], []) 2022-02-08 07:25:30,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 07:25:30,749 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 07:25:30,776 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['313 memcached\n'], []) 2022-02-08 07:25:30,776 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:25:30,779 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:25:30,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:31,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:25:31,164 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:31,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:25:31,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:25:31,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:25:31,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:25:31,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:25:31,249 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:25:31,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:25:31,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:25:31,296 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 07:25:31,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:25:31,336 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:25:31,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:25:32,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:25:32,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:25:33,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:25:33,289 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 07:25:33,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:25:33,303 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:25:33,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:25:33,382 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 07:25:33,382 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:25:33,384 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:25:33,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:33,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:25:33,733 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:33,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:25:33,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:25:33,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:25:33,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:25:33,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:25:33,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:25:33,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:25:33,834 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:25:33,849 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 07:25:33,877 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:25:33,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:25:36,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:25:37,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:25:37,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:25:37,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:25:37,990 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 07:25:37,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:25:38,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:25:38,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:25:38,085 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 07:25:38,085 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:25:38,088 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:25:38,165 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:38,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:25:38,421 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:38,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:25:38,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:25:38,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:25:38,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:25:38,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:25:38,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:25:38,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:25:38,515 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:25:38,529 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 07:25:38,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:25:38,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:25:40,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:25:40,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:25:41,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:25:41,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:25:41,500 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 07:25:41,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:25:41,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:25:41,581 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:25:41,595 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 07:25:41,596 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:25:41,598 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 07:25:41,678 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:41,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 07:25:41,983 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:41,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 07:25:41,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 07:25:42,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 07:25:42,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 07:25:42,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 07:25:42,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 07:25:42,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 07:25:42,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:25:42,109 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 07:25:42,135 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:25:42,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:25:42,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:25:42,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:25:42,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:25:42,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:25:42,233 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 07:25:42,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:25:42,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:25:42,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:25:42,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:42,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:25:42,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:42,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:42,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:42,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:42,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:42,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:42,423 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:42,438 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:42,454 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:42,473 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:25:42,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:42,512 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:42,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:42,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:42,563 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:42,581 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:42,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:42,615 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:42,632 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:42,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:25:42,665 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:42,683 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:42,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:42,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:42,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:42,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:42,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:42,780 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:42,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:42,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:25:42,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:42,844 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:42,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:42,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:42,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:42,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:42,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:42,931 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:42,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:42,964 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:25:42,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:42,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:43,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:43,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:43,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:43,056 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:43,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:43,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:43,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:43,117 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:25:43,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:43,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:43,163 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:43,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:43,191 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:43,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:43,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:43,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:43,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:25:43,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 07:25:43,306 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:25:43,329 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:25:43,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:25:43,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:25:43,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:25:43,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:25:43,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:25:43,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:25:43,444 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 07:25:43,447 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:25:43,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:43,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:25:43,947 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:43,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:25:43,967 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:25:44,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:25:44,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:25:44,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:25:44,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:25:44,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:25:44,249 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:25:44,336 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 07:25:44,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:25:44,418 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:25:46,272 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:25:46,877 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:25:47,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:25:47,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:25:47,563 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 07:25:47,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:25:47,630 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:25:47,701 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:25:47,792 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 07:25:47,793 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:25:47,796 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 07:25:47,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:48,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 07:25:48,187 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:48,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 07:25:48,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 07:25:48,220 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 07:25:48,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 07:25:48,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 07:25:48,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 07:25:48,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 07:25:48,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:25:48,303 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 07:25:48,336 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:25:48,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:25:48,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:25:48,385 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:25:48,408 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:25:48,423 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:25:48,437 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 07:25:48,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:25:48,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:25:48,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:25:48,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:48,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:25:48,515 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:48,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:48,543 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:48,558 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:48,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:48,594 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:48,609 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:48,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:25:48,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:48,655 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:25:48,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:48,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:48,706 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:48,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:48,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:48,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:48,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:48,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:25:48,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:48,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:25:48,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:48,862 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:48,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:48,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:48,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:48,931 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:48,948 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:48,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:25:48,982 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:48,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:25:49,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:49,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:49,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:49,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:49,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:49,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:49,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:49,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:25:49,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:49,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:25:49,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:49,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:49,201 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:49,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:49,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:49,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:49,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:49,285 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:25:49,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:49,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:25:49,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:49,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:49,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:49,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:49,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:49,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:49,426 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:49,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:25:49,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:25:49,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 07:25:49,497 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:25:49,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 5 queries submitted, 0 failed, 9673 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:25:49,529 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:25:49,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:25:49,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:25:49,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:25:49,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:25:49,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:25:49,628 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 07:25:49,630 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:25:49,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:49,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:25:49,996 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:49,996 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:25:50,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:25:50,023 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:25:50,036 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:25:50,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:25:50,066 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:25:50,079 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:25:50,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:25:50,111 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 07:25:50,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:25:50,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:25:52,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:25:53,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:25:53,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:25:54,163 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:25:54,234 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 07:25:54,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:25:54,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:25:54,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:25:54,408 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 07:25:54,410 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:25:54,411 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:25:54,503 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:25:54,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:25:54,859 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:25:54,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:25:54,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:25:54,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:25:54,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:25:54,957 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:25:54,982 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:25:55,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:25:55,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:25:55,084 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 07:25:55,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:25:55,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:25:58,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:25:59,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:26:00,028 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:26:00,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:26:00,400 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 07:26:00,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:26:00,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:26:00,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:26:00,789 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 07:26:00,789 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:26:00,790 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 07:26:00,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:26:01,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 07:26:01,269 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:26:01,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 07:26:01,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 07:26:01,306 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 07:26:01,326 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 07:26:01,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 07:26:01,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 07:26:01,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 07:26:01,413 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:26:01,434 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 07:26:01,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:26:01,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:26:01,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:26:01,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:26:01,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:26:01,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:26:01,611 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 07:26:01,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:26:01,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:26:01,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:26:01,678 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:01,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:26:01,721 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:01,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:01,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:01,790 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:01,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:01,834 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:01,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:01,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:01,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:01,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:26:01,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:01,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:01,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:02,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:02,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:02,071 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:02,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:02,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:02,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:02,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:26:02,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:02,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:02,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:02,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:02,272 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:02,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:02,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:02,344 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:02,367 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:02,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:26:02,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:02,438 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:02,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:02,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:02,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:02,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:02,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:02,581 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:02,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:02,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:26:02,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:02,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:02,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:02,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:02,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:02,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:02,788 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:02,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:02,828 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:02,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:26:02,877 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:02,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:02,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:02,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:02,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:03,005 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:03,028 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:03,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:03,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:03,091 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 07:26:03,112 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:03,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:03,161 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:03,184 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:03,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:03,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:03,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:03,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:03,306 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 07:26:03,309 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 07:26:03,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:26:03,627 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 07:26:03,694 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:26:03,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 07:26:03,711 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 07:26:03,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 07:26:03,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 07:26:03,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 07:26:03,770 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 07:26:03,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 07:26:03,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:26:03,816 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 07:26:03,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:26:03,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:26:03,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:26:03,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:26:03,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:26:03,918 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:26:03,931 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 07:26:03,931 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:26:03,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:26:03,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:26:03,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:03,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:26:04,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:04,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:04,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:04,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:04,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:04,096 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:04,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:04,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:04,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:04,173 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:26:04,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:04,211 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:04,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:04,247 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:04,266 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:04,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:04,303 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:04,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:04,339 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:04,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:26:04,375 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:04,392 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:04,413 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:04,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:04,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:04,469 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:04,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:04,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:04,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:04,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:26:04,561 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:04,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:04,596 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:04,614 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:04,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:04,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:04,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:04,694 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:04,714 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:04,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:26:04,752 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:04,767 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:04,785 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:04,801 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:04,816 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:04,834 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:04,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:04,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:04,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:04,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:26:04,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:04,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:04,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:04,969 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:04,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:05,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:05,017 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:05,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:05,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:05,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 07:26:05,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:05,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:05,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:05,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:05,160 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:05,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:05,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:05,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:05,236 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 07:26:05,239 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:26:05,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:26:05,589 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:26:05,799 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:26:05,801 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:26:05,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:26:05,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:26:05,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:26:05,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:26:05,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:26:05,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:26:06,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:26:06,046 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 07:26:06,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:26:06,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:26:09,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:26:09,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:26:10,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:26:10,357 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:26:10,380 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 07:26:10,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:26:10,395 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:26:10,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:26:10,480 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 07:26:10,480 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:26:10,482 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:26:10,632 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:26:10,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:26:11,007 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:26:11,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:26:11,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:26:11,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:26:11,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:26:11,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:26:11,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:26:11,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:26:11,140 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:26:11,160 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 07:26:11,191 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:26:11,211 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:26:13,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:26:14,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:26:14,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:26:14,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:26:14,625 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 07:26:14,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:26:14,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:26:14,678 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:26:14,697 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 07:26:14,697 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:26:14,700 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:26:15,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:26:15,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:26:15,755 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:26:15,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:26:15,773 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:26:15,789 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:26:15,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:26:15,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:26:15,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:26:15,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:26:15,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:26:15,920 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 07:26:15,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:26:15,969 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:26:18,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:26:19,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:26:19,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:26:19,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:26:19,632 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 07:26:19,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:26:19,647 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:26:19,696 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:26:19,726 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 07:26:19,726 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:26:19,727 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:26:19,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:26:20,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:26:20,121 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:26:20,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:26:20,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:26:20,150 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:26:20,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:26:20,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:26:20,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:26:20,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:26:20,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:26:20,243 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 07:26:20,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:26:20,290 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:26:21,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:26:22,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:26:22,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:26:22,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:26:22,769 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 07:26:22,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:26:22,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:26:22,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:26:22,848 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 07:26:22,849 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:26:22,852 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 07:26:22,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:26:23,155 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 07:26:23,230 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:26:23,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 07:26:23,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 07:26:23,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 07:26:23,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 07:26:23,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 07:26:23,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 07:26:23,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 07:26:23,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:26:23,349 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 07:26:23,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:26:23,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:26:23,418 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:26:23,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:26:23,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:26:23,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:26:23,476 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 07:26:23,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:26:23,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:26:23,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:26:23,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:23,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:26:23,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:23,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:23,585 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:23,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:23,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:23,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:23,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:23,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:26:23,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:23,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:26:23,724 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:23,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:23,762 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:23,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:23,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:23,816 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:23,839 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:23,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:26:23,877 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:23,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:26:23,913 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:23,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:23,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:23,964 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:23,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:24,003 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:24,020 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:24,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:26:24,055 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:24,073 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:26:24,092 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:24,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:24,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:24,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:24,160 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:24,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:24,191 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:24,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:26:24,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:24,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:26:24,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:24,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:24,290 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:24,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:24,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:24,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:24,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:24,381 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:26:24,398 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:24,415 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:26:24,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:24,448 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:24,464 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:24,482 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:24,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:24,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:24,530 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:24,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:26:24,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:24,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 07:26:24,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:24,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:24,634 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:24,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:24,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:24,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:24,709 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:24,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:26:24,743 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 07:26:24,744 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:26:24,746 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:26:24,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:26:25,072 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:26:25,147 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:26:25,147 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:26:25,170 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:26:25,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:26:25,204 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:26:25,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:26:25,247 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:26:25,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:26:25,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:26:25,769 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:26:25,769 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.02600002289. 2022-02-08 07:26:25,769 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:26:25,772 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:26:25,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:26:26,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:26:26,214 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:26:26,216 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:26:26,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:26:26,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:26:26,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:26:26,428 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:26:26,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:26:26,474 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:26:26,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:26:26,678 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:26:26,680 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:26:28,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:26:28,851 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:26:28,852 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:26:30,852 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:26:31,312 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:26:31,312 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:26:33,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:26:33,476 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:26:33,476 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:26:35,476 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:26:35,665 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:26:35,665 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:26:37,667 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:26:37,861 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:26:37,862 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:26:39,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:26:40,073 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:26:40,073 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:26:42,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:26:42,253 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:26:42,253 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:26:44,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:26:44,420 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:26:44,421 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:26:46,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:26:46,624 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:26:46,624 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:26:48,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:26:48,812 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:26:48,812 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 9677 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:26:50,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:26:51,101 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:26:51,101 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:26:53,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:26:53,323 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:26:53,323 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:26:55,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:26:55,575 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:26:55,575 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:26:57,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:26:57,862 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:26:57,864 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:26:59,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:27:00,114 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:27:00,114 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:27:02,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:27:02,355 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:27:02,355 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:27:04,358 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:27:04,561 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:27:04,561 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:27:06,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:27:06,813 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:27:06,813 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:27:08,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:27:09,062 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:27:09,062 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:27:11,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:27:11,305 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:27:11,306 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 45.5369999409. 2022-02-08 07:27:11,306 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:27:11,309 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:27:12,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:27:12,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:27:13,098 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:27:13,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:27:13,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:27:13,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:27:13,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:27:13,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:27:13,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:27:13,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:27:13,295 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:27:13,913 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:27:13,913 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.60600018501. 2022-02-08 07:27:13,914 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:27:13,917 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:27:14,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:27:15,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:27:15,329 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:27:15,331 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:27:15,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:27:15,374 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:27:15,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:27:15,428 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:27:15,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:27:15,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:27:15,523 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:27:16,197 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:27:16,197 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 07:27:18,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:27:18,466 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:27:18,466 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.55200004578. 2022-02-08 07:27:18,466 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:27:18,467 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:27:19,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:27:19,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:27:19,894 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:27:19,894 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:27:19,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:27:19,940 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:27:19,964 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:27:19,990 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:27:20,015 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:27:20,109 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:27:20,155 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:27:20,891 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:27:20,891 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.42499995232. 2022-02-08 07:27:20,891 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:27:20,892 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:27:21,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:27:22,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:27:22,388 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:27:22,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:27:22,413 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:27:22,448 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:27:22,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:27:22,490 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:27:22,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:27:22,546 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:27:22,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:27:23,492 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:27:23,493 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.60199999809. 2022-02-08 07:27:23,493 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:27:23,494 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:27:24,125 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:27:24,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:27:24,477 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:27:24,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:27:24,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:27:24,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:27:24,528 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:27:24,545 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:27:24,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:27:24,584 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:27:24,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:27:25,180 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:27:25,180 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.68700003624. 2022-02-08 07:27:25,180 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:27:25,181 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:27:26,058 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:27:26,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:27:27,193 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:27:27,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:27:27,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:27:27,242 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:27:27,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:27:27,303 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:27:27,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:27:27,362 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:27:27,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:27:33,476 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:27:33,477 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 8.29799985886. 2022-02-08 07:27:33,477 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:27:33,480 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:27:34,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:27:34,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:27:34,592 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:27:34,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:27:34,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:27:34,640 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:27:34,660 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:27:34,684 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:27:34,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:27:34,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:27:34,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:27:36,263 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:27:36,265 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.78600001335. 2022-02-08 07:27:36,265 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:27:36,266 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:27:37,084 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:27:37,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:27:37,517 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:27:37,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:27:37,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:27:37,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:27:37,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:27:37,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:27:37,618 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:27:37,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:27:37,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:27:38,323 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:27:38,325 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.05999994278. ############################################################ # # RAM FootPrint: 4384.44921875 # ############################################################ 2022-02-08 07:27:38,357 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 07:27:38,357 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 73 seconds. Reason: Iteration:20 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 9688 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 9711 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:28:51,362 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 07:28:51,364 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 07:28:56,365 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:28:56,484 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:28:56,765 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:28:56,846 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:28:56,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:28:56,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:28:56,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:28:56,926 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:28:56,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:28:56,979 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:28:57,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:28:57,032 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:28:57,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 07:28:57,108 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['23615 memcached\n'], []) 2022-02-08 07:28:57,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 07:28:57,141 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 07:28:57,165 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['23615 memcached\n'], []) 2022-02-08 07:28:57,165 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:28:57,167 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:28:57,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:28:57,808 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:28:57,905 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:28:57,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:28:57,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:28:57,961 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:28:57,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:28:57,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:28:58,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:28:58,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:28:58,052 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:28:58,052 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 07:28:58,127 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['74108 memcached\n'], []) 2022-02-08 07:28:58,127 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 07:28:58,160 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 07:28:58,183 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['74108 memcached\n'], []) 2022-02-08 07:28:58,183 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:28:58,187 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:28:58,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:28:58,493 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:28:58,555 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:28:58,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:28:58,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:28:58,584 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:28:58,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:28:58,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:28:58,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:28:58,647 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:28:58,664 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:28:58,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 07:28:58,732 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['57695 memcached\n'], []) 2022-02-08 07:28:58,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 07:28:58,757 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 07:28:58,782 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['57695 memcached\n'], []) 2022-02-08 07:28:58,782 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:28:58,785 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:28:58,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:28:59,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:28:59,151 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:28:59,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:28:59,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:28:59,178 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:28:59,193 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:28:59,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:28:59,223 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:28:59,236 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:28:59,250 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:28:59,250 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 07:28:59,290 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['88163 memcached\n'], []) 2022-02-08 07:28:59,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 07:28:59,316 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 07:28:59,338 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['88163 memcached\n'], []) 2022-02-08 07:28:59,338 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:28:59,339 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:28:59,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:28:59,648 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:28:59,720 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:28:59,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:28:59,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:28:59,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:28:59,759 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:28:59,775 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:28:59,788 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:28:59,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:28:59,812 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:28:59,812 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 07:28:59,858 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['82008 memcached\n'], []) 2022-02-08 07:28:59,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 07:28:59,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 07:28:59,907 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['82008 memcached\n'], []) 2022-02-08 07:28:59,907 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:28:59,910 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:28:59,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:00,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:29:00,276 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:00,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:29:00,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:29:00,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:29:00,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:29:00,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:29:00,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:29:00,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:29:00,377 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:29:00,377 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 07:29:00,453 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['116993 memcached\n'], []) 2022-02-08 07:29:00,453 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 07:29:00,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 07:29:00,500 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['116993 memcached\n'], []) 2022-02-08 07:29:00,500 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:29:00,502 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:29:00,601 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:00,815 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:29:00,872 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:00,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:29:00,885 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:29:00,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:29:00,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:29:00,927 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:29:00,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:29:00,951 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:29:00,966 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:29:00,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 07:29:01,013 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['50881 memcached\n'], []) 2022-02-08 07:29:01,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 07:29:01,039 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 07:29:01,062 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['50881 memcached\n'], []) 2022-02-08 07:29:01,062 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:29:01,065 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:29:01,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:01,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:29:01,430 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:01,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:29:01,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:29:01,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:29:01,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:29:01,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:29:01,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:29:01,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:29:01,526 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:29:01,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 07:29:01,588 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['65792 memcached\n'], []) 2022-02-08 07:29:01,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 07:29:01,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 07:29:01,634 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['65792 memcached\n'], []) 2022-02-08 07:29:01,634 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:29:01,635 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:29:01,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:01,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:29:01,971 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:01,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:29:01,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:29:01,999 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:29:02,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:29:02,035 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:29:02,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:29:02,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:29:02,072 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.121.128 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:29:02,073 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 07:29:02,115 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.121.128 - Process info before sending signal: (['115166 memcached\n'], []) 2022-02-08 07:29:02,115 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: kill -9 $(pgrep memcached) 2022-02-08 07:29:02,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: pgrep -l memcached 2022-02-08 07:29:02,158 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.121.128 - Process info after sending signal: (['115166 memcached\n'], []) 2022-02-08 07:29:02,158 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:29:02,161 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:29:02,242 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:02,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:29:02,582 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:02,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:29:02,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:29:02,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:29:02,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:29:02,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:29:02,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:29:02,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:29:02,686 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 07:29:02,686 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 07:29:02,732 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['1123 memcached\n'], []) 2022-02-08 07:29:02,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 07:29:02,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 07:29:02,788 | infra | DEBUG | Thread-10676 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['1123 memcached\n'], []) 2022-02-08 07:29:02,789 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:29:02,790 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:29:02,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:03,104 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:29:03,171 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:03,171 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:29:03,187 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:29:03,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:29:03,214 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:29:03,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:29:03,247 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:29:03,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:29:03,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:29:03,295 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 07:29:03,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:29:03,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:03,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:29:04,586 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:04,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:05,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:29:05,309 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 07:29:05,311 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:29:05,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:05,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:29:05,398 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 07:29:05,400 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:29:05,401 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:29:05,497 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:05,729 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:29:05,790 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:05,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:29:05,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:29:05,816 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:29:05,831 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:29:05,846 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:29:05,861 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:29:05,874 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:29:05,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:29:05,907 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 07:29:05,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:29:05,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:06,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:29:07,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:07,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:07,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:29:07,882 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 07:29:07,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:29:07,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:07,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:29:07,986 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 07:29:07,986 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:29:07,990 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:29:08,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:08,286 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:29:08,351 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:08,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:29:08,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:29:08,382 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:29:08,397 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:29:08,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:29:08,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:29:08,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:29:08,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:29:08,473 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 07:29:08,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:29:08,516 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:10,275 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:29:10,752 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:11,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:11,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:29:11,319 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 07:29:11,319 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:29:11,331 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:11,400 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:29:11,413 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 07:29:11,414 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:29:11,415 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 07:29:11,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:11,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 07:29:11,808 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:11,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 07:29:11,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 07:29:11,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 07:29:11,867 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 07:29:11,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 07:29:11,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 07:29:11,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 07:29:11,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:29:11,953 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 07:29:11,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:29:11,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:12,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:29:12,022 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:12,035 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:12,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:29:12,062 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 07:29:12,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:29:12,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:12,091 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:29:12,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:12,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:29:12,137 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:12,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:12,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:12,183 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:12,198 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:12,214 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:12,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:12,244 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:12,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:12,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:29:12,296 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:12,313 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:12,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:12,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:12,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:12,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:12,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:12,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:12,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:12,451 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:29:12,470 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:12,489 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:12,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:12,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:12,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:12,559 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:12,578 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:12,595 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:12,611 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:12,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:29:12,647 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:12,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:12,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:12,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:12,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:12,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:12,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:12,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:12,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:12,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:29:12,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:12,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:12,841 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:12,857 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:12,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:12,888 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:12,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:12,920 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:12,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:12,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:29:12,969 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:12,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:13,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:13,015 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:13,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:13,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:13,062 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:13,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:13,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:13,111 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 07:29:13,131 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:13,148 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:13,164 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:13,181 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:13,198 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:13,216 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:13,233 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:13,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:13,276 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 07:29:13,279 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:29:13,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:14,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:29:14,651 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:14,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:29:14,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:29:15,013 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:29:15,177 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:29:15,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:29:15,492 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:29:15,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:29:15,762 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:29:15,917 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 07:29:15,950 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:29:16,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:16,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:29:17,272 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:17,608 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:17,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:29:18,086 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 07:29:18,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:29:18,207 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:18,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:29:18,454 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 07:29:18,456 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:29:18,457 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 07:29:18,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:18,792 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 07:29:18,855 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:18,857 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 07:29:18,871 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 07:29:18,884 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 07:29:18,900 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 07:29:18,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 07:29:18,934 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 07:29:18,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 07:29:18,963 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:29:18,980 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 07:29:19,006 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:29:19,033 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:19,053 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:29:19,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:19,084 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:19,098 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:29:19,111 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 07:29:19,112 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:29:19,128 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:19,144 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:29:19,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:19,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:29:19,193 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:19,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:19,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:19,243 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:19,259 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:19,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:19,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:19,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:19,325 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:19,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:29:19,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:19,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:19,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:19,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:19,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:19,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:19,464 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:19,483 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:19,502 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:19,520 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:29:19,539 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:19,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:19,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:19,588 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:19,605 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:19,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:19,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:19,655 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:19,673 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:19,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:29:19,707 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:19,723 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:19,740 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:19,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:19,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:19,786 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:19,802 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:19,816 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:19,832 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:19,848 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:29:19,864 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:19,880 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:19,895 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:19,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:19,928 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:19,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:19,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:19,977 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:19,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:20,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:29:20,028 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:20,045 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:20,066 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:20,088 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:20,109 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:20,132 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:20,150 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:20,165 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:20,183 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:20,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 07:29:20,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:20,234 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:20,252 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:20,269 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:20,288 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:20,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:20,326 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:20,345 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:20,361 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 07:29:20,364 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:29:20,443 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:20,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:29:20,717 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:20,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:29:20,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:29:20,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:29:20,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:29:20,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:29:20,793 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:29:20,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:29:20,822 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:29:20,838 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 07:29:20,862 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:29:20,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:23,641 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:29:24,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:24,658 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:24,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:29:25,000 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 07:29:25,000 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:29:25,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:25,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:29:25,098 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 07:29:25,098 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:29:25,107 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:29:25,193 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:25,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:29:25,480 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:25,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:29:25,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:29:25,510 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:29:25,525 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:29:25,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:29:25,555 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:29:25,569 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:29:25,584 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:29:25,601 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 07:29:25,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:29:25,653 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:28,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:29:28,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:29,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:29,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:29:29,430 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 07:29:29,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:29:29,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:29,509 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:29:29,525 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 07:29:29,526 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:29:29,528 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 07:29:29,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:29,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 07:29:29,891 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:29,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 07:29:29,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 07:29:29,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 07:29:29,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 07:29:29,947 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 07:29:29,960 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 07:29:29,973 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 07:29:29,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:29:30,002 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 07:29:30,032 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:29:30,049 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:30,063 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:29:30,081 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:30,094 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:30,108 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:29:30,122 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 07:29:30,122 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:29:30,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:30,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:29:30,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:30,183 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:29:30,198 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:30,213 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:30,226 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:30,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:30,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:30,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:30,289 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:30,305 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:30,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:30,351 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:29:30,377 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:30,394 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:30,410 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:30,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:30,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:30,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:30,480 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:30,499 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:30,515 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:30,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:29:30,551 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:30,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:30,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:30,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:30,614 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:30,631 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:30,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:30,668 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:30,683 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:30,698 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:29:30,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:30,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:30,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:30,762 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:30,779 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:30,795 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:30,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:30,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:30,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:30,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:29:30,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:30,892 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:30,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:30,923 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:30,938 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:30,953 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:30,971 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:30,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:31,012 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:31,028 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:29:31,043 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:31,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:31,075 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:31,091 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:31,107 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:31,124 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:31,138 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:31,153 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:31,167 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:31,183 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 07:29:31,198 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:31,214 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:31,230 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:31,244 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:31,260 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:31,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:31,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:31,309 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:31,325 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 07:29:31,326 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 07:29:31,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:31,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 07:29:31,678 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:31,680 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 07:29:31,691 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 07:29:31,703 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 07:29:31,716 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 07:29:31,730 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 07:29:31,743 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 07:29:31,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 07:29:31,769 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:29:31,782 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 07:29:31,809 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:29:31,823 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:31,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:29:31,851 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:31,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:31,878 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:29:31,891 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 07:29:31,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:29:31,904 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:31,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:29:31,930 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:31,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:29:31,961 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:31,976 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:31,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:32,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:32,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:32,069 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:32,089 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:32,105 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:32,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:32,140 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:29:32,158 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:32,176 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:32,194 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:32,210 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:32,229 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:32,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:32,265 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:32,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:32,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:32,318 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:29:32,335 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:32,354 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:32,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:32,390 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:32,407 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:32,426 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:32,444 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:32,461 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:32,477 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:32,494 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:29:32,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:32,533 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:32,549 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:32,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:32,581 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:32,596 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:32,612 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:32,630 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:32,644 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:32,660 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:29:32,676 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:32,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:32,706 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:32,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:32,733 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:32,750 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:32,766 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:32,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:32,796 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:32,811 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:29:32,826 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:32,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:32,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:32,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:32,887 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:32,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:32,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:32,931 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:32,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:32,961 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 07:29:32,979 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:32,994 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:33,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:33,026 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:33,043 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:33,061 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:33,078 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:33,095 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:33,118 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 07:29:33,121 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:29:33,206 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:33,447 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:29:33,512 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:33,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:29:33,528 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:29:33,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:29:33,556 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:29:33,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:29:33,591 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:29:33,607 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:29:33,621 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:29:33,638 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 07:29:33,661 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:29:33,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:36,114 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:29:36,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:37,030 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:37,334 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:29:37,348 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 07:29:37,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:29:37,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:37,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:29:37,430 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 07:29:37,431 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:29:37,434 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:29:37,532 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:37,798 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:29:37,872 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:37,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:29:37,890 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:29:37,905 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:29:37,924 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:29:37,946 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:29:37,961 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:29:37,980 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:29:37,997 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:29:38,016 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 07:29:38,042 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:29:38,059 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:38,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:29:39,109 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:39,414 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:39,713 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:29:39,732 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 07:29:39,732 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:29:39,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:39,795 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:29:39,813 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 07:29:39,813 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:29:39,816 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:29:40,753 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:41,273 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:29:41,421 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:41,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:29:41,437 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:29:41,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:29:41,486 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:29:41,515 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:29:41,543 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:29:41,575 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:29:41,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:29:41,609 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 07:29:41,635 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:29:41,651 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:44,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:29:44,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:45,154 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:45,434 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:29:45,457 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 07:29:45,457 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:29:45,471 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:45,522 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:29:45,538 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 07:29:45,538 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:29:45,540 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:29:45,664 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:45,901 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:29:45,984 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:45,984 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:29:46,010 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:29:46,028 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:29:46,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:29:46,082 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:29:46,102 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:29:46,121 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:29:46,147 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:29:46,174 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 07:29:46,203 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:29:46,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 9725 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:29:51,369 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:29:52,040 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:52,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:52,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:29:52,841 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 07:29:52,842 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:29:52,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:52,895 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:29:52,913 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 07:29:52,914 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:29:52,915 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 07:29:53,019 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:53,256 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 07:29:53,322 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:53,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 07:29:53,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 07:29:53,355 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 07:29:53,371 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 07:29:53,388 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 07:29:53,403 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 07:29:53,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 07:29:53,431 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 07:29:53,448 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 07:29:53,496 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 07:29:53,513 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:53,530 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 07:29:53,548 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:53,562 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 07:29:53,576 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 07:29:53,592 | test | DEBUG | Thread-10676 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 07:29:53,592 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 07:29:53,607 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 07:29:53,622 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 07:29:53,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:53,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 07:29:53,671 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:53,687 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:53,704 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:53,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:53,736 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:53,755 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:53,772 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:53,788 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 07:29:53,803 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:53,821 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 07:29:53,838 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:53,858 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:53,875 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:53,897 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:53,917 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:53,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:53,959 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:53,992 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 07:29:54,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:54,029 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 07:29:54,048 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:54,065 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:54,081 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:54,099 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:54,118 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:54,134 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:54,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:54,168 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 07:29:54,186 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:54,200 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 07:29:54,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:54,232 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:54,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:54,263 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:54,276 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:54,292 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:54,306 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:54,322 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 07:29:54,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:54,352 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 07:29:54,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:54,384 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:54,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:54,417 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:54,433 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:54,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:54,467 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:54,484 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 07:29:54,500 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:54,517 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 07:29:54,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:54,552 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:54,617 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:54,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:54,655 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:54,674 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:54,693 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:54,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 07:29:54,727 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:54,746 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 07:29:54,763 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:54,782 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:54,799 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:54,819 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:54,836 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:54,855 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:54,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:54,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 07:29:54,910 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 07:29:54,910 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:29:54,913 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 07:29:54,986 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:55,224 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 07:29:55,299 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:55,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 07:29:55,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 07:29:55,328 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 07:29:55,342 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 07:29:55,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 07:29:55,378 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 07:29:55,392 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 07:29:55,411 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:29:55,904 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 07:29:55,905 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.99599981308. 2022-02-08 07:29:55,905 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:29:55,907 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 07:29:56,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:29:56,240 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 07:29:56,302 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:29:56,302 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 07:29:56,316 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 07:29:56,332 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 07:29:56,348 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 07:29:56,365 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 07:29:56,380 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 07:29:56,401 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 07:29:56,421 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:29:56,572 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:29:56,572 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:29:58,572 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:29:58,726 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:29:58,726 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:00,726 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:00,881 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:00,881 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:02,881 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:03,051 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:03,051 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:05,051 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:05,207 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:05,207 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:07,209 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:07,362 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:07,364 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:09,364 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:09,526 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:09,526 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:11,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:11,688 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:11,688 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:13,690 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:13,844 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:13,844 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:15,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:16,006 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:16,007 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:18,007 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:18,180 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:18,180 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:20,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:20,338 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:20,338 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:22,338 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:22,581 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:22,581 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:24,582 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:24,755 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:24,756 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:26,756 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:26,941 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:26,941 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:28,943 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:29,099 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:29,099 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:31,101 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:31,311 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:31,312 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:33,312 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:33,477 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:33,477 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:35,479 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:35,653 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:35,654 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:37,654 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:37,834 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:37,835 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:39,835 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:40,009 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:40,009 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:42,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:42,180 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:42,180 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:44,180 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:44,359 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:44,361 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:46,361 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:46,528 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:46,528 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:48,528 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:48,716 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:48,716 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 9725 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:30:50,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:50,920 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:50,920 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:52,921 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:53,085 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:53,086 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:55,086 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:55,260 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:55,260 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:57,262 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:57,517 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:57,519 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:30:59,519 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:30:59,698 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:30:59,698 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:31:01,700 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:31:01,905 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:31:01,907 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 07:31:03,907 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:31:04,269 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 07:31:04,269 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 68.364000082. 2022-02-08 07:31:04,269 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:31:04,272 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 07:31:05,211 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:31:05,915 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 07:31:06,529 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:31:06,530 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 07:31:06,720 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 07:31:06,943 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 07:31:07,085 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 07:31:07,197 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 07:31:07,217 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 07:31:07,253 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 07:31:07,282 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:31:08,151 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 07:31:08,151 | test | INFO | Thread-10676 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 07:31:10,151 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:31:10,335 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 07:31:10,335 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.06599998474. 2022-02-08 07:31:10,336 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:31:10,338 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 07:31:10,868 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:31:11,157 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 07:31:11,278 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:31:11,279 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 07:31:11,299 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 07:31:11,323 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 07:31:11,346 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 07:31:11,372 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 07:31:11,392 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 07:31:11,427 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 07:31:11,450 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:31:11,749 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 07:31:11,750 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.41300010681. 2022-02-08 07:31:11,750 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:31:11,752 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 07:31:12,404 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:31:12,729 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 07:31:12,865 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:31:12,865 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 07:31:12,882 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 07:31:12,914 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 07:31:12,937 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 07:31:12,966 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 07:31:12,983 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 07:31:13,016 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 07:31:13,043 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:31:14,479 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 07:31:14,479 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.72900009155. 2022-02-08 07:31:14,480 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:31:14,482 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 07:31:15,278 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:31:15,762 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 07:31:15,890 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:31:15,891 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 07:31:15,911 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 07:31:15,944 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 07:31:15,970 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 07:31:15,993 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 07:31:16,009 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 07:31:16,046 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 07:31:16,068 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:31:17,825 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 07:31:17,825 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.34500002861. 2022-02-08 07:31:17,825 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:31:17,828 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 07:31:18,246 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:31:18,506 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 07:31:18,588 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:31:18,589 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 07:31:18,604 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 07:31:18,625 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 07:31:18,638 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 07:31:18,657 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 07:31:18,670 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 07:31:18,697 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 07:31:18,710 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:31:19,519 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 07:31:19,519 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.69499993324. 2022-02-08 07:31:19,519 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:31:19,523 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 07:31:20,542 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:31:21,058 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 07:31:21,299 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:31:21,301 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 07:31:21,321 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 07:31:21,349 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 07:31:21,368 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 07:31:21,391 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 07:31:21,460 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 07:31:21,507 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 07:31:21,526 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:31:22,661 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 07:31:22,661 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.14100003242. 2022-02-08 07:31:22,661 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:31:22,664 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 07:31:23,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:31:24,430 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 07:31:24,676 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:31:24,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 07:31:24,737 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 07:31:24,776 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 07:31:24,805 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 07:31:24,825 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 07:31:24,845 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 07:31:24,872 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 07:31:24,898 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:31:25,801 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 07:31:25,801 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.13900017738. 2022-02-08 07:31:25,801 | test | DEBUG | Thread-10676 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 07:31:25,802 | infra | DEBUG | Thread-10676 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 07:31:26,788 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 07:31:27,315 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 07:31:27,503 | infra | DEBUG | Thread-10676 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 07:31:27,505 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 07:31:27,536 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 07:31:27,565 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 07:31:27,598 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 07:31:27,628 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 07:31:27,647 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 07:31:27,677 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 07:31:27,717 | infra | DEBUG | Thread-10676 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 07:31:28,869 | infra | DEBUG | Thread-10676 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 07:31:28,869 | test | INFO | Thread-10676 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.0689997673. 2022-02-08 07:31:28,875 | test | DEBUG | Thread-10676 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L836 2022-02-08 07:31:28,875 | test | DEBUG | Thread-10676 | [common_lib:sleep:24] Sleep 300 seconds. Reason: None 5 queries submitted, 0 failed, 9743 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 9765 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 9787 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 9813 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 9839 passed, 0 rejected, 0 cancelled, 0 timeout ############################################################ # # Step 11: Failover a node and RebalanceOut that node with loading in parallel # ############################################################ 2022-02-08 07:36:29,529 | test | DEBUG | MainThread | [rest_client:fail_over:1389] ns_1@172.23.121.128 - Failover successful 2022-02-08 07:36:29,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/Hospital.py -> SystemTestMagma():L600 2022-02-08 07:36:29,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 30 seconds. Reason: None 5 queries submitted, 0 failed, 9863 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:36:59,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.61 % 2022-02-08 07:36:59,585 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:36:59,585 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:02,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.63 % 2022-02-08 07:37:02,617 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:02,617 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:05,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.52 % 2022-02-08 07:37:05,638 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:05,638 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:08,655 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.46 % 2022-02-08 07:37:08,663 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:08,663 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:11,686 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.40 % 2022-02-08 07:37:11,697 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:11,697 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:14,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.41 % 2022-02-08 07:37:14,719 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:14,719 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:17,740 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.36 % 2022-02-08 07:37:17,747 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:17,747 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:20,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.30 % 2022-02-08 07:37:20,775 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:20,775 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:23,796 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.31 % 2022-02-08 07:37:23,805 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:23,806 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:26,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.19 % 2022-02-08 07:37:26,832 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:26,834 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:29,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.20 % 2022-02-08 07:37:29,861 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:29,861 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:32,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.15 % 2022-02-08 07:37:32,884 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:32,884 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:35,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.09 % 2022-02-08 07:37:35,910 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:35,910 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:38,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.09 % 2022-02-08 07:37:38,931 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:38,931 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:41,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.09 % 2022-02-08 07:37:41,973 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:41,973 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:44,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 68.98 % 2022-02-08 07:37:45,000 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:45,000 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:48,019 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 71.93 % 2022-02-08 07:37:48,028 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:48,028 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 9890 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:37:51,045 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 74.88 % 2022-02-08 07:37:51,055 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:51,056 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:54,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 79.76 % 2022-02-08 07:37:54,085 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:54,085 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:37:57,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 84.64 % 2022-02-08 07:37:57,118 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:37:57,118 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:38:00,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 94.96 % 2022-02-08 07:38:00,148 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:38:00,148 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:38:03,183 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 97.99 % 2022-02-08 07:38:03,191 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:38:03,191 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:38:06,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 97.99 % 2022-02-08 07:38:06,227 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:38:06,239 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:38:09,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 100.00 % 2022-02-08 07:38:09,282 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:38:09,282 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:38:12,321 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 100.00 % 2022-02-08 07:38:12,328 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:38:12,328 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:38:15,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 100.00 % 2022-02-08 07:38:15,374 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:38:15,374 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:38:18,420 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 93.33 % 2022-02-08 07:38:18,430 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:38:18,430 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:38:21,451 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-08 07:38:21,453 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 07:38:26,470 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:38:26,470 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:38:29,483 | test | INFO | MainThread | [rest_client:monitorRebalance:1599] Rebalance done. Taken 89.9440000057 seconds to complete 2022-02-08 07:38:29,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1602 2022-02-08 07:38:29,490 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-08 07:38:34,548 | test | DEBUG | MainThread | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.128,ns_1@172.23.121.115,ns_1@172.23.121.132,ns_1@172.23.121.135,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.134,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123,ns_1@172.23.121.131,ns_1@172.23.121.130,ns_1@172.23.121.133', 'ejectedNodes': u'ns_1@172.23.121.128', 'user': 'Administrator'} 2022-02-08 07:38:34,625 | test | DEBUG | MainThread | [rest_client:rebalance:1459] Rebalance operation started 2022-02-08 07:38:34,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 07:38:34,641 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:38:34,641 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:38:37,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 07:38:37,671 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:38:37,673 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:38:40,688 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 07:38:40,697 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:38:40,697 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:38:43,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 07:38:43,717 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:38:43,717 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:38:46,732 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 07:38:46,742 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:38:46,742 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 9912 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:38:49,760 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 07:38:49,769 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:38:49,770 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:38:52,788 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 07:38:52,795 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:38:52,795 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:38:55,808 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 07:38:55,815 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:38:55,815 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:38:58,826 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 07:38:58,835 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:38:58,835 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:01,872 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.64 % 2022-02-08 07:39:01,880 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:01,881 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:04,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.61 % 2022-02-08 07:39:04,901 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:04,901 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:07,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.61 % 2022-02-08 07:39:07,928 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:07,928 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:10,987 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.61 % 2022-02-08 07:39:10,997 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:10,999 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:14,010 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.61 % 2022-02-08 07:39:14,016 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:14,016 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:17,028 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.61 % 2022-02-08 07:39:17,035 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:17,035 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:20,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.61 % 2022-02-08 07:39:20,053 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:20,053 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:23,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.61 % 2022-02-08 07:39:23,075 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:23,076 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:26,091 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.93 % 2022-02-08 07:39:26,096 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:26,098 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:29,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.93 % 2022-02-08 07:39:29,118 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:29,118 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:32,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.93 % 2022-02-08 07:39:32,138 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:32,138 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:35,155 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.58 % 2022-02-08 07:39:35,163 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:35,163 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:38,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.58 % 2022-02-08 07:39:38,184 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:38,184 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:41,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.89 % 2022-02-08 07:39:41,203 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:41,203 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:44,221 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.13 % 2022-02-08 07:39:44,230 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:44,230 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:47,247 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.13 % 2022-02-08 07:39:47,256 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:47,256 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 9935 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:39:50,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.46 % 2022-02-08 07:39:50,276 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:50,276 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:53,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.77 % 2022-02-08 07:39:53,301 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:53,301 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:56,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.10 % 2022-02-08 07:39:56,328 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:56,328 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:39:59,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.10 % 2022-02-08 07:39:59,355 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:39:59,355 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:02,377 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.10 % 2022-02-08 07:40:02,384 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:02,384 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:05,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.10 % 2022-02-08 07:40:05,411 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:05,411 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:08,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.10 % 2022-02-08 07:40:08,437 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:08,438 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:11,492 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.10 % 2022-02-08 07:40:11,502 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:11,502 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:14,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.41 % 2022-02-08 07:40:14,529 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:14,529 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:17,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.06 % 2022-02-08 07:40:17,553 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:17,553 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:20,573 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.38 % 2022-02-08 07:40:20,582 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:20,584 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:23,609 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.71 % 2022-02-08 07:40:23,615 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:23,615 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:26,632 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.33 % 2022-02-08 07:40:26,648 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:26,648 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:29,673 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.96 % 2022-02-08 07:40:29,681 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:29,683 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:32,698 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.28 % 2022-02-08 07:40:32,706 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:32,707 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:35,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.91 % 2022-02-08 07:40:35,726 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:35,726 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:38,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.91 % 2022-02-08 07:40:38,746 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:38,746 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:41,759 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.22 % 2022-02-08 07:40:41,766 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:41,766 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:44,778 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.22 % 2022-02-08 07:40:44,785 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:44,786 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:47,811 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.52 % 2022-02-08 07:40:47,822 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:47,822 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 9952 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:40:50,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.47 % 2022-02-08 07:40:50,845 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:50,845 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:53,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.47 % 2022-02-08 07:40:53,869 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:53,869 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:56,885 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.47 % 2022-02-08 07:40:56,891 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:56,892 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:40:59,948 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.47 % 2022-02-08 07:40:59,956 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:40:59,957 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:02,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.76 % 2022-02-08 07:41:02,983 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:02,983 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:06,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.76 % 2022-02-08 07:41:06,006 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:06,007 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:09,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.41 % 2022-02-08 07:41:09,030 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:09,030 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:12,043 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.41 % 2022-02-08 07:41:12,051 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:12,051 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:15,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.38 % 2022-02-08 07:41:15,069 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:15,069 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:18,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.01 % 2022-02-08 07:41:18,091 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:18,092 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:21,164 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.31 % 2022-02-08 07:41:21,171 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:21,171 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:24,184 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.31 % 2022-02-08 07:41:24,191 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:24,193 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:27,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.31 % 2022-02-08 07:41:27,217 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:27,217 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:30,237 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.31 % 2022-02-08 07:41:30,246 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:30,246 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:33,260 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.64 % 2022-02-08 07:41:33,267 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:33,267 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:36,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.64 % 2022-02-08 07:41:36,303 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:36,303 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:39,318 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.96 % 2022-02-08 07:41:39,325 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:39,325 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:42,339 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.96 % 2022-02-08 07:41:42,348 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:42,348 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:45,359 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.96 % 2022-02-08 07:41:45,368 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:45,368 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:48,385 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.61 % 2022-02-08 07:41:48,394 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:48,395 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 9976 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:41:51,418 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.56 % 2022-02-08 07:41:51,427 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:51,428 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:54,441 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.88 % 2022-02-08 07:41:54,450 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:54,450 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:41:57,463 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.88 % 2022-02-08 07:41:57,469 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:41:57,470 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:00,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.88 % 2022-02-08 07:42:00,506 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:00,506 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:03,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.21 % 2022-02-08 07:42:03,572 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:03,572 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:06,586 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.21 % 2022-02-08 07:42:06,595 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:06,595 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:09,612 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.21 % 2022-02-08 07:42:09,625 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:09,625 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:12,640 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.21 % 2022-02-08 07:42:12,647 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:12,647 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:15,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.21 % 2022-02-08 07:42:15,680 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:15,680 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:18,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.51 % 2022-02-08 07:42:18,703 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:18,703 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:21,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.51 % 2022-02-08 07:42:21,727 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:21,729 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:24,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.15 % 2022-02-08 07:42:24,756 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:24,756 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:27,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.47 % 2022-02-08 07:42:27,808 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:27,809 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:30,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.09 % 2022-02-08 07:42:30,832 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:30,832 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:33,852 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.39 % 2022-02-08 07:42:33,861 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:33,861 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:36,872 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.39 % 2022-02-08 07:42:36,880 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:36,880 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:39,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.04 % 2022-02-08 07:42:39,904 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:39,904 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:42,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.04 % 2022-02-08 07:42:42,931 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:42,931 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:45,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.04 % 2022-02-08 07:42:45,951 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:45,951 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:48,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.36 % 2022-02-08 07:42:48,970 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:48,970 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10001 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:42:51,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.01 % 2022-02-08 07:42:52,006 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:52,006 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:55,017 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.34 % 2022-02-08 07:42:55,025 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:55,026 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:42:58,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.34 % 2022-02-08 07:42:58,049 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:42:58,049 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:01,066 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.34 % 2022-02-08 07:43:01,075 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:01,075 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:04,091 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.66 % 2022-02-08 07:43:04,099 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:04,099 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:07,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.99 % 2022-02-08 07:43:07,121 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:07,121 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:10,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.99 % 2022-02-08 07:43:10,148 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:10,148 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:13,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.31 % 2022-02-08 07:43:13,207 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:13,207 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:16,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.31 % 2022-02-08 07:43:16,236 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:16,236 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:19,247 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.64 % 2022-02-08 07:43:19,256 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:19,256 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:22,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.64 % 2022-02-08 07:43:22,279 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:22,279 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:25,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.64 % 2022-02-08 07:43:25,302 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:25,302 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:28,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.96 % 2022-02-08 07:43:28,334 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:28,335 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:31,352 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.29 % 2022-02-08 07:43:31,364 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:31,364 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:34,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.58 % 2022-02-08 07:43:34,390 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:34,391 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:37,408 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.58 % 2022-02-08 07:43:37,417 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:37,418 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:40,434 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.91 % 2022-02-08 07:43:40,441 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:40,441 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:43,454 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.23 % 2022-02-08 07:43:43,460 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:43,460 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:46,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 24.17 % 2022-02-08 07:43:46,493 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:46,493 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:49,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 24.50 % 2022-02-08 07:43:49,515 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:49,515 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10026 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:43:52,530 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.14 % 2022-02-08 07:43:52,538 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:52,539 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:55,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.14 % 2022-02-08 07:43:55,559 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:55,559 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:43:58,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.14 % 2022-02-08 07:43:58,582 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:43:58,582 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:01,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.14 % 2022-02-08 07:44:01,608 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:01,609 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:04,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.14 % 2022-02-08 07:44:04,628 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:04,628 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:07,642 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.14 % 2022-02-08 07:44:07,650 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:07,651 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:10,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.14 % 2022-02-08 07:44:10,670 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:10,670 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:13,683 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.14 % 2022-02-08 07:44:13,688 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:13,688 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:16,704 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.14 % 2022-02-08 07:44:16,713 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:16,713 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:19,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.79 % 2022-02-08 07:44:19,736 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:19,736 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:22,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.11 % 2022-02-08 07:44:22,756 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:22,756 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:25,770 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.11 % 2022-02-08 07:44:25,778 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:25,778 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:28,790 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.08 % 2022-02-08 07:44:28,798 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:28,798 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:31,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.40 % 2022-02-08 07:44:31,819 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:31,819 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:34,839 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.40 % 2022-02-08 07:44:34,848 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:34,848 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:37,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.40 % 2022-02-08 07:44:37,884 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:37,884 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:40,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.72 % 2022-02-08 07:44:40,907 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:40,907 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:43,920 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.05 % 2022-02-08 07:44:43,927 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:43,927 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:46,941 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.05 % 2022-02-08 07:44:46,948 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:46,948 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10051 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:44:49,966 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.37 % 2022-02-08 07:44:49,974 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:49,974 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:52,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.70 % 2022-02-08 07:44:52,993 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:52,993 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:56,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.02 % 2022-02-08 07:44:56,013 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:56,013 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:44:59,028 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.02 % 2022-02-08 07:44:59,035 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:44:59,035 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:02,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.35 % 2022-02-08 07:45:02,063 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:02,065 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:05,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.29 % 2022-02-08 07:45:05,102 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:05,104 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:08,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.29 % 2022-02-08 07:45:08,153 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:08,153 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:11,170 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.29 % 2022-02-08 07:45:11,180 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:11,180 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:14,196 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 31.24 % 2022-02-08 07:45:14,206 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:14,206 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:17,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 31.24 % 2022-02-08 07:45:17,243 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:17,243 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:20,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 31.24 % 2022-02-08 07:45:20,286 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:20,286 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:23,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 31.54 % 2022-02-08 07:45:23,326 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:23,328 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:26,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.19 % 2022-02-08 07:45:26,369 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:26,371 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:29,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.19 % 2022-02-08 07:45:29,398 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:29,398 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:32,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.51 % 2022-02-08 07:45:32,423 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:32,423 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:35,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.81 % 2022-02-08 07:45:35,446 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:35,446 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:38,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.81 % 2022-02-08 07:45:38,467 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:38,467 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:41,486 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.13 % 2022-02-08 07:45:41,496 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:41,496 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:44,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.13 % 2022-02-08 07:45:44,532 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:44,532 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:47,558 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.76 % 2022-02-08 07:45:47,566 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:47,566 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10078 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:45:50,579 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.76 % 2022-02-08 07:45:50,588 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:50,589 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:53,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.09 % 2022-02-08 07:45:53,612 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:53,612 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:56,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.09 % 2022-02-08 07:45:56,632 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:56,632 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:45:59,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.41 % 2022-02-08 07:45:59,651 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:45:59,651 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:02,665 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.01 % 2022-02-08 07:46:02,674 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:02,674 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:05,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.32 % 2022-02-08 07:46:05,697 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:05,697 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:08,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.62 % 2022-02-08 07:46:08,736 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:08,736 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:11,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.62 % 2022-02-08 07:46:11,785 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:11,785 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:14,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.62 % 2022-02-08 07:46:14,812 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:14,812 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:17,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.92 % 2022-02-08 07:46:17,836 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:17,836 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:20,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.92 % 2022-02-08 07:46:20,861 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:20,861 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:23,874 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.54 % 2022-02-08 07:46:23,881 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:23,881 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:26,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.54 % 2022-02-08 07:46:26,913 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:26,914 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:29,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.54 % 2022-02-08 07:46:29,931 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:29,931 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:32,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.85 % 2022-02-08 07:46:32,950 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:32,950 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:35,966 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.17 % 2022-02-08 07:46:35,971 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:35,973 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:38,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.47 % 2022-02-08 07:46:38,994 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:38,996 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:42,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.47 % 2022-02-08 07:46:42,017 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:42,019 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:45,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.47 % 2022-02-08 07:46:45,049 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:45,049 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:48,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.77 % 2022-02-08 07:46:48,072 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:48,072 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10104 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:46:51,124 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.37 % 2022-02-08 07:46:51,134 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:51,134 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:54,155 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.70 % 2022-02-08 07:46:54,163 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:54,164 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:46:57,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.70 % 2022-02-08 07:46:57,187 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:46:57,187 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:00,198 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.70 % 2022-02-08 07:47:00,206 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:00,207 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:03,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.33 % 2022-02-08 07:47:03,230 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:03,230 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:06,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.33 % 2022-02-08 07:47:06,252 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:06,252 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:09,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.33 % 2022-02-08 07:47:09,275 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:09,276 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:12,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.33 % 2022-02-08 07:47:12,301 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:12,302 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:15,316 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.64 % 2022-02-08 07:47:15,325 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:15,325 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:18,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.27 % 2022-02-08 07:47:18,357 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:18,358 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:21,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.27 % 2022-02-08 07:47:21,378 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:21,378 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:24,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.57 % 2022-02-08 07:47:24,403 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:24,403 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:27,418 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.57 % 2022-02-08 07:47:27,428 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:27,430 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:30,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.57 % 2022-02-08 07:47:30,459 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:30,459 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:33,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.87 % 2022-02-08 07:47:33,486 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:33,486 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:36,502 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.87 % 2022-02-08 07:47:36,510 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:36,512 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:39,525 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 41.20 % 2022-02-08 07:47:39,530 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:39,530 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:42,548 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 41.50 % 2022-02-08 07:47:42,556 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:42,558 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:45,573 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 41.50 % 2022-02-08 07:47:45,582 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:45,582 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:48,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.15 % 2022-02-08 07:47:48,605 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:48,605 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10132 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:47:51,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.15 % 2022-02-08 07:47:51,630 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:51,630 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:54,645 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.15 % 2022-02-08 07:47:54,653 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:54,653 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:47:57,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.15 % 2022-02-08 07:47:57,691 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:47:57,691 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:00,706 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.15 % 2022-02-08 07:48:00,711 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:00,711 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:03,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.15 % 2022-02-08 07:48:03,739 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:03,739 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:06,755 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.15 % 2022-02-08 07:48:06,763 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:06,765 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:09,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.48 % 2022-02-08 07:48:09,788 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:09,789 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:12,803 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.48 % 2022-02-08 07:48:12,811 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:12,811 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:15,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.78 % 2022-02-08 07:48:15,832 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:15,832 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:18,849 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 43.08 % 2022-02-08 07:48:18,857 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:18,857 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:21,872 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 43.40 % 2022-02-08 07:48:21,884 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:21,884 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:24,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 43.69 % 2022-02-08 07:48:24,907 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:24,907 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:27,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.34 % 2022-02-08 07:48:27,930 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:27,930 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:30,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.34 % 2022-02-08 07:48:30,951 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:30,951 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:33,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.63 % 2022-02-08 07:48:33,970 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:33,970 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:36,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.63 % 2022-02-08 07:48:36,993 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:36,994 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:40,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.63 % 2022-02-08 07:48:40,020 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:40,020 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:43,035 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.63 % 2022-02-08 07:48:43,043 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:43,043 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:46,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.63 % 2022-02-08 07:48:46,073 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:46,075 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:49,086 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.63 % 2022-02-08 07:48:49,092 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:49,092 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10152 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:48:52,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.96 % 2022-02-08 07:48:52,115 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:52,115 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:55,127 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.96 % 2022-02-08 07:48:55,134 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:55,134 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:48:58,147 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.91 % 2022-02-08 07:48:58,153 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:48:58,154 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:01,168 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.91 % 2022-02-08 07:49:01,176 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:01,176 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:04,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.91 % 2022-02-08 07:49:04,194 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:04,196 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:07,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.91 % 2022-02-08 07:49:07,220 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:07,220 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:10,234 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.91 % 2022-02-08 07:49:10,243 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:10,243 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:13,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.91 % 2022-02-08 07:49:13,263 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:13,263 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:16,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.91 % 2022-02-08 07:49:16,282 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:16,282 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:19,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.91 % 2022-02-08 07:49:19,301 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:19,301 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:22,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 46.21 % 2022-02-08 07:49:22,322 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:22,322 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:25,338 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 46.53 % 2022-02-08 07:49:25,345 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:25,345 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:28,357 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 46.83 % 2022-02-08 07:49:28,362 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:28,362 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:31,380 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 46.83 % 2022-02-08 07:49:31,387 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:31,388 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:34,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 46.83 % 2022-02-08 07:49:34,405 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:34,407 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:37,441 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 46.83 % 2022-02-08 07:49:37,448 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:37,448 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:40,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 46.83 % 2022-02-08 07:49:40,473 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:40,474 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:43,505 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 46.83 % 2022-02-08 07:49:43,516 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:43,517 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:46,530 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.16 % 2022-02-08 07:49:46,536 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:46,536 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:49,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 48.13 % 2022-02-08 07:49:49,563 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:49,563 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10173 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:49:52,576 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 48.13 % 2022-02-08 07:49:52,585 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:52,585 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:55,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 48.13 % 2022-02-08 07:49:55,609 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:55,611 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:49:58,635 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 48.43 % 2022-02-08 07:49:58,647 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:49:58,648 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:01,673 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 48.43 % 2022-02-08 07:50:01,681 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:01,681 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:04,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 48.43 % 2022-02-08 07:50:04,700 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:04,701 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:07,714 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 48.74 % 2022-02-08 07:50:07,721 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:07,721 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:10,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 48.74 % 2022-02-08 07:50:10,746 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:10,746 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:13,760 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.04 % 2022-02-08 07:50:13,767 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:13,767 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:16,780 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.04 % 2022-02-08 07:50:16,788 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:16,788 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:19,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.04 % 2022-02-08 07:50:19,808 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:19,808 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:22,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.04 % 2022-02-08 07:50:22,838 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:22,838 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:25,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.04 % 2022-02-08 07:50:25,858 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:25,859 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:28,888 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.04 % 2022-02-08 07:50:28,895 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:28,895 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:31,907 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.66 % 2022-02-08 07:50:31,914 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:31,914 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:34,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.66 % 2022-02-08 07:50:34,936 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:34,936 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:37,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.66 % 2022-02-08 07:50:37,964 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:37,964 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:40,989 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.99 % 2022-02-08 07:50:41,010 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:41,012 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:44,025 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.99 % 2022-02-08 07:50:44,032 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:44,032 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:47,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 50.31 % 2022-02-08 07:50:47,062 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:47,062 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10192 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:50:50,079 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 50.31 % 2022-02-08 07:50:50,088 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:50,088 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:53,104 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 50.31 % 2022-02-08 07:50:53,111 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:53,111 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:56,127 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 50.31 % 2022-02-08 07:50:56,137 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:56,138 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:50:59,153 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 50.64 % 2022-02-08 07:50:59,161 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:50:59,163 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:02,176 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 50.64 % 2022-02-08 07:51:02,183 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:02,183 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:05,196 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 50.64 % 2022-02-08 07:51:05,203 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:05,203 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:08,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 50.96 % 2022-02-08 07:51:08,230 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:08,230 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:11,249 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 50.96 % 2022-02-08 07:51:11,256 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:11,257 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:14,272 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.28 % 2022-02-08 07:51:14,282 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:14,282 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:17,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.28 % 2022-02-08 07:51:17,303 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:17,303 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:20,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.28 % 2022-02-08 07:51:20,323 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:20,323 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:23,339 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.28 % 2022-02-08 07:51:23,355 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:23,355 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:26,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.58 % 2022-02-08 07:51:26,377 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:26,377 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:29,395 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.58 % 2022-02-08 07:51:29,405 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:29,405 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:32,418 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.90 % 2022-02-08 07:51:32,427 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:32,427 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:35,441 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.90 % 2022-02-08 07:51:35,450 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:35,450 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:38,469 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.20 % 2022-02-08 07:51:38,479 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:38,479 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:41,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.53 % 2022-02-08 07:51:41,516 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:41,516 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:44,535 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.53 % 2022-02-08 07:51:44,543 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:44,543 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:47,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.53 % 2022-02-08 07:51:47,569 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:47,569 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10215 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:51:50,584 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.53 % 2022-02-08 07:51:50,591 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:50,591 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:53,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.53 % 2022-02-08 07:51:53,615 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:53,615 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:56,630 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.84 % 2022-02-08 07:51:56,638 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:56,640 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:51:59,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.84 % 2022-02-08 07:51:59,684 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:51:59,684 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:02,704 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.84 % 2022-02-08 07:52:02,710 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:02,711 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:05,734 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.84 % 2022-02-08 07:52:05,743 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:05,743 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:08,755 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.84 % 2022-02-08 07:52:08,763 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:08,763 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:11,775 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.84 % 2022-02-08 07:52:11,782 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:11,782 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:14,796 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.84 % 2022-02-08 07:52:14,805 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:14,805 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:17,818 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 53.14 % 2022-02-08 07:52:17,825 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:17,825 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:20,841 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 53.14 % 2022-02-08 07:52:20,848 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:20,848 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:23,862 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 53.14 % 2022-02-08 07:52:23,869 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:23,869 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:26,885 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 53.47 % 2022-02-08 07:52:26,895 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:26,897 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:29,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 53.79 % 2022-02-08 07:52:29,921 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:29,921 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:32,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 53.79 % 2022-02-08 07:52:32,946 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:32,947 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:35,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 54.11 % 2022-02-08 07:52:35,974 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:35,974 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:38,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 54.11 % 2022-02-08 07:52:39,005 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:39,005 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:42,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 54.11 % 2022-02-08 07:52:42,032 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:42,032 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:45,045 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 54.11 % 2022-02-08 07:52:45,051 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:45,051 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:48,063 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 54.11 % 2022-02-08 07:52:48,072 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:48,072 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10239 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:52:51,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 54.41 % 2022-02-08 07:52:51,096 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:51,098 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:54,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 54.74 % 2022-02-08 07:52:54,118 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:54,118 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:52:57,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 54.74 % 2022-02-08 07:52:57,147 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:52:57,148 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:00,160 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 54.74 % 2022-02-08 07:53:00,167 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:00,167 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:03,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 55.03 % 2022-02-08 07:53:03,188 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:03,190 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:06,201 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 55.03 % 2022-02-08 07:53:06,210 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:06,210 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:09,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 55.33 % 2022-02-08 07:53:09,236 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:09,236 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:12,267 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 55.95 % 2022-02-08 07:53:12,273 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:12,273 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:15,286 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.28 % 2022-02-08 07:53:15,292 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:15,292 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:18,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.28 % 2022-02-08 07:53:18,315 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:18,315 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:21,326 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.28 % 2022-02-08 07:53:21,336 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:21,336 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:24,349 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.28 % 2022-02-08 07:53:24,355 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:24,355 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:27,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.28 % 2022-02-08 07:53:27,385 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:27,385 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:30,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.28 % 2022-02-08 07:53:30,410 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:30,410 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:33,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.28 % 2022-02-08 07:53:33,440 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:33,441 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:36,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.28 % 2022-02-08 07:53:36,471 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:36,473 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:39,486 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.28 % 2022-02-08 07:53:39,496 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:39,496 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:42,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.28 % 2022-02-08 07:53:42,552 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:42,552 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:45,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.28 % 2022-02-08 07:53:45,573 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:45,573 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:48,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.60 % 2022-02-08 07:53:48,611 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:48,612 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10263 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:53:51,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.60 % 2022-02-08 07:53:51,635 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:51,637 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:54,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.90 % 2022-02-08 07:53:54,655 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:54,655 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:53:57,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.23 % 2022-02-08 07:53:57,674 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:53:57,674 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:00,687 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.23 % 2022-02-08 07:54:00,694 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:00,694 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:03,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.23 % 2022-02-08 07:54:03,717 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:03,717 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:06,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.23 % 2022-02-08 07:54:06,744 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:06,746 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:09,762 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.23 % 2022-02-08 07:54:09,769 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:09,769 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:12,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.23 % 2022-02-08 07:54:12,790 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:12,790 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:15,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.54 % 2022-02-08 07:54:15,809 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:15,809 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:18,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.54 % 2022-02-08 07:54:18,832 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:18,832 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:21,846 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.87 % 2022-02-08 07:54:21,857 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:21,857 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:24,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.87 % 2022-02-08 07:54:24,877 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:24,878 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:27,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 58.79 % 2022-02-08 07:54:27,911 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:27,911 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:30,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.09 % 2022-02-08 07:54:30,938 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:30,938 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:33,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.09 % 2022-02-08 07:54:33,971 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:33,971 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:36,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.09 % 2022-02-08 07:54:37,000 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:37,002 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:40,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.09 % 2022-02-08 07:54:40,019 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:40,019 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:43,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.09 % 2022-02-08 07:54:43,049 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:43,049 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:46,066 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.09 % 2022-02-08 07:54:46,076 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:46,078 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:49,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.09 % 2022-02-08 07:54:49,101 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:49,101 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10289 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:54:52,119 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.09 % 2022-02-08 07:54:52,125 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:52,125 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:55,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.09 % 2022-02-08 07:54:55,145 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:55,145 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:54:58,163 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.09 % 2022-02-08 07:54:58,173 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:54:58,174 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:01,184 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.09 % 2022-02-08 07:55:01,190 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:01,190 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:04,206 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.40 % 2022-02-08 07:55:04,213 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:04,213 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:07,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.40 % 2022-02-08 07:55:07,233 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:07,233 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:10,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.40 % 2022-02-08 07:55:10,252 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:10,253 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:13,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.40 % 2022-02-08 07:55:13,285 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:13,286 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:16,301 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.40 % 2022-02-08 07:55:16,309 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:16,309 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:19,331 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.70 % 2022-02-08 07:55:19,339 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:19,339 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:22,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.70 % 2022-02-08 07:55:22,368 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:22,369 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:25,382 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.70 % 2022-02-08 07:55:25,391 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:25,391 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:28,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.70 % 2022-02-08 07:55:28,410 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:28,410 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:31,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.70 % 2022-02-08 07:55:31,436 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:31,437 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:34,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.70 % 2022-02-08 07:55:34,460 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:34,460 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:37,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.00 % 2022-02-08 07:55:37,516 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:37,516 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:40,533 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.00 % 2022-02-08 07:55:40,542 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:40,542 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:43,563 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.32 % 2022-02-08 07:55:43,572 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:43,572 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:46,589 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.32 % 2022-02-08 07:55:46,598 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:46,598 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10312 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:55:49,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.32 % 2022-02-08 07:55:49,618 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:49,618 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:52,668 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.32 % 2022-02-08 07:55:52,680 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:52,680 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:55,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.32 % 2022-02-08 07:55:55,701 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:55,703 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:55:58,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.32 % 2022-02-08 07:55:58,733 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:55:58,733 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:01,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.32 % 2022-02-08 07:56:01,766 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:01,766 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:04,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.32 % 2022-02-08 07:56:04,792 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:04,792 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:07,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.32 % 2022-02-08 07:56:07,822 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:07,822 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:10,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.32 % 2022-02-08 07:56:10,849 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:10,849 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:13,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.32 % 2022-02-08 07:56:13,887 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:13,887 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:16,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.64 % 2022-02-08 07:56:16,943 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:16,944 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:19,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.64 % 2022-02-08 07:56:19,970 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:19,970 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:22,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.94 % 2022-02-08 07:56:22,993 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:22,993 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:26,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.94 % 2022-02-08 07:56:26,023 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:26,023 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:29,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.94 % 2022-02-08 07:56:29,048 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:29,048 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:32,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.94 % 2022-02-08 07:56:32,071 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:32,072 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:35,086 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 61.26 % 2022-02-08 07:56:35,095 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:35,095 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:38,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 61.26 % 2022-02-08 07:56:38,128 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:38,128 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:41,147 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 61.26 % 2022-02-08 07:56:41,155 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:41,155 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:44,168 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 61.26 % 2022-02-08 07:56:44,177 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:44,177 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:47,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 61.26 % 2022-02-08 07:56:47,197 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:47,197 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10331 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:56:50,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 61.58 % 2022-02-08 07:56:50,221 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:50,221 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:53,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 61.58 % 2022-02-08 07:56:53,283 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:53,283 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:56,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 61.58 % 2022-02-08 07:56:56,302 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:56,302 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:56:59,318 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 61.58 % 2022-02-08 07:56:59,325 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:56:59,325 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:02,336 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 61.88 % 2022-02-08 07:57:02,344 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:02,345 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:05,357 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 61.88 % 2022-02-08 07:57:05,365 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:05,365 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:08,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.18 % 2022-02-08 07:57:08,391 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:08,391 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:11,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.18 % 2022-02-08 07:57:11,418 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:11,420 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:14,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.18 % 2022-02-08 07:57:14,456 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:14,457 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:17,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.18 % 2022-02-08 07:57:17,496 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:17,496 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:20,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.18 % 2022-02-08 07:57:20,526 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:20,526 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:23,546 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.18 % 2022-02-08 07:57:23,559 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:23,559 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:26,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.18 % 2022-02-08 07:57:26,582 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:26,584 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:29,596 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.18 % 2022-02-08 07:57:29,607 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:29,607 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:32,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.18 % 2022-02-08 07:57:32,640 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:32,641 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:35,655 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.18 % 2022-02-08 07:57:35,664 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:35,664 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:38,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.47 % 2022-02-08 07:57:38,688 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:38,688 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:41,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.47 % 2022-02-08 07:57:41,711 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:41,711 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:44,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.47 % 2022-02-08 07:57:44,734 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:44,734 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:47,757 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.47 % 2022-02-08 07:57:47,766 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:47,767 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10355 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:57:50,780 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.47 % 2022-02-08 07:57:50,788 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:50,789 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:53,808 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.47 % 2022-02-08 07:57:53,815 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:53,815 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:56,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.47 % 2022-02-08 07:57:56,836 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:56,836 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:57:59,849 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.47 % 2022-02-08 07:57:59,857 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:57:59,857 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:02,882 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.77 % 2022-02-08 07:58:02,894 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:02,894 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:05,907 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.77 % 2022-02-08 07:58:05,915 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:05,915 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:08,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.77 % 2022-02-08 07:58:08,943 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:08,943 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:11,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.77 % 2022-02-08 07:58:11,967 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:11,967 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:14,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.77 % 2022-02-08 07:58:14,999 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:15,000 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:18,035 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.77 % 2022-02-08 07:58:18,046 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:18,046 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:21,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.77 % 2022-02-08 07:58:21,073 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:21,073 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:24,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.77 % 2022-02-08 07:58:24,098 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:24,098 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:27,112 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.77 % 2022-02-08 07:58:27,119 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:27,119 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:30,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.77 % 2022-02-08 07:58:30,148 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:30,148 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:33,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.77 % 2022-02-08 07:58:33,180 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:33,180 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:36,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.10 % 2022-02-08 07:58:36,206 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:36,206 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:39,221 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.10 % 2022-02-08 07:58:39,232 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:39,232 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:42,252 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.10 % 2022-02-08 07:58:42,263 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:42,263 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:45,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.10 % 2022-02-08 07:58:45,285 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:45,286 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:48,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.10 % 2022-02-08 07:58:48,306 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:48,308 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10379 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:58:51,321 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.10 % 2022-02-08 07:58:51,328 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:51,328 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:54,344 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.10 % 2022-02-08 07:58:54,351 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:54,351 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:58:57,369 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.10 % 2022-02-08 07:58:57,378 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:58:57,378 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:00,410 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.10 % 2022-02-08 07:59:00,417 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:00,417 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:03,434 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.10 % 2022-02-08 07:59:03,443 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:03,443 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:06,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.39 % 2022-02-08 07:59:06,466 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:06,466 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:09,489 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.39 % 2022-02-08 07:59:09,500 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:09,500 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:12,538 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.39 % 2022-02-08 07:59:12,548 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:12,549 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:15,566 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.39 % 2022-02-08 07:59:15,578 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:15,578 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:18,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.39 % 2022-02-08 07:59:18,618 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:18,618 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:21,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.39 % 2022-02-08 07:59:21,638 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:21,638 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:24,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.39 % 2022-02-08 07:59:24,661 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:24,661 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:27,673 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.39 % 2022-02-08 07:59:27,680 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:27,680 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:30,693 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.69 % 2022-02-08 07:59:30,700 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:30,700 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:33,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.69 % 2022-02-08 07:59:33,720 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:33,720 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:36,733 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.69 % 2022-02-08 07:59:36,742 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:36,743 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:39,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.99 % 2022-02-08 07:59:39,790 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:39,790 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:42,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.99 % 2022-02-08 07:59:42,816 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:42,818 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:45,832 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.99 % 2022-02-08 07:59:45,842 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:45,842 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:48,859 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.99 % 2022-02-08 07:59:48,868 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:48,868 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 10403 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 07:59:51,885 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.99 % 2022-02-08 07:59:51,895 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:51,897 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:54,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.99 % 2022-02-08 07:59:54,933 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:54,933 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 07:59:57,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.99 % 2022-02-08 07:59:57,956 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 07:59:57,956 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 08:00:00,969 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.99 % 2022-02-08 08:00:00,976 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 08:00:00,976 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 08:00:03,997 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 64.29 % 2022-02-08 08:00:04,006 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 08:00:04,006 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 08:00:07,033 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-08 08:00:07,033 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 08:00:12,049 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 08:00:12,051 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 08:00:15,063 | test | INFO | MainThread | [rest_client:monitorRebalance:1599] Rebalance done. Taken 1300.43799996 seconds to complete 2022-02-08 08:00:15,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1602 2022-02-08 08:00:15,071 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-08 08:00:20,145 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3970] Stats: {u'opsPerSec': 18477.8, u'diskFetches': 9, u'dataUsed': 137459281758L, u'itemCount': 127722786, u'quotaPercentUsed': 79.48009802424122, u'storageTotals': {u'hdd': {u'usedByData': 245315503301L, u'used': 352628478560L, u'quotaTotal': 1168799956992L, u'total': 1168799956992L, u'free': 692076519018L}, u'ram': {u'usedByData': 61888136208L, u'used': 107078262784L, u'quotaTotal': 78809923584L, u'total': 110647066624L, u'quotaUsed': 77866205184L, u'quotaUsedPerNode': 8651800576L, u'quotaTotalPerNode': 8756658176L}}, u'diskUsed': 245315503301L, u'vbActiveNumNonResident': 97282292, u'memUsed': 61888136208L} 2022-02-08 08:00:21,121 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+-----------+-----------+-----------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+-----------+-----------+-----------+------------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 127722786 | 72.52 GiB | 57.64 GiB | 228.47 GiB | 23.8332524316 | +-----------------+-----------+-----------------+----------+------------+-----+-----------+-----------+-----------+------------+---------------+ 2022-02-08 08:00:21,121 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+------------------+------------------+-----------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+------------------+------------------+-----------------+---------------+ | 40000000 | 380000000 | 2000000-20000000 | 2000000-20000000 | 1000000-2000000 | 0-0 | +---------------+---------------+------------------+------------------+-----------------+---------------+ 2022-02-08 08:00:21,421 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 23.8332524316 2022-02-08 08:00:21,424 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:00:21,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:22,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:00:22,306 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:22,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:00:22,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:00:22,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:00:22,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:00:22,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:00:22,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:00:22,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:00:22,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 08:00:22,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 08:00:22,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 08:00:22,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 08:00:22,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 08:00:23,246 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:00:23,247 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:00:23,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:24,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:00:24,473 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:24,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:00:24,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:00:24,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:00:24,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:00:24,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:00:24,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:00:24,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:00:24,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 08:00:24,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 08:00:24,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 08:00:24,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 08:00:25,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 08:00:25,546 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:00:25,549 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 08:00:25,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:25,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 08:00:25,923 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:25,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 08:00:25,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 08:00:25,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 08:00:25,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 08:00:26,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 08:00:26,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 08:00:26,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 08:00:26,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 08:00:26,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 08:00:26,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 08:00:26,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 08:00:26,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 08:00:26,173 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 08:00:26,174 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:00:26,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:26,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:00:27,076 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:27,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:00:27,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:00:27,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:00:27,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:00:27,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:00:27,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:00:27,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:00:27,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 08:00:27,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 08:00:27,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 08:00:27,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 08:00:27,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 08:00:28,088 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:00:28,091 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 08:00:28,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:28,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 08:00:28,490 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:28,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 08:00:28,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 08:00:28,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 08:00:28,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 08:00:28,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 08:00:28,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 08:00:28,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 08:00:28,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 08:00:28,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 08:00:28,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 08:00:28,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 08:00:28,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 08:00:28,750 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 08:00:28,753 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:00:29,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:29,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:00:30,198 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:30,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:00:30,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:00:30,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:00:30,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:00:30,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:00:30,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:00:30,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:00:30,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 08:00:30,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 08:00:30,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 08:00:30,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 08:00:30,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 08:00:31,246 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:00:31,249 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:00:31,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:32,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:00:32,368 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:32,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:00:32,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:00:32,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:00:32,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:00:32,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:00:32,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:00:32,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:00:32,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 08:00:32,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 08:00:32,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 08:00:32,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 08:00:32,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 08:00:33,375 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:00:33,378 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 08:00:33,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:33,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 08:00:33,894 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:33,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 08:00:33,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 08:00:33,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 08:00:33,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 08:00:33,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 08:00:33,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 08:00:33,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 08:00:34,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 08:00:34,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 08:00:34,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 08:00:34,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 08:00:34,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 08:00:34,128 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 08:00:34,134 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 08:00:34,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:34,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 08:00:34,516 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:34,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 08:00:34,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 08:00:34,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 08:00:34,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 08:00:34,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 08:00:34,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 08:00:34,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 08:00:34,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 08:00:34,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 08:00:34,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 08:00:34,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 08:00:34,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 08:00:34,750 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 08:00:34,753 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:00:35,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:36,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:00:36,276 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:36,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:00:36,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:00:36,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:00:36,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:00:36,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:00:36,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:00:36,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:00:36,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 08:00:36,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 08:00:36,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 08:00:36,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 08:00:36,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 08:00:37,417 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:00:37,418 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:00:37,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:38,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:00:38,249 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:38,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:00:38,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:00:38,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:00:38,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:00:38,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:00:38,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:00:38,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:00:38,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 08:00:38,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 08:00:38,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 08:00:38,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 08:00:38,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 08:00:39,427 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:00:39,428 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:00:40,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:40,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:00:40,673 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:40,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:00:40,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:00:40,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:00:40,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:00:40,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:00:40,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:00:40,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:00:40,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 08:00:40,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 08:00:40,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 08:00:41,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 08:00:41,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 08:00:41,757 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:00:41,759 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:00:42,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:42,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:00:42,681 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:42,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:00:42,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:00:42,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:00:42,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:00:42,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:00:42,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:00:42,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:00:42,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 08:00:43,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 08:00:43,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 08:00:43,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 08:00:43,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 08:00:43,765 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:00:43,767 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 08:00:43,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:44,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 08:00:44,151 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:44,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 08:00:44,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 08:00:44,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 08:00:44,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 08:00:44,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 08:00:44,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 08:00:44,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 08:00:44,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 08:00:44,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 08:00:44,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 08:00:44,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 08:00:44,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 08:00:44,362 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 08:00:44,362 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for kvstore is 230530MB 2022-02-08 08:00:44,364 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for wal is 5199MB 2022-02-08 08:00:44,364 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:642] Total Disk usage for keyTree is 15647MB 2022-02-08 08:00:44,364 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:643] Total Disk usage for seqTree is 214595MB 2022-02-08 08:00:44,368 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:00:44,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:44,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:00:44,756 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:44,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:00:44,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:00:44,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:00:44,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:00:44,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:00:44,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:00:44,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:00:44,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 08:00:44,907 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.116 - core(s): 8 2022-02-08 08:00:44,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 08:00:45,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 08:00:45,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 08:00:45,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 08:00:46,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 08:00:46,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 08:00:46,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 08:00:46,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 08:00:46,765 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:00:46,766 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:00:46,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:47,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:00:47,213 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:47,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:00:47,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:00:47,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:00:47,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:00:47,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:00:47,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:00:47,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:00:47,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 08:00:47,417 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.123 - core(s): 8 2022-02-08 08:00:47,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 08:00:48,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 08:00:48,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 08:00:48,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 08:00:48,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 08:00:48,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 08:00:48,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 08:00:49,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 08:00:49,275 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:00:49,278 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:00:49,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:49,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 5 queries submitted, 0 failed, 10427 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:00:49,638 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:49,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:00:49,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:00:49,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:00:49,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:00:49,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:00:49,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:00:49,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:00:49,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 08:00:49,802 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.124 - core(s): 8 2022-02-08 08:00:49,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 08:00:50,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 08:00:50,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 08:00:50,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 08:00:50,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 08:00:50,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 08:00:51,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 08:00:51,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 08:00:51,440 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:00:51,443 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:00:51,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:51,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:00:51,849 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:51,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:00:51,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:00:51,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:00:51,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:00:51,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:00:51,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:00:51,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:00:51,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 08:00:51,989 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.126 - core(s): 8 2022-02-08 08:00:51,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 08:00:52,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 08:00:52,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 08:00:53,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 08:00:53,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 08:00:53,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 08:00:53,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 08:00:53,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 08:00:53,845 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:00:53,846 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:00:53,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:54,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:00:54,345 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:54,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:00:54,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:00:54,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:00:54,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:00:54,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:00:54,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:00:54,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:00:54,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 08:00:54,536 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.127 - core(s): 8 2022-02-08 08:00:54,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 08:00:55,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 08:00:55,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 08:00:55,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 08:00:55,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 08:00:55,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 08:00:56,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 08:00:56,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 08:00:56,576 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:00:56,578 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:00:56,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:56,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:00:56,963 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:56,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:00:56,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:00:56,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:00:57,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:00:57,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:00:57,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:00:57,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:00:57,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 08:00:57,147 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.129 - core(s): 8 2022-02-08 08:00:57,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 08:00:57,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 08:00:58,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 08:00:58,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 08:00:58,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 08:00:58,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 08:00:58,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 08:00:58,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 08:00:59,206 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:00:59,211 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:00:59,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:00:59,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:00:59,697 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:00:59,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:00:59,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:00:59,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:00:59,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:00:59,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:00:59,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:00:59,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:00:59,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 08:00:59,897 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.135 - core(s): 8 2022-02-08 08:00:59,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 08:01:00,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 08:01:00,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 08:01:00,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 08:01:00,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 08:01:01,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 08:01:01,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 08:01:01,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 08:01:01,569 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:01:01,571 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:01:01,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:01:01,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:01:01,960 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:01:01,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:01:01,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:01:01,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:01:02,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:01:02,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:01:02,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:01:02,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:01:02,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 08:01:02,089 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.115 - core(s): 8 2022-02-08 08:01:02,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 08:01:02,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 08:01:02,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 08:01:03,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 08:01:03,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 08:01:03,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 08:01:03,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 08:01:03,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 08:01:03,943 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:01:03,973 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:01:04,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:01:04,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:01:04,549 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:01:04,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:01:04,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:01:04,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:01:04,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:01:04,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:01:04,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:01:04,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:01:04,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 08:01:04,773 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.120.170 - core(s): 8 2022-02-08 08:01:04,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 08:01:05,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 08:01:05,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 08:01:05,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 08:01:05,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 08:01:05,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 08:01:06,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 08:01:06,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 08:01:06,706 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:01:06,706 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:723] {'172.23.121.116': {'rw_0:magma': {u'NReadBytesGetPerSec': 912703.875, u'ReplayWALOffset': u'1626:5570560', u'NTablesCreated': 18040, u'RecentReadIOAmpSet': 0.025093598291277885, u'ReadIOAmp': 2.895728826522827, u'BlockCacheMisses': 279933, u'NFlushes': 4418, u'RecentWriteAmp': 2.5003819465637207, u'NumSyncFlushes': 82, u'TotalMemUsed': 71164247, u'NBloomFilterHits': 7708731, u'Fragmentation': 0.36945587396621704, u'NumNormalFlushes': 2238, u'RecentBlockCacheHitRatio': 57.28633499145508, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 8754, u'ReadIOAmp': 49.934065934065934, u'MemtablesMemUsed': 5023160, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 27064800, u'CheckpointOverhead': 15601664, u'NItemsGet': 91, u'NWriteIOs': 11839, u'NReadBytes': 265626971, u'LevelStats': [{u'TargetSize': 130023424, u'LogicalSize': 22886564, u'SizeRatio': 0.1760187764321604, u'NWriteBytes': 38305792, u'BloomFilterMemUsed': 0, u'NumTables': 262, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2547712}, {u'TargetSize': 130023424, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 130023424, u'LogicalSize': 102699506, u'SizeRatio': 0.7898538804823353, u'NWriteBytes': 17797120, u'BloomFilterMemUsed': 0, u'NumTables': 62, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 1708032}, {u'TargetSize': 1615235795, u'LogicalSize': 1183921319, u'SizeRatio': 0.7329712000346055, u'NWriteBytes': 6852608, u'BloomFilterMemUsed': 0, u'NumTables': 532, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 18673664}, {u'TargetSize': 6306311820L, u'LogicalSize': 3230471595L, u'SizeRatio': 0.5122600479022935, u'NWriteBytes': 964734976, u'BloomFilterMemUsed': 0, u'NumTables': 1113, u'WastedSpace': 77315685, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 3230429184L}], u'NItemsGarbageCollected': 764054, u'NInternalFragmentationCompacts': 18, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 22528560, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 235584, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 122897, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 692028, u'CompactWriter': {u'NDataBlockWrites': 60316, u'NWriteBytes': 180047872, u'NIndexBlockWrites': 1064, u'NBytesIncoming': 277844547, u'NSyncs': 0, u'NInserts': 1515131, u'NWriteIOs': 978}, u'NonResidentBloomFilterSize': 0, u'IteratorMemUsed': 0, u'NWriteBytes': 1031174395, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 4544, u'NReadBytes': 812998, u'NDataBlockReads': 91, u'NBytesOutgoing': 0, u'NIndexBlockReads': 182}, u'NBloomFilterMisses': 0, u'NItemsMerged': 202695, u'LogicalDiskSize': 3230471595L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.5723684338994564, u'BufferMemUsed': 1329152, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 2046550276, u'NTableFiles': 2300, u'NFileCountCompacts': 216, u'TableObjectMemUsed': 854400, u'TreeSnapshotMemoryUsed': 333424, u'TotalDiskUsage': 3861827584L, u'BytesIncoming': 1801592006, u'NItemsDel': 0, u'PendingCheckpointBytes': 565248, u'NCompacts': 430, u'NItemsPut': 2568351, u'NSyncs': 2191, u'CompactReader': {u'NReadIOs': 110811, u'NReadBytes': 262074185, u'NDataBlockReads': 105631, u'NBytesOutgoing': 0, u'NIndexBlockReads': 5180}, u'ActiveDiskUsage': 3253358592L, u'NTablesDeleted': 4441, u'IterateReader': {u'NReadIOs': 7542, u'NReadBytes': 2739788, u'NDataBlockReads': 3843, u'NBytesOutgoing': 0, u'NIndexBlockReads': 3699}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 279544, u'NWriteBytes': 851126523, u'NIndexBlockWrites': 14028, u'NBytesIncoming': 1286227557, u'NSyncs': 2191, u'NInserts': 1807650, u'NWriteIOs': 10861}}, u'NReadBytes': 2242833395L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 289.20001220703125, u'NInternalFragmentationCompacts': 18, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 25123400, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 24629, u'NTxns': 258439, u'NWriteBytes': 2667712512L, u'NSegments': 179, u'NSyncs': 258439, u'NLoggedBytes': 2667712512L, u'NActiveSegments': 2, u'NWrites': 264788, u'PreallocateSupported': True, u'NInserts': 3500911, u'DiskUsed': 37666816}, u'NGets': 155364, u'NReadBytesSet': 103416714, u'NReadBytesCompact': 847937222, u'IndexResidentRatio': 0.55007404088974, u'CompactQueueSize': 0, u'NSyncsPerSec': 5.300000190734863, u'BlockCacheHitRatio': 61.50271987915039, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 1392156385, u'BloomFilterMemUsedRatio': 0.05026046186685562, u'NonResidentBloomFilterSize': 1349759, u'NWriteBytes': 4498184393L, u'BytesOutgoing': 81679, u'RecentReadIOAmp': 2.5675389766693115, u'WriteAmp': 2.0994818210601807, u'NBlocksCached': 16144, u'LogicalDataSize': 2115900227, u'ActiveBloomFilterMemUsed': 5133464, u'NFileCountCompacts': 721, u'NTableFiles': 3642, u'FlushQueueSize': 8, u'TableObjectMemUsed': 1176480, u'BlockCacheQuota': 80172486, u'TotalDiskUsage': 4128313344L, u'NPurgerRuns': 0, u'ReadIOAmpSet': 0.0, u'NSyncs': 6573, u'NReadIOsPerSec': 740.2666625976562, u'NReadIOsSetPerSec': 27.03333282470703, u'NTablesDeleted': 13677, u'BloomFilterFPR': 0.016065282747149467, u'NBloomFilterHitsPerSec': 6793.7998046875, u'WriteCacheMemUsed': 7401768, u'RecentBloomFilterFPR': 0.015277133323252201, u'BytesPerRead': 8294.9697265625, u'BytesIncomingPerSec': 1205735.75, u'NReadIOsGetPerSec': 323.76666259765625, u'NumKVStores': 31, u'NGetStats': 1702, u'TotalIndexBlocksSize': 30348401, u'NGetStatsPerSec': 0.9666666388511658, u'NWriteBytesCompact': 822964224, u'NReadIOsSet': 36025, u'ActiveDataSize': 2115900227, u'NWriteBytesCompactPerSec': 1000516.25, u'NBloomFilterMissesPerSec': 105.4000015258789, u'NCommitBatches': 261541, u'NGetStatsComputed': 1702, u'NWriteIOs': 291373, u'NReadIOsGet': 485917, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 20043122, u'LSMTreeObjectMemUsed': 534688, u'NSetsPerSec': 1077.300048828125, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 3.366666555404663, u'ManagedFileObjectMemUsed': 364416, u'ReadAmpGet': 15778.103515625, u'BlockCacheMemUsed': 52854264, u'NWriterCompacts': 0, u'NReadIOs': 812960, u'WALBufferMemUsed': 81920, u'ReadAmp': 26192.984375, u'keyStats': {u'NItemsTransformed': 11112904, u'NTablesCreated': 4721, u'ReadIOAmp': 0.33380337426219364, u'MemtablesMemUsed': 796088, u'NBloomFilterHits': 7707921, u'TotalIndexBlocksSize': 3252887, u'CheckpointOverhead': 46665728, u'NItemsGet': 1435336, u'NWriteIOs': 7990, u'NReadBytes': 1968649692, u'LevelStats': [{u'TargetSize': 130023424, u'LogicalSize': 4672234, u'SizeRatio': 0.03593378682290354, u'NWriteBytes': 117612544, u'BloomFilterMemUsed': 54528, u'NumTables': 225, u'WastedSpace': 286, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 4247552}, {u'TargetSize': 130023424, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 130023424, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 0}, {u'TargetSize': 130023424, u'LogicalSize': 75582550, u'SizeRatio': 0.581299489544284, u'NWriteBytes': 559173632, u'BloomFilterMemUsed': 2659840, u'NumTables': 53, u'WastedSpace': 156303, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 75718656}, {u'TargetSize': 137338433, u'LogicalSize': 124872846, u'SizeRatio': 0.9092345330603853, u'NWriteBytes': 77549568, u'BloomFilterMemUsed': 2414592, u'NumTables': 61, u'WastedSpace': 333682, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 125206528}], u'NItemsGarbageCollected': 720648, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 2582223, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 68352, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 679082, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 374713, u'NFailedCompacts': 49, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 217863, u'CompactWriter': {u'NDataBlockWrites': 214981, u'NWriteBytes': 640819200, u'NIndexBlockWrites': 3074, u'NBytesIncoming': 805901003, u'NSyncs': 0, u'NInserts': 16659623, u'NWriteIOs': 1499}, u'NonResidentBloomFilterSize': 1345256, u'IteratorMemUsed': 0, u'NWriteBytes': 760796420, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 479120, u'NReadBytes': 1390577752, u'NDataBlockReads': 1627318, u'NBytesOutgoing': 0, u'NIndexBlockReads': 2115246}, u'NBloomFilterMisses': 125563, u'NItemsMerged': 0, u'LogicalDiskSize': 124872846, u'TotalBloomFilterMemUsed': 5430136, u'WriteAmp': 7.9403671860986815, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 5128960, u'LogicalDataSize': 69020925, u'NTableFiles': 712, u'NFileCountCompacts': 249, u'TableObjectMemUsed': 170880, u'TreeSnapshotMemoryUsed': 182544, u'TotalDiskUsage': 261324800, u'BytesIncoming': 95813758, u'NItemsDel': 0, u'PendingCheckpointBytes': 18116608, u'NCompacts': 252, u'NItemsPut': 1926974, u'NSyncs': 2191, u'CompactReader': {u'NReadIOs': 199962, u'NReadBytes': 578071940, u'NDataBlockReads': 222136, u'NBytesOutgoing': 0, u'NIndexBlockReads': 7095}, u'ActiveDiskUsage': 205172736, u'NTablesDeleted': 4633, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.01602901084625947, u'Writer': {u'NDataBlockWrites': 23442, u'NWriteBytes': 119977220, u'NIndexBlockWrites': 4314, u'NBytesIncoming': 79761472, u'NSyncs': 2191, u'NInserts': 1360961, u'NWriteIOs': 6491}}, u'NReadBytesCompactPerSec': 1087321.75, u'TableMetaMemUsed': 1090695, u'NDeletes': 0, u'BlockCacheHitsPerSec': 387.8666687011719, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 125865, u'LogicalDiskSize': 3355673467L, u'NSets': 1291656, u'TotalBloomFilterMemUsed': 5435544, u'BufferMemUsed': 1329152, u'RecentReadAmp': 0.0, u'NGetsPerSec': 126.0999984741211, u'NInserts': 635318, u'NCommitBatchesPerSec': 147.60000610351562, u'FragmentationWithOverhead': 0.48746615648269653, u'NCompactsPerSec': 0.8333333134651184, u'WALMemUsed': 147536, u'MemoryQuota': 108147512, u'NReadBytesPerSec': 2000025.625, u'TreeSnapshotMemoryUsed': 646000, u'BytesIncoming': 2142521214, u'NCompacts': 938, u'NumPersistentFlushes': 2098, u'NWriteIOsPerSec': 169.43333435058594, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 7237.93701171875, u'ActiveDiskUsage': 3460636672L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 0.9666666388511658, u'NReadBytesSetPerSec': 77400.796875, u'BlockCacheHits': 447217, u'NWriteBytesPerSec': 3014799.75, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 4565, u'ReadIOAmp': 1.3106457242582896, u'MemtablesMemUsed': 1582520, u'NBloomFilterHits': 810, u'TotalIndexBlocksSize': 30714, u'CheckpointOverhead': 2613248, u'NItemsGet': 1719, u'NWriteIOs': 6756, u'NReadBytes': 8556732, u'LevelStats': [{u'TargetSize': 130023424, u'LogicalSize': 2304193, u'SizeRatio': 0.0177213684205086, u'NWriteBytes': 34848768, u'BloomFilterMemUsed': 3512, u'NumTables': 226, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 1851392}, {u'TargetSize': 130023424, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 130023424, u'LogicalSize': 329026, u'SizeRatio': 0.0025305132712087324, u'NWriteBytes': 2048000, u'BloomFilterMemUsed': 992, u'NumTables': 31, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 253952}], u'NItemsGarbageCollected': 832092, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 12617, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 60480, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 10981, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 180804, u'CompactWriter': {u'NDataBlockWrites': 256, u'NWriteBytes': 2097152, u'NIndexBlockWrites': 256, u'NBytesIncoming': 790128, u'NSyncs': 0, u'NInserts': 5888, u'NWriteIOs': 256}, u'NonResidentBloomFilterSize': 4503, u'IteratorMemUsed': 0, u'NWriteBytes': 38501066, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 2253, u'NReadBytes': 765635, u'NDataBlockReads': 1256, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1256}, u'NBloomFilterMisses': 302, u'NItemsMerged': 0, u'LogicalDiskSize': 329026, u'TotalBloomFilterMemUsed': 5408, u'WriteAmp': 0.10338887617569131, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 4504, u'LogicalDataSize': 329026, u'NTableFiles': 630, u'NFileCountCompacts': 256, u'TableObjectMemUsed': 151200, u'TreeSnapshotMemoryUsed': 130032, u'TotalDiskUsage': 5160960, u'BytesIncoming': 372390797, u'NItemsDel': 0, u'PendingCheckpointBytes': 475136, u'NCompacts': 256, u'NItemsPut': 834300, u'NSyncs': 2191, u'CompactReader': {u'NReadIOs': 8728, u'NReadBytes': 7791097, u'NDataBlockReads': 4603, u'NBytesOutgoing': 0, u'NIndexBlockReads': 4603}, u'ActiveDiskUsage': 2105344, u'NTablesDeleted': 4603, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.2715827338129496, u'Writer': {u'NDataBlockWrites': 4309, u'NWriteBytes': 36403914, u'NIndexBlockWrites': 4309, u'NBytesIncoming': 7008673, u'NSyncs': 2191, u'NInserts': 45411, u'NWriteIOs': 6500}}}}} 2022-02-08 08:01:06,707 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:728] magma stats fragmentation result {'172.23.121.123': [0.3867143392562866, 0.3800716698169708, 0.33163169026374817, 0.33604538440704346, 0.40216708183288574, 0.4188482165336609, 0.3771202564239502, 0.38117051124572754], '172.23.121.124': [0.38209959864616394, 0.3875235319137573, 0.3455490469932556, 0.3215281665325165, 0.3275505602359772, 0.34704142808914185, 0.37601861357688904, 0.4246307611465454], '172.23.121.135': [0.3736761510372162, 0.4095742106437683, 0.37357527017593384, 0.4058084785938263, 0.36906003952026367, 0.39574629068374634, 0.3902589976787567, 0.3999808430671692], '172.23.121.126': [0.3679594397544861, 0.36996161937713623, 0.4023682475090027, 0.41004449129104614, 0.39287620782852173, 0.3715973198413849, 0.34488651156425476, 0.3505438566207886], '172.23.121.115': [0.42054077982902527, 0.42233940958976746, 0.402383953332901, 0.43091702461242676, 0.37392276525497437, 0.3376690745353699, 0.3227273225784302, 0.3789064884185791], '172.23.121.116': [0.36945587396621704, 0.3804539740085602, 0.3892451226711273, 0.4166453182697296, 0.4103744626045227, 0.39357149600982666, 0.4223020374774933, 0.3670068681240082], '172.23.121.127': [0.37172093987464905, 0.30937495827674866, 0.329144150018692, 0.3648707866668701, 0.38877755403518677, 0.3989248275756836, 0.3950669765472412, 0.36975812911987305], '172.23.121.129': [0.3824237287044525, 0.35763785243034363, 0.3797720968723297, 0.36213698983192444, 0.3688015639781952, 0.33581629395484924, 0.3498271703720093, 0.3975359797477722], '172.23.120.170': [0.3681388199329376, 0.37352442741394043, 0.33676818013191223, 0.3512914776802063, 0.35924309492111206, 0.40219008922576904, 0.3704485595226288, 0.3995441496372223]} 2022-02-08 08:01:07,135 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.116 is 40.0 2022-02-08 08:01:07,549 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.123 is 40.0 2022-02-08 08:01:08,003 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.124 is 40.0 2022-02-08 08:01:08,328 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.126 is 40.0 2022-02-08 08:01:08,670 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.127 is 40.0 2022-02-08 08:01:09,091 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.129 is 40.0 2022-02-08 08:01:09,374 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.135 is 40.0 2022-02-08 08:01:09,805 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.115 is 40.0 2022-02-08 08:01:10,138 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.120.170 is 40.0 2022-02-08 08:01:10,140 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:677] KV stats fragmentation values {'172.23.121.123': 40.0, '172.23.121.124': 40.0, '172.23.121.135': 40.0, '172.23.121.126': 40.0, '172.23.121.115': 40.0, '172.23.121.116': 40.0, '172.23.121.127': 40.0, '172.23.121.129': 40.0, '172.23.120.170': 40.0} ############################################################ # # RAM FootPrint: 4470.06640625 # ############################################################ 2022-02-08 08:01:10,200 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 08:01:10,200 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 112 seconds. Reason: Iteration:0 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 10450 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 10476 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:03:02,207 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 08:03:02,207 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 08:03:07,209 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:03:07,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:07,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:03:07,687 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:07,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:03:07,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:03:07,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:03:07,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:03:07,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:03:07,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:03:07,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:03:07,855 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:03:07,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:03:07,927 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['24324 memcached\n'], []) 2022-02-08 08:03:07,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 08:03:07,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:03:07,990 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['24324 memcached\n'], []) 2022-02-08 08:03:07,992 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:03:07,994 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:03:08,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:08,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:03:08,444 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:08,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:03:08,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:03:08,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:03:08,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:03:08,505 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:03:08,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:03:08,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:03:08,549 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:03:08,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:03:08,609 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['75009 memcached\n'], []) 2022-02-08 08:03:08,609 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 08:03:08,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:03:08,660 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['75009 memcached\n'], []) 2022-02-08 08:03:08,660 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:03:08,661 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:03:08,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:08,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:03:09,052 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:09,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:03:09,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:03:09,084 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:03:09,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:03:09,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:03:09,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:03:09,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:03:09,171 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:03:09,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:03:09,224 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['58494 memcached\n'], []) 2022-02-08 08:03:09,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 08:03:09,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:03:09,276 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['58494 memcached\n'], []) 2022-02-08 08:03:09,276 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:03:09,279 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:03:09,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:09,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:03:09,641 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:09,642 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:03:09,655 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:03:09,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:03:09,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:03:09,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:03:09,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:03:09,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:03:09,737 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:03:09,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:03:09,971 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['88875 memcached\n'], []) 2022-02-08 08:03:09,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 08:03:10,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:03:10,028 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['88875 memcached\n'], []) 2022-02-08 08:03:10,028 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:03:10,029 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:03:10,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:10,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:03:10,388 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:10,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:03:10,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:03:10,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:03:10,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:03:10,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:03:10,451 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:03:10,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:03:10,482 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:03:10,482 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:03:10,525 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['82714 memcached\n'], []) 2022-02-08 08:03:10,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 08:03:10,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:03:10,565 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['82714 memcached\n'], []) 2022-02-08 08:03:10,565 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:03:10,568 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:03:10,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:10,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:03:10,908 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:10,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:03:10,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:03:10,936 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:03:10,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:03:10,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:03:10,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:03:10,987 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:03:11,000 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:03:11,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:03:11,089 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['117699 memcached\n'], []) 2022-02-08 08:03:11,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 08:03:11,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:03:11,144 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['117699 memcached\n'], []) 2022-02-08 08:03:11,144 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:03:11,145 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:03:11,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:11,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:03:11,525 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:11,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:03:11,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:03:11,551 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:03:11,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:03:11,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:03:11,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:03:11,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:03:11,617 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:03:11,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:03:11,671 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['51589 memcached\n'], []) 2022-02-08 08:03:11,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 08:03:11,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:03:11,717 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['51589 memcached\n'], []) 2022-02-08 08:03:11,717 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:03:11,720 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:03:11,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:12,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:03:12,105 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:12,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:03:12,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:03:12,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:03:12,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:03:12,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:03:12,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:03:12,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:03:12,221 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:03:12,221 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:03:12,286 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['66500 memcached\n'], []) 2022-02-08 08:03:12,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 08:03:12,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:03:12,332 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['66500 memcached\n'], []) 2022-02-08 08:03:12,332 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:03:12,335 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:03:12,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:12,632 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:03:12,700 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:12,701 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:03:12,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:03:12,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:03:12,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:03:12,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:03:12,767 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:03:12,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:03:12,792 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:03:12,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:03:12,839 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['1838 memcached\n'], []) 2022-02-08 08:03:12,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 08:03:12,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:03:12,884 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['1838 memcached\n'], []) 2022-02-08 08:03:12,884 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:03:12,888 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:03:12,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:13,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:03:13,278 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:13,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:03:13,295 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:03:13,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:03:13,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:03:13,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:03:13,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:03:13,372 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:03:13,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:03:13,404 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 08:03:13,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:03:13,464 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:18,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:03:19,451 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:19,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:20,280 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:03:20,296 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 08:03:20,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:03:20,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:20,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:03:20,394 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 08:03:20,394 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:03:20,395 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:03:20,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:20,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:03:20,752 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:20,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:03:20,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:03:20,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:03:20,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:03:20,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:03:20,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:03:20,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:03:20,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:03:20,884 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 08:03:20,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:03:20,926 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:22,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:03:22,670 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:23,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:23,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:03:23,431 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 08:03:23,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:03:23,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:23,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:03:23,529 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 08:03:23,529 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:03:23,530 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 08:03:23,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:23,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 08:03:23,927 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:23,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 08:03:23,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 08:03:23,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 08:03:23,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 08:03:23,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 08:03:24,010 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 08:03:24,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 08:03:24,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:03:24,068 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 08:03:24,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:03:24,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:24,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:03:24,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:24,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:24,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:03:24,188 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 08:03:24,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:03:24,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:24,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:03:24,237 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:24,255 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:03:24,270 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:24,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:24,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:24,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:24,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:24,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:24,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:24,385 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:24,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:24,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:03:24,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:24,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:24,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:24,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:24,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:24,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:24,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:24,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:24,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:24,609 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:03:24,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:24,647 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:24,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:24,681 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:24,698 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:24,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:24,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:24,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:24,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:24,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:03:24,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:24,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:24,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:24,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:24,877 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:24,895 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:24,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:24,933 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:24,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:24,969 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:03:24,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:25,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:25,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:25,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:25,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:25,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:25,086 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:25,104 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:25,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:25,137 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:03:25,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:25,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:25,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:25,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:25,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:25,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:25,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:25,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:25,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:25,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:03:25,318 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:25,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:25,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:25,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:25,385 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:25,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:25,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:25,438 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:25,457 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 08:03:25,459 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:03:25,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:26,375 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:03:26,878 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:26,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:03:27,022 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:03:27,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:03:27,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:03:27,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:03:27,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:03:27,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:03:27,262 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:03:27,299 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 08:03:27,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:03:27,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:31,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:03:31,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:32,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:32,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:03:32,326 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 08:03:32,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:03:32,341 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:32,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:03:32,415 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 08:03:32,415 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:03:32,418 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 08:03:32,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:32,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 08:03:32,799 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:32,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 08:03:32,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 08:03:32,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 08:03:32,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 08:03:32,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 08:03:32,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 08:03:32,897 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 08:03:32,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:03:32,926 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 08:03:32,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:03:32,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:32,984 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:03:33,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:33,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:33,025 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:03:33,036 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 08:03:33,038 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:03:33,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:33,063 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:03:33,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:33,094 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:03:33,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:33,124 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:33,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:33,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:33,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:33,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:33,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:33,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:33,244 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:33,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:03:33,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:33,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:33,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:33,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:33,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:33,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:33,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:33,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:33,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:33,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:03:33,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:33,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:33,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:33,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:33,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:33,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:33,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:33,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:33,576 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:33,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:03:33,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:33,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:33,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:33,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:33,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:33,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:33,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:33,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:33,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:33,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:03:33,752 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:33,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:33,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:33,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:33,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:33,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:33,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:33,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:33,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:33,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:03:33,928 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:33,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:33,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:33,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:33,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:34,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:34,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:34,030 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:34,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:34,061 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:03:34,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:34,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:34,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:34,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:34,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:34,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:34,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:34,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:34,200 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 08:03:34,203 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:03:34,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:34,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:03:34,605 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:34,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:03:34,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:03:34,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:03:34,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:03:34,658 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:03:34,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:03:34,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:03:34,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:03:34,711 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 08:03:34,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:03:34,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:39,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:03:40,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:41,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:41,505 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:03:41,519 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 08:03:41,520 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:03:41,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:41,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:03:41,608 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 08:03:41,608 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:03:41,609 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:03:41,701 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:41,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:03:42,069 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:42,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:03:42,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:03:42,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:03:42,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:03:42,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:03:42,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:03:42,183 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:03:42,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:03:42,230 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 08:03:42,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:03:42,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:47,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:03:48,357 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:48,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:49,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:03:49,145 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 08:03:49,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:03:49,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:49,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:03:49,247 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 08:03:49,249 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:03:49,250 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 08:03:49,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:49,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 5 queries submitted, 0 failed, 10498 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:03:49,632 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:49,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 08:03:49,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 08:03:49,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 08:03:49,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 08:03:49,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 08:03:49,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 08:03:49,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 08:03:49,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:03:49,753 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 08:03:49,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:03:49,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:49,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:03:49,829 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:49,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:49,859 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:03:49,875 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 08:03:49,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:03:49,897 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:49,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:03:49,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:49,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:03:49,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:49,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:50,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:50,025 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:50,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:50,058 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:50,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:50,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:50,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:50,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:03:50,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:50,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:50,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:50,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:50,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:50,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:50,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:50,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:50,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:50,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:03:50,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:50,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:50,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:50,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:50,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:50,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:50,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:50,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:50,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:50,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:03:50,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:50,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:50,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:50,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:50,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:50,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:50,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:50,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:50,655 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:50,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:03:50,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:50,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:50,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:50,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:50,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:50,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:50,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:50,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:50,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:50,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:03:50,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:50,864 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:50,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:50,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:50,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:50,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:50,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:50,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:50,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:50,994 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:03:51,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:51,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:51,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:51,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:51,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:51,096 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:51,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:51,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:51,148 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 08:03:51,150 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 08:03:51,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:51,438 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 08:03:51,496 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:51,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 08:03:51,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 08:03:51,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 08:03:51,533 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 08:03:51,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 08:03:51,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 08:03:51,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 08:03:51,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:03:51,601 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 08:03:51,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:03:51,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:51,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:03:51,681 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:51,693 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:51,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:03:51,717 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 08:03:51,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:03:51,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:51,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:03:51,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:51,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:03:51,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:51,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:51,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:51,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:51,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:51,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:51,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:51,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:03:51,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:51,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:03:51,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:51,971 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:51,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:52,002 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:52,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:52,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:52,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:52,066 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:03:52,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:52,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:03:52,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:52,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:52,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:52,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:52,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:52,196 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:52,211 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:52,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:03:52,242 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:52,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:03:52,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:52,290 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:52,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:52,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:52,334 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:52,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:52,364 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:52,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:03:52,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:52,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:03:52,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:52,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:52,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:52,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:52,482 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:52,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:52,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:52,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:03:52,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:52,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:03:52,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:52,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:52,609 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:52,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:52,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:52,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:52,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:52,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:03:52,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:52,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:03:52,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:52,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:52,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:52,778 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:52,793 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:52,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:52,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:52,839 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:03:52,855 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 08:03:52,857 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:03:52,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:03:53,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:03:53,263 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:03:53,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:03:53,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:03:53,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:03:53,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:03:53,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:03:53,341 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:03:53,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:03:53,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:03:53,385 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 08:03:53,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:03:53,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:58,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:03:59,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:59,538 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:03:59,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:03:59,900 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 08:03:59,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:03:59,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:03:59,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:03:59,984 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 08:03:59,986 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:03:59,987 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:04:00,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:04:00,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:04:00,380 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:04:00,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:04:00,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:04:00,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:04:00,426 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:04:00,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:04:00,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:04:00,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:04:00,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:04:00,500 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 08:04:00,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:04:00,543 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:04:04,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:04:05,436 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:04:05,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:04:06,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:04:06,036 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 08:04:06,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:04:06,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:04:06,096 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:04:06,111 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 08:04:06,112 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:04:06,117 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:04:06,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:04:07,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:04:07,102 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:04:07,104 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:04:07,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:04:07,137 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:04:07,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:04:07,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:04:07,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:04:07,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:04:07,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:04:07,345 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 08:04:07,410 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:04:07,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:04:11,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:04:11,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:04:12,290 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:04:12,556 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:04:12,572 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 08:04:12,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:04:12,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:04:12,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:04:12,684 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 08:04:12,684 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:04:12,687 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:04:12,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:04:12,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:04:13,061 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:04:13,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:04:13,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:04:13,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:04:13,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:04:13,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:04:13,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:04:13,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:04:13,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:04:13,214 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 08:04:13,242 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:04:13,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:04:19,390 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:04:20,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:04:20,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:04:21,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:04:21,382 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 08:04:21,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:04:21,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:04:21,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:04:21,477 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 08:04:21,479 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:04:21,480 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 08:04:21,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:04:21,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 08:04:21,865 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:04:21,867 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 08:04:21,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 08:04:21,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 08:04:21,920 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 08:04:21,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 08:04:21,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 08:04:21,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 08:04:21,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:04:22,015 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 08:04:22,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:04:22,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:04:22,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:04:22,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:04:22,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:04:22,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:04:22,138 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 08:04:22,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:04:22,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:04:22,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:04:22,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:04:22,206 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:04:22,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:04:22,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:04:22,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:04:22,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:04:22,295 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:04:22,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:04:22,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:04:22,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:04:22,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:04:22,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:04:22,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:04:22,426 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:04:22,443 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:04:22,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:04:22,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:04:22,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:04:22,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:04:22,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:04:22,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:04:22,566 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:04:22,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:04:22,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:04:22,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:04:22,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:04:22,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:04:22,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:04:22,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:04:22,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:04:22,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:04:22,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:04:22,762 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:04:22,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:04:22,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:04:22,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:04:22,829 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:04:22,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:04:22,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:04:22,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:04:22,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:04:22,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:04:22,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:04:22,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:04:22,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:04:22,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:04:23,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:04:23,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:04:23,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:04:23,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:04:23,066 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:04:23,084 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:04:23,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:04:23,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:04:23,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:04:23,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:04:23,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:04:23,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:04:23,196 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:04:23,211 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:04:23,227 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:04:23,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:04:23,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:04:23,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:04:23,295 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:04:23,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:04:23,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:04:23,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:04:23,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:04:23,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:04:23,424 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 08:04:23,424 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:04:23,427 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:04:23,517 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:04:23,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:04:23,815 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:04:23,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:04:23,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:04:23,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:04:23,859 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:04:23,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:04:23,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:04:23,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:04:23,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:24,181 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:04:24,183 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.758000135422. 2022-02-08 08:04:24,183 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:04:24,184 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:04:24,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:04:25,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:04:25,341 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:04:25,341 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:04:25,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:04:25,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:04:25,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:04:25,459 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:04:25,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:04:25,540 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:04:25,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:25,966 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:04:25,966 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:04:27,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:28,180 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:04:28,180 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:04:30,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:30,341 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:04:30,341 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:04:32,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:32,598 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:04:32,598 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:04:34,599 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:34,868 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:04:34,868 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:04:36,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:37,025 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:04:37,025 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:04:39,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:39,181 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:04:39,181 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:04:41,183 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:41,338 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:04:41,339 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:04:43,339 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:43,500 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:04:43,502 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:04:45,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:45,663 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:04:45,664 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:04:47,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:47,818 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:04:47,818 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 10507 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:04:49,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:49,983 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:04:49,983 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:04:51,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:52,137 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:04:52,138 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:04:54,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:54,295 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:04:54,296 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:04:56,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:56,441 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:04:56,443 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 32.2599999905. 2022-02-08 08:04:56,443 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:04:56,446 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:04:56,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:04:57,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:04:57,479 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:04:57,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:04:57,494 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:04:57,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:04:57,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:04:57,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:04:57,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:04:57,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:04:57,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:58,095 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:04:58,096 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.65399980545. 2022-02-08 08:04:58,096 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:04:58,099 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:04:58,658 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:04:58,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:04:59,006 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:04:59,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:04:59,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:04:59,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:04:59,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:04:59,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:04:59,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:04:59,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:04:59,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:04:59,392 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:04:59,392 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.29600000381. 2022-02-08 08:04:59,392 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:04:59,394 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:05:00,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:05:00,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:05:00,375 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:05:00,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:05:00,390 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:05:00,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:05:00,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:05:00,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:05:00,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:05:00,482 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:05:00,494 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:05:00,766 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:05:00,766 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.37400007248. 2022-02-08 08:05:00,766 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:05:00,769 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:05:01,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:05:01,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:05:01,782 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:05:01,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:05:01,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:05:01,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:05:01,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:05:01,859 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:05:01,874 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:05:01,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:05:01,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:05:02,174 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:05:02,176 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.40900015831. 2022-02-08 08:05:02,176 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:05:02,178 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:05:02,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:05:02,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:05:03,019 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:05:03,020 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:05:03,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:05:03,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:05:03,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:05:03,096 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:05:03,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:05:03,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:05:03,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:05:03,450 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:05:03,450 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.27499985695. 2022-02-08 08:05:03,451 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:05:03,453 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:05:03,895 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:05:04,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:05:04,224 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:05:04,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:05:04,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:05:04,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:05:04,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:05:04,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:05:04,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:05:04,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:05:04,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:05:04,602 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:05:04,604 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.15200018883. 2022-02-08 08:05:04,604 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:05:04,605 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:05:05,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:05:05,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:05:06,147 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:05:06,147 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:05:06,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:05:06,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:05:06,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:05:06,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:05:06,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:05:06,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:05:06,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:05:07,585 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:05:07,586 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:05:09,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:05:10,115 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:05:10,115 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:05:12,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:05:12,266 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:05:12,267 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.66399979591. ############################################################ # # RAM FootPrint: 4534.125 # ############################################################ 2022-02-08 08:05:12,308 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 08:05:12,309 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 109 seconds. Reason: Iteration:1 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 10528 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 10549 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:07:01,315 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 08:07:01,316 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 08:07:06,323 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:07:06,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:06,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:07:06,724 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:06,724 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:07:06,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:07:06,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:07:06,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:07:06,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:07:06,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:07:06,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:07:06,835 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:07:06,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:07:06,890 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['28033 memcached\n'], []) 2022-02-08 08:07:06,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 08:07:06,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:07:06,940 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['28033 memcached\n'], []) 2022-02-08 08:07:06,940 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:07:06,944 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:07:07,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:07,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:07:07,312 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:07,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:07:07,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:07:07,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:07:07,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:07:07,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:07:07,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:07:07,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:07:07,417 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:07:07,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:07:07,479 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['79098 memcached\n'], []) 2022-02-08 08:07:07,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 08:07:07,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:07:07,525 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['79098 memcached\n'], []) 2022-02-08 08:07:07,525 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:07:07,528 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:07:07,599 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:07,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:07:07,888 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:07,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:07:07,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:07:07,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:07:07,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:07:07,956 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:07:07,971 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:07:07,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:07:08,005 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:07:08,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:07:08,048 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['62409 memcached\n'], []) 2022-02-08 08:07:08,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 08:07:08,073 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:07:08,098 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['62409 memcached\n'], []) 2022-02-08 08:07:08,098 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:07:08,099 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:07:08,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:08,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:07:08,463 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:08,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:07:08,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:07:08,489 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:07:08,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:07:08,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:07:08,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:07:08,543 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:07:08,556 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:07:08,556 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:07:08,602 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['92586 memcached\n'], []) 2022-02-08 08:07:08,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 08:07:08,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:07:08,660 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['92586 memcached\n'], []) 2022-02-08 08:07:08,661 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:07:08,663 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:07:08,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:08,961 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:07:09,045 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:09,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:07:09,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:07:09,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:07:09,084 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:07:09,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:07:09,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:07:09,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:07:09,138 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:07:09,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:07:09,186 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['86429 memcached\n'], []) 2022-02-08 08:07:09,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 08:07:09,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:07:09,230 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['86429 memcached\n'], []) 2022-02-08 08:07:09,230 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:07:09,233 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:07:09,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:09,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:07:09,585 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:09,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:07:09,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:07:09,614 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:07:09,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:07:09,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:07:09,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:07:09,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:07:09,680 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:07:09,681 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:07:09,726 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['121402 memcached\n'], []) 2022-02-08 08:07:09,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 08:07:09,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:07:09,780 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['121402 memcached\n'], []) 2022-02-08 08:07:09,782 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:07:09,783 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:07:09,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:10,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:07:10,233 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:10,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:07:10,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:07:10,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:07:10,270 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:07:10,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:07:10,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:07:10,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:07:10,326 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:07:10,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:07:10,380 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['55329 memcached\n'], []) 2022-02-08 08:07:10,380 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 08:07:10,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:07:10,424 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['55329 memcached\n'], []) 2022-02-08 08:07:10,424 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:07:10,427 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:07:10,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:10,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:07:10,795 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:10,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:07:10,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:07:10,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:07:10,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:07:10,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:07:10,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:07:10,885 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:07:10,901 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:07:10,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:07:10,966 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['70245 memcached\n'], []) 2022-02-08 08:07:10,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 08:07:10,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:07:11,016 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['70245 memcached\n'], []) 2022-02-08 08:07:11,016 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:07:11,019 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:07:11,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:11,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:07:11,921 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:11,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:07:11,936 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:07:12,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:07:12,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:07:12,035 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:07:12,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:07:12,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:07:12,099 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:07:12,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:07:12,160 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['5584 memcached\n'], []) 2022-02-08 08:07:12,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 08:07:12,183 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:07:12,203 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['5584 memcached\n'], []) 2022-02-08 08:07:12,203 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:07:12,204 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:07:12,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:12,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:07:12,563 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:12,563 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:07:12,579 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:07:12,594 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:07:12,609 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:07:12,630 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:07:12,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:07:12,660 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:07:12,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:07:12,700 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 08:07:12,724 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:07:12,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:13,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:07:14,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:14,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:14,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:07:14,928 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 08:07:14,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:07:14,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:14,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:07:15,010 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 08:07:15,010 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:07:15,013 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:07:15,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:15,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:07:15,408 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:15,410 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:07:15,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:07:15,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:07:15,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:07:15,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:07:15,487 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:07:15,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:07:15,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:07:15,542 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 08:07:15,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:07:15,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:19,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:07:20,081 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:20,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:20,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:07:20,911 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 08:07:20,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:07:20,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:21,002 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:07:21,017 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 08:07:21,017 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:07:21,019 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 08:07:21,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:21,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 08:07:21,361 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:21,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 08:07:21,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 08:07:21,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 08:07:21,395 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 08:07:21,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 08:07:21,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 08:07:21,433 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 08:07:21,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:07:21,466 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 08:07:21,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:07:21,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:21,520 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:07:21,533 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:21,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:21,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:07:21,571 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 08:07:21,571 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:07:21,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:21,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:07:21,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:21,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:07:21,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:21,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:21,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:21,681 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:21,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:21,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:21,724 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:21,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:21,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:21,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:07:21,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:21,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:21,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:21,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:21,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:21,867 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:21,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:21,897 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:21,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:21,926 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:07:21,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:21,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:21,969 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:21,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:21,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:22,010 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:22,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:22,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:22,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:22,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:07:22,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:22,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:22,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:22,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:22,137 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:22,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:22,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:22,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:22,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:22,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:07:22,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:22,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:22,244 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:22,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:22,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:22,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:22,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:22,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:22,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:22,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:07:22,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:22,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:22,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:22,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:22,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:22,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:22,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:22,443 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:22,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:22,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:07:22,494 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:22,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:22,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:22,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:22,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:22,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:22,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:22,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:22,609 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 08:07:22,611 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:07:22,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:22,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:07:22,947 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:22,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:07:22,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:07:22,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:07:22,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:07:23,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:07:23,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:07:23,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:07:23,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:07:23,055 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 08:07:23,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:07:23,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:23,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:07:23,971 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:24,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:24,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:07:24,473 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 08:07:24,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:07:24,487 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:24,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:07:24,559 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 08:07:24,559 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:07:24,562 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 08:07:24,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:24,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 08:07:24,936 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:24,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 08:07:24,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 08:07:24,961 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 08:07:24,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 08:07:24,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 08:07:25,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 08:07:25,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 08:07:25,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:07:25,036 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 08:07:25,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:07:25,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:25,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:07:25,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:25,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:25,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:07:25,141 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 08:07:25,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:07:25,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:25,168 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:07:25,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:25,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:07:25,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:25,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:25,244 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:25,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:25,275 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:25,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:25,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:25,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:25,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:25,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:07:25,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:25,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:25,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:25,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:25,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:25,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:25,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:25,471 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:25,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:25,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:07:25,517 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:25,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:25,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:25,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:25,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:25,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:25,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:25,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:25,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:25,647 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:07:25,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:25,676 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:25,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:25,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:25,719 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:25,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:25,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:25,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:25,778 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:25,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:07:25,806 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:25,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:25,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:25,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:25,874 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:25,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:25,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:25,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:25,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:25,987 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:07:26,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:26,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:26,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:26,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:26,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:26,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:26,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:26,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:26,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:26,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:07:26,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:26,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:26,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:26,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:26,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:26,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:26,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:26,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:26,272 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 08:07:26,275 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:07:26,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:26,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:07:26,638 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:26,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:07:26,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:07:26,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:07:26,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:07:26,698 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:07:26,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:07:26,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:07:26,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:07:26,766 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 08:07:26,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:07:26,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:27,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:07:28,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:28,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:29,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:07:29,207 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 08:07:29,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:07:29,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:29,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:07:29,288 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 08:07:29,288 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:07:29,289 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:07:29,375 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:29,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:07:29,664 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:29,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:07:29,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:07:29,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:07:29,716 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:07:29,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:07:29,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:07:29,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:07:29,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:07:29,816 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 08:07:29,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:07:29,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:30,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:07:31,216 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:31,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:31,994 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:07:32,013 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 08:07:32,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:07:32,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:32,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:07:32,130 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 08:07:32,131 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:07:32,134 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 08:07:32,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:32,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 08:07:32,546 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:32,548 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 08:07:32,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 08:07:32,576 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 08:07:32,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 08:07:32,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 08:07:32,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 08:07:32,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 08:07:32,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:07:32,674 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 08:07:32,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:07:32,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:32,739 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:07:32,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:32,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:32,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:07:32,894 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 08:07:32,895 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:07:32,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:32,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:07:32,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:32,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:07:33,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:33,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:33,038 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:33,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:33,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:33,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:33,117 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:33,135 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:33,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:33,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:07:33,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:33,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:33,244 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:33,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:33,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:33,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:33,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:33,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:33,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:33,392 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:07:33,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:33,433 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:33,451 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:33,471 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:33,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:33,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:33,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:33,548 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:33,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:33,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:07:33,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:33,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:33,647 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:33,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:33,688 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:33,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:33,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:33,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:33,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:33,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:07:33,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:33,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:33,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:33,874 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:33,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:33,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:33,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:33,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:33,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:33,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:07:34,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:34,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:34,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:34,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:34,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:34,107 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:34,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:34,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:34,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:34,183 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:07:34,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:34,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:34,244 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:34,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:34,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:34,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:34,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:34,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:34,368 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 08:07:34,369 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 08:07:34,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:34,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 08:07:34,707 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:34,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 08:07:34,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 08:07:34,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 08:07:34,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 08:07:34,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 08:07:34,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 08:07:34,790 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 08:07:34,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:07:34,823 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 08:07:34,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:07:34,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:34,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:07:34,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:34,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:34,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:07:34,940 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 08:07:34,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:07:34,956 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:34,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:07:34,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:35,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:07:35,020 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:35,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:35,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:35,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:35,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:35,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:35,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:35,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:35,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:35,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:07:35,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:35,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:35,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:35,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:35,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:35,262 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:35,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:35,290 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:35,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:35,321 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:07:35,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:35,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:35,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:35,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:35,392 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:35,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:35,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:35,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:35,451 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:35,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:07:35,482 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:35,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:35,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:35,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:35,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:35,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:35,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:35,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:35,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:35,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:07:35,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:35,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:35,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:35,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:35,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:35,706 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:35,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:35,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:35,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:35,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:07:35,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:35,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:35,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:35,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:35,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:35,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:35,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:35,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:35,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:35,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:07:35,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:35,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:35,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:35,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:35,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:36,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:36,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:36,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:36,058 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 08:07:36,059 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:07:36,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:36,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:07:36,614 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:36,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:07:36,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:07:36,655 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:07:36,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:07:36,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:07:36,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:07:36,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:07:36,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:07:36,803 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 08:07:36,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:07:36,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:38,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:07:38,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:39,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:39,571 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:07:39,588 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 08:07:39,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:07:39,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:39,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:07:39,678 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 08:07:39,680 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:07:39,681 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:07:39,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:40,066 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:07:40,155 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:40,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:07:40,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:07:40,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:07:40,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:07:40,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:07:40,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:07:40,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:07:40,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:07:40,302 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 08:07:40,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:07:40,357 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:41,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:07:41,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:42,022 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:42,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:07:42,326 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 08:07:42,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:07:42,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:42,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:07:42,405 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 08:07:42,405 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:07:42,408 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:07:42,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:42,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:07:42,773 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:42,775 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:07:42,788 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:07:42,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:07:42,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:07:42,829 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:07:42,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:07:42,859 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:07:42,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:07:42,894 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 08:07:42,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:07:42,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:43,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:07:43,877 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:44,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:44,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:07:44,415 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 08:07:44,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:07:44,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:44,443 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:07:44,456 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 08:07:44,456 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:07:44,459 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:07:44,533 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:44,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:07:44,818 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:44,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:07:44,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:07:44,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:07:44,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:07:44,895 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:07:44,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:07:44,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:07:44,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:07:44,957 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 08:07:44,984 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:07:45,005 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:45,632 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:07:46,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:46,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:47,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:07:47,190 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 08:07:47,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:07:47,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:47,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:07:47,266 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 08:07:47,266 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:07:47,269 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 08:07:47,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:47,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 08:07:47,630 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:47,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 08:07:47,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 08:07:47,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 08:07:47,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 08:07:47,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 08:07:47,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 08:07:47,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 08:07:47,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:07:47,733 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 08:07:47,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:07:47,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:47,788 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:07:47,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:47,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:07:47,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:07:47,845 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 08:07:47,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:07:47,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:07:47,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:07:47,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:47,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:07:47,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:47,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:47,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:47,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:48,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:48,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:48,035 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:48,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:07:48,063 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:48,081 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:07:48,096 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:48,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:48,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:48,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:48,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:48,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:48,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:48,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:07:48,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:48,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:07:48,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:48,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:48,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:48,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:48,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:48,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:48,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:48,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:07:48,367 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:48,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:07:48,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:48,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:48,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:48,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:48,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:48,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:48,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:48,499 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:07:48,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:48,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:07:48,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:48,556 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:48,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:48,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:48,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:48,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:48,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:48,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:07:48,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:48,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:07:48,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:48,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:48,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:48,724 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:48,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:48,752 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:48,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:48,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:07:48,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:48,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:07:48,829 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:48,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:48,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:48,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:48,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:48,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:48,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:48,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:07:48,950 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 08:07:48,951 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:07:48,953 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:07:49,022 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:07:49,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:07:49,283 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:07:49,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:07:49,298 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:07:49,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:07:49,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:07:49,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:07:49,349 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:07:49,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:07:49,375 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:07:49,507 | test | WARNING | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 08:07:49,513 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:07:49,513 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 5 queries submitted, 0 failed, 10569 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:07:51,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:07:51,627 | test | WARNING | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 08:07:51,632 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:07:51,632 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:07:53,632 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:07:53,755 | test | WARNING | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 08:07:53,760 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:07:53,760 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:07:55,762 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:07:55,872 | test | WARNING | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 08:07:55,878 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:07:55,878 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:07:57,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:07:57,984 | test | WARNING | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 08:07:57,990 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:07:57,990 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:07:59,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:00,115 | test | WARNING | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 08:08:00,121 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:00,121 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:02,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:02,226 | test | WARNING | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 08:08:02,232 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:02,232 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:04,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:04,342 | test | WARNING | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 08:08:04,348 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:04,348 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:06,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:06,464 | test | WARNING | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 08:08:06,470 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:06,470 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:08,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:08,582 | test | WARNING | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 08:08:08,586 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:08,586 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:10,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:10,701 | test | WARNING | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 08:08:10,709 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:10,709 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:12,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:12,825 | test | WARNING | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 08:08:12,832 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:12,832 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:14,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:14,947 | test | WARNING | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 08:08:14,953 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:14,953 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:16,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:17,063 | test | WARNING | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 08:08:17,069 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:17,069 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:19,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:19,206 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:19,206 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:21,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:21,346 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:21,348 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:23,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:23,551 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:23,551 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:25,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:25,986 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:25,986 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:27,987 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:28,141 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:28,141 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:30,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:30,292 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:30,292 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:32,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:32,434 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:32,434 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:34,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:34,575 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:34,575 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:36,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:36,727 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:36,729 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:38,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:39,032 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:39,032 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:41,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:41,345 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:41,345 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:43,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:43,487 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:43,489 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:45,489 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:45,628 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:08:45,628 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 08:08:47,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:47,770 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:08:47,772 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 58.8200001717. 2022-02-08 08:08:47,772 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:08:47,773 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:08:47,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:08:48,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:08:48,190 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:08:48,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:08:48,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:08:48,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:08:48,234 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:08:48,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:08:48,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:08:48,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:08:48,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:48,881 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:08:48,881 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.10899996758. 2022-02-08 08:08:48,881 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:08:48,884 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 5 queries submitted, 0 failed, 10571 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:08:50,086 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:08:50,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:08:50,565 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:08:50,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:08:50,581 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:08:50,609 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:08:50,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:08:50,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:08:50,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:08:50,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:08:50,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:51,059 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:08:51,059 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.1779999733. 2022-02-08 08:08:51,059 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:08:51,062 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:08:51,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:08:51,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:08:51,930 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:08:51,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:08:51,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:08:51,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:08:51,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:08:51,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:08:52,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:08:52,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:08:52,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:52,332 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:08:52,332 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.27200007439. 2022-02-08 08:08:52,332 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:08:52,335 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:08:52,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:08:53,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:08:53,358 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:08:53,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:08:53,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:08:53,405 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:08:53,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:08:53,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:08:53,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:08:53,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:08:53,505 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:54,029 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:08:54,030 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.69899988174. 2022-02-08 08:08:54,030 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:08:54,032 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:08:54,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:08:54,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:08:55,049 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:08:55,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:08:55,063 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:08:55,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:08:55,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:08:55,117 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:08:55,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:08:55,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:08:55,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:55,608 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:08:55,608 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.57800006866. 2022-02-08 08:08:55,608 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:08:55,611 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:08:56,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:08:56,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:08:56,645 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:08:56,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:08:56,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:08:56,693 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:08:56,719 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:08:56,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:08:56,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:08:56,790 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:08:56,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:57,299 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:08:57,301 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.69200015068. 2022-02-08 08:08:57,301 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:08:57,302 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:08:57,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:08:58,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:08:58,292 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:08:58,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:08:58,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:08:58,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:08:58,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:08:58,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:08:58,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:08:58,505 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:08:58,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:08:58,875 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:08:58,875 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.57399988174. 2022-02-08 08:08:58,875 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:08:58,878 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:08:59,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:08:59,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:08:59,681 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:08:59,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:08:59,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:08:59,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:08:59,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:08:59,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:08:59,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:08:59,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:08:59,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:09:00,309 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:09:00,309 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.43400001526. ############################################################ # # RAM FootPrint: 4596.0625 # ############################################################ 2022-02-08 08:09:00,345 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 08:09:00,345 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 95 seconds. Reason: Iteration:2 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 10593 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:10:35,351 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 08:10:35,352 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 08:10:40,358 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:10:40,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:10:41,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:10:41,332 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:10:41,334 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:10:41,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:10:41,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:10:41,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:10:41,494 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:10:41,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:10:41,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:10:41,607 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:10:41,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:10:41,688 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['28761 memcached\n'], []) 2022-02-08 08:10:41,688 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 08:10:41,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:10:41,802 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['28761 memcached\n'], []) 2022-02-08 08:10:41,802 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:10:41,805 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:10:41,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:10:42,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:10:42,273 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:10:42,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:10:42,290 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:10:42,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:10:42,318 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:10:42,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:10:42,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:10:42,367 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:10:42,380 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:10:42,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:10:42,447 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['79995 memcached\n'], []) 2022-02-08 08:10:42,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 08:10:42,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:10:42,496 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['79995 memcached\n'], []) 2022-02-08 08:10:42,496 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:10:42,499 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:10:42,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:10:42,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:10:42,890 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:10:42,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:10:42,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:10:42,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:10:42,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:10:42,956 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:10:42,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:10:42,984 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:10:43,002 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:10:43,002 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:10:43,046 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['63233 memcached\n'], []) 2022-02-08 08:10:43,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 08:10:43,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:10:43,094 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['63233 memcached\n'], []) 2022-02-08 08:10:43,094 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:10:43,095 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:10:43,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:10:43,665 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:10:43,726 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:10:43,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:10:43,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:10:43,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:10:43,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:10:43,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:10:43,793 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:10:43,806 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:10:43,821 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:10:43,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:10:43,900 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['93311 memcached\n'], []) 2022-02-08 08:10:43,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 08:10:43,987 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:10:44,015 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['93311 memcached\n'], []) 2022-02-08 08:10:44,016 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:10:44,017 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:10:44,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:10:44,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:10:44,387 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:10:44,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:10:44,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:10:44,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:10:44,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:10:44,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:10:44,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:10:44,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:10:44,486 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:10:44,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:10:44,539 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['87154 memcached\n'], []) 2022-02-08 08:10:44,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 08:10:44,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:10:44,586 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['87154 memcached\n'], []) 2022-02-08 08:10:44,586 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:10:44,588 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:10:44,660 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:10:44,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:10:44,924 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:10:44,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:10:44,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:10:44,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:10:44,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:10:44,984 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:10:44,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:10:45,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:10:45,023 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:10:45,025 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:10:45,101 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['122129 memcached\n'], []) 2022-02-08 08:10:45,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 08:10:45,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:10:45,151 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['122129 memcached\n'], []) 2022-02-08 08:10:45,151 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:10:45,154 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:10:45,242 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:10:45,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:10:45,529 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:10:45,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:10:45,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:10:45,556 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:10:45,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:10:45,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:10:45,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:10:45,630 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:10:45,644 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:10:45,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:10:45,697 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['56057 memcached\n'], []) 2022-02-08 08:10:45,698 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 08:10:45,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:10:45,760 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['56057 memcached\n'], []) 2022-02-08 08:10:45,762 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:10:45,763 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:10:45,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:10:46,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:10:46,145 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:10:46,147 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:10:46,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:10:46,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:10:46,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:10:46,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:10:46,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:10:46,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:10:46,243 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:10:46,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:10:46,311 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['70973 memcached\n'], []) 2022-02-08 08:10:46,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 08:10:46,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:10:46,364 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['70973 memcached\n'], []) 2022-02-08 08:10:46,364 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:10:46,368 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:10:46,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:10:46,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:10:46,763 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:10:46,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:10:46,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:10:46,793 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:10:46,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:10:46,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:10:46,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:10:46,864 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:10:46,880 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:10:46,880 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:10:46,927 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['6335 memcached\n'], []) 2022-02-08 08:10:46,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 08:10:46,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:10:46,973 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['6335 memcached\n'], []) 2022-02-08 08:10:46,973 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:10:46,976 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:10:47,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:10:47,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:10:47,391 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:10:47,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:10:47,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:10:47,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:10:47,433 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:10:47,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:10:47,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:10:47,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:10:47,489 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:10:47,506 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 08:10:47,551 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:10:47,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 10619 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:10:49,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:10:50,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:10:51,058 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:10:51,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:10:51,463 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 08:10:51,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:10:51,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:10:51,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:10:51,559 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 08:10:51,559 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:10:51,569 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:10:51,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:10:51,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:10:51,950 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:10:51,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:10:51,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:10:51,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:10:51,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:10:52,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:10:52,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:10:52,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:10:52,058 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:10:52,082 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 08:10:52,114 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:10:52,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:10:53,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:10:54,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:10:54,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:10:54,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:10:54,836 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 08:10:54,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:10:54,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:10:54,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:10:54,948 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 08:10:54,950 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:10:54,951 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 08:10:55,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:10:55,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 08:10:55,309 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:10:55,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 08:10:55,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 08:10:55,334 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 08:10:55,346 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 08:10:55,362 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 08:10:55,375 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 08:10:55,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 08:10:55,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:10:55,417 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 08:10:55,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:10:55,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:10:55,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:10:55,492 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:10:55,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:10:55,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:10:55,529 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 08:10:55,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:10:55,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:10:55,556 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:10:55,571 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:55,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:10:55,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:55,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:55,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:55,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:55,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:55,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:55,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:55,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:55,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:55,757 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:10:55,775 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:55,790 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:55,806 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:55,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:55,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:55,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:55,867 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:55,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:55,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:10:55,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:10:55,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:10:55,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:10:55,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:10:55,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:10:55,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:10:56,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:10:56,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:10:56,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:10:56,058 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:10:56,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:10:56,086 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:10:56,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:10:56,117 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:10:56,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:10:56,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:10:56,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:10:56,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:10:56,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:10:56,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:10:56,216 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:10:56,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:10:56,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:10:56,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:10:56,272 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:10:56,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:10:56,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:10:56,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:10:56,331 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:10:56,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:10:56,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:10:56,372 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:10:56,385 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:10:56,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:10:56,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:10:56,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:10:56,443 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:10:56,459 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:10:56,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:10:56,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:10:56,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:10:56,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:10:56,548 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:10:56,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:10:56,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:10:56,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:10:56,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:10:56,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:10:56,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:10:56,696 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 08:10:56,698 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:10:56,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:10:57,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:10:57,102 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:10:57,104 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:10:57,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:10:57,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:10:57,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:10:57,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:10:57,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:10:57,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:10:57,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:10:57,223 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 08:10:57,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:10:57,275 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:10:57,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:10:58,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:10:58,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:10:58,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:10:58,694 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 08:10:58,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:10:58,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:10:58,762 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:10:58,778 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 08:10:58,779 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:10:58,780 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 08:10:58,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:10:59,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 08:10:59,161 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:10:59,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 08:10:59,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 08:10:59,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 08:10:59,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 08:10:59,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 08:10:59,234 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 08:10:59,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 08:10:59,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:10:59,279 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 08:10:59,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:10:59,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:10:59,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:10:59,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:10:59,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:10:59,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:10:59,397 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 08:10:59,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:10:59,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:10:59,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:10:59,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:59,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:10:59,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:59,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:59,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:59,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:59,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:59,594 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:59,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:59,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:10:59,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:59,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:10:59,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:59,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:59,732 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:59,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:59,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:59,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:59,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:59,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:10:59,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:10:59,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:10:59,877 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:10:59,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:10:59,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:10:59,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:10:59,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:10:59,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:10:59,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:10:59,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:00,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:00,030 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:11:00,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:00,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:00,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:00,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:00,127 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:00,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:00,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:00,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:00,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:00,206 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:11:00,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:00,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:00,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:00,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:00,278 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:00,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:00,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:00,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:00,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:00,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:11:00,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:00,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:00,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:00,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:00,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:00,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:00,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:00,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:00,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:00,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:11:00,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:00,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:00,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:00,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:00,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:00,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:00,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:00,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:00,644 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 08:11:00,648 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:11:00,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:11:00,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:11:00,993 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:11:00,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:11:01,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:11:01,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:11:01,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:11:01,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:11:01,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:11:01,084 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:11:01,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:11:01,112 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 08:11:01,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:11:01,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:02,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:11:03,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:03,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:04,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:11:04,111 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 08:11:04,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:11:04,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:04,168 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:11:04,180 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 08:11:04,181 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:11:04,184 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:11:04,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:11:04,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:11:04,559 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:11:04,561 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:11:04,576 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:11:04,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:11:04,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:11:04,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:11:04,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:11:04,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:11:04,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:11:04,694 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 08:11:04,729 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:11:04,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:05,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:11:05,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:06,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:06,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:11:06,736 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 08:11:06,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:11:06,762 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:06,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:11:06,844 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 08:11:06,844 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:11:06,845 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 08:11:06,933 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:11:07,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 08:11:07,220 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:11:07,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 08:11:07,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 08:11:07,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 08:11:07,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 08:11:07,275 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 08:11:07,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 08:11:07,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 08:11:07,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:11:07,328 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 08:11:07,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:11:07,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:07,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:11:07,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:07,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:07,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:11:07,444 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 08:11:07,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:11:07,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:07,471 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:11:07,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:07,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:11:07,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:07,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:07,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:07,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:07,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:07,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:07,655 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:07,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:07,701 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:07,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:11:07,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:07,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:07,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:07,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:07,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:07,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:07,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:07,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:07,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:07,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:11:07,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:07,959 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:07,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:07,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:08,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:08,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:08,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:08,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:08,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:08,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:11:08,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:08,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:08,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:08,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:08,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:08,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:08,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:08,255 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:08,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:08,290 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:11:08,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:08,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:08,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:08,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:08,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:08,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:08,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:08,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:08,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:08,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:11:08,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:08,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:08,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:08,540 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:08,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:08,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:08,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:08,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:08,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:08,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:11:08,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:08,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:08,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:08,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:08,739 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:08,757 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:08,778 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:08,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:08,812 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 08:11:08,815 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 08:11:08,946 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:11:09,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 08:11:09,250 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:11:09,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 08:11:09,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 08:11:09,280 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 08:11:09,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 08:11:09,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 08:11:09,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 08:11:09,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 08:11:09,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:11:09,371 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 08:11:09,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:11:09,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:09,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:11:09,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:09,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:09,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:11:09,493 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 08:11:09,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:11:09,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:09,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:11:09,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:09,563 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:11:09,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:09,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:09,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:09,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:09,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:09,693 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:09,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:09,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:09,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:09,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:11:09,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:09,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:09,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:09,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:09,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:09,885 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:09,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:09,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:09,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:09,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:11:09,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:09,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:10,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:10,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:10,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:10,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:10,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:10,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:10,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:10,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:11:10,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:10,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:10,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:10,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:10,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:10,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:10,252 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:10,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:10,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:10,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:11:10,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:10,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:10,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:10,367 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:10,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:10,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:10,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:10,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:10,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:10,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:11:10,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:10,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:10,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:10,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:10,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:10,561 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:10,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:10,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:10,632 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:10,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:11:10,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:10,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:10,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:10,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:10,762 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:10,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:10,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:10,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:10,844 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 08:11:10,845 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:11:11,002 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:11:11,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:11:11,525 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:11:11,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:11:11,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:11:11,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:11:11,642 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:11:11,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:11:11,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:11:11,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:11:11,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:11:11,842 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 08:11:11,874 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:11:11,920 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:12,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:11:13,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:13,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:13,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:11:13,967 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 08:11:13,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:11:13,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:14,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:11:14,051 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 08:11:14,051 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:11:14,053 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:11:14,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:11:14,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:11:14,440 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:11:14,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:11:14,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:11:14,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:11:14,494 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:11:14,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:11:14,533 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:11:14,556 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:11:14,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:11:14,589 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 08:11:14,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:11:14,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:15,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:11:15,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:15,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:16,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:11:16,255 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 08:11:16,255 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:11:16,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:16,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:11:16,322 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 08:11:16,323 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:11:16,325 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:11:16,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:11:17,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:11:17,844 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:11:17,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:11:17,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:11:18,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:11:18,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:11:18,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:11:18,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:11:18,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:11:18,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:11:18,946 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 08:11:18,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:11:19,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:20,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:11:20,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:21,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:21,612 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:11:21,753 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 08:11:21,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:11:21,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:21,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:11:22,042 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 08:11:22,042 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:11:22,046 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:11:22,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:11:22,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:11:22,506 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:11:22,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:11:22,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:11:22,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:11:22,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:11:22,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:11:22,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:11:22,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:11:22,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:11:22,691 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 08:11:22,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:11:22,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:23,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:11:24,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:24,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:25,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:11:25,134 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 08:11:25,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:11:25,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:25,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:11:25,204 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 08:11:25,206 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:11:25,207 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 08:11:25,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:11:25,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 08:11:25,619 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:11:25,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 08:11:25,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 08:11:25,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 08:11:25,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 08:11:25,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 08:11:25,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 08:11:25,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 08:11:25,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:11:25,743 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 08:11:25,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:11:25,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:25,806 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:11:25,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:25,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:11:25,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:11:25,867 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 08:11:25,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:11:25,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:11:25,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:11:25,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:25,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:11:25,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:25,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:25,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:26,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:26,035 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:26,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:26,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:26,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:11:26,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:26,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:11:26,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:26,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:26,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:26,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:26,234 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:26,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:26,270 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:26,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:11:26,306 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:26,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:11:26,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:26,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:26,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:26,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:26,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:26,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:26,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:26,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:11:26,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:26,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:11:26,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:26,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:26,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:26,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:26,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:26,630 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:26,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:26,665 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:11:26,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:26,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:11:26,719 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:26,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:26,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:26,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:26,788 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:26,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:26,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:26,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:11:26,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:26,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:11:26,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:26,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:26,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:26,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:26,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:26,987 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:27,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:27,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:11:27,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:27,061 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:11:27,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:27,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:27,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:27,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:27,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:27,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:27,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:27,206 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:11:27,221 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 08:11:27,223 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:11:27,224 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:11:27,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:11:27,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:11:27,609 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:11:27,609 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:11:27,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:11:27,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:11:27,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:11:27,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:11:27,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:11:27,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:11:27,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:11:27,907 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:11:27,907 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.684000015259. 2022-02-08 08:11:27,907 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:11:27,908 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:11:28,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:11:28,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:11:28,302 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:11:28,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:11:28,318 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:11:28,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:11:28,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:11:28,364 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:11:28,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:11:28,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:11:28,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:11:28,555 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:11:28,555 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:11:30,556 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:11:30,706 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:11:30,706 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:11:32,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:11:32,858 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:11:32,858 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:11:34,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:11:35,302 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:11:35,303 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:11:37,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:11:37,540 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:11:37,540 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:11:39,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:11:39,759 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:11:39,759 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:11:41,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:11:41,977 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:11:41,979 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:11:43,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:11:44,190 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:11:44,190 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:11:46,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:11:46,427 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:11:46,427 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:11:48,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:11:48,750 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:11:48,750 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 10627 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:11:50,752 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:11:51,009 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:11:51,009 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:11:53,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:11:53,276 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:11:53,278 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:11:55,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:11:55,552 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:11:55,552 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:11:57,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:11:57,766 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:11:57,766 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:11:59,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:11:59,931 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:11:59,931 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:12:01,933 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:12:02,088 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:12:02,088 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:12:04,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:12:04,257 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:12:04,257 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:12:06,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:12:06,407 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:12:06,408 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:12:08,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:12:08,561 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:12:08,562 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:12:10,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:12:10,700 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:12:10,700 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:12:12,701 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:12:12,859 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:12:12,861 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:12:14,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:12:15,007 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:12:15,009 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:12:17,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:12:17,157 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:12:17,157 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:12:19,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:12:19,298 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:12:19,298 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:12:21,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:12:21,433 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:12:21,434 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 53.5269999504. 2022-02-08 08:12:21,434 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:12:21,436 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:12:22,030 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:12:22,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:12:22,391 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:12:22,392 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:12:22,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:12:22,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:12:22,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:12:22,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:12:22,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:12:22,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:12:22,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:12:22,967 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:12:22,967 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.53399991989. 2022-02-08 08:12:22,967 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:12:22,970 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:12:23,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:12:23,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:12:23,938 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:12:23,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:12:23,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:12:23,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:12:23,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:12:23,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:12:24,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:12:24,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:12:24,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:12:24,358 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:12:24,359 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.39199995995. 2022-02-08 08:12:24,359 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:12:24,362 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:12:25,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:12:25,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:12:25,648 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:12:25,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:12:25,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:12:25,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:12:25,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:12:25,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:12:25,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:12:25,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:12:25,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:12:26,634 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:12:26,634 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.27500009537. 2022-02-08 08:12:26,634 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:12:26,637 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:12:27,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:12:27,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:12:27,809 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:12:27,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:12:27,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:12:27,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:12:27,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:12:27,874 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:12:27,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:12:27,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:12:27,926 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:12:28,437 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:12:28,437 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.80200004578. 2022-02-08 08:12:28,437 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:12:28,438 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:12:29,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:12:29,270 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:12:29,371 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:12:29,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:12:29,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:12:29,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:12:29,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:12:29,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:12:29,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:12:29,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:12:29,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:12:29,983 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:12:29,984 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.54699993134. 2022-02-08 08:12:29,984 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:12:29,987 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:12:30,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:12:30,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:12:31,019 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:12:31,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:12:31,035 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:12:31,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:12:31,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:12:31,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:12:31,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:12:31,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:12:31,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:12:32,240 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:12:32,240 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.25500011444. 2022-02-08 08:12:32,240 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:12:32,243 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:12:33,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:12:33,543 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:12:33,671 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:12:33,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:12:33,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:12:33,701 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:12:33,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:12:33,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:12:33,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:12:33,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:12:33,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:12:34,421 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:12:34,423 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.18200016022. ############################################################ # # RAM FootPrint: 4599.15234375 # ############################################################ 2022-02-08 08:12:34,461 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 08:12:34,461 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 110 seconds. Reason: Iteration:3 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 10639 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 10664 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:14:24,476 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 08:14:24,476 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 08:14:29,480 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:14:29,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:14:29,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:14:29,875 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:14:29,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:14:29,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:14:29,900 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:14:29,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:14:29,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:14:29,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:14:29,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:14:29,970 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:14:29,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:14:30,036 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['29760 memcached\n'], []) 2022-02-08 08:14:30,038 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 08:14:30,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:14:30,098 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['29760 memcached\n'], []) 2022-02-08 08:14:30,098 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:14:30,101 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:14:30,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:14:30,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:14:30,766 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:14:30,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:14:30,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:14:30,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:14:30,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:14:30,829 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:14:30,852 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:14:30,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:14:30,885 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:14:30,885 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:14:30,948 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['80698 memcached\n'], []) 2022-02-08 08:14:30,948 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 08:14:30,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:14:31,005 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['80698 memcached\n'], []) 2022-02-08 08:14:31,006 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:14:31,009 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:14:31,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:14:31,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:14:31,414 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:14:31,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:14:31,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:14:31,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:14:31,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:14:31,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:14:31,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:14:31,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:14:31,522 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:14:31,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:14:31,566 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['64036 memcached\n'], []) 2022-02-08 08:14:31,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 08:14:31,596 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:14:31,621 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['64036 memcached\n'], []) 2022-02-08 08:14:31,622 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:14:31,625 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:14:31,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:14:31,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:14:32,006 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:14:32,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:14:32,022 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:14:32,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:14:32,073 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:14:32,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:14:32,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:14:32,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:14:32,137 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:14:32,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:14:32,184 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['93970 memcached\n'], []) 2022-02-08 08:14:32,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 08:14:32,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:14:32,230 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['93970 memcached\n'], []) 2022-02-08 08:14:32,230 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:14:32,233 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:14:32,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:14:32,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:14:32,651 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:14:32,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:14:32,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:14:32,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:14:32,706 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:14:32,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:14:32,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:14:32,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:14:32,780 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:14:32,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:14:32,826 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['87858 memcached\n'], []) 2022-02-08 08:14:32,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 08:14:32,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:14:32,872 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['87858 memcached\n'], []) 2022-02-08 08:14:32,872 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:14:32,875 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:14:32,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:14:33,168 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:14:33,253 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:14:33,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:14:33,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:14:33,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:14:33,290 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:14:33,306 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:14:33,318 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:14:33,331 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:14:33,344 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:14:33,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:14:33,395 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['122833 memcached\n'], []) 2022-02-08 08:14:33,395 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 08:14:33,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:14:33,440 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['122833 memcached\n'], []) 2022-02-08 08:14:33,440 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:14:33,444 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:14:33,517 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:14:33,747 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:14:33,821 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:14:33,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:14:33,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:14:33,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:14:33,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:14:33,877 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:14:33,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:14:33,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:14:33,937 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:14:33,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:14:33,982 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['56734 memcached\n'], []) 2022-02-08 08:14:33,982 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 08:14:34,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:14:34,032 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['56734 memcached\n'], []) 2022-02-08 08:14:34,033 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:14:34,036 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:14:34,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:14:34,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:14:34,421 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:14:34,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:14:34,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:14:34,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:14:34,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:14:34,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:14:34,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:14:34,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:14:34,526 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:14:34,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:14:34,595 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['71633 memcached\n'], []) 2022-02-08 08:14:34,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 08:14:34,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:14:34,648 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['71633 memcached\n'], []) 2022-02-08 08:14:34,648 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:14:34,651 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:14:34,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:14:35,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:14:35,088 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:14:35,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:14:35,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:14:35,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:14:35,135 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:14:35,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:14:35,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:14:35,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:14:35,196 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:14:35,196 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:14:35,266 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['7049 memcached\n'], []) 2022-02-08 08:14:35,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 08:14:35,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:14:35,322 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['7049 memcached\n'], []) 2022-02-08 08:14:35,322 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:14:35,325 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:14:35,410 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:14:35,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:14:35,756 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:14:35,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:14:35,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:14:35,788 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:14:35,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:14:35,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:14:35,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:14:35,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:14:35,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:14:35,884 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 08:14:35,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:14:35,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:14:37,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:14:38,025 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:14:38,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:14:38,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:14:38,888 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 08:14:38,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:14:38,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:14:38,969 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:14:38,984 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 08:14:38,986 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:14:38,989 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:14:39,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:14:39,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:14:39,345 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:14:39,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:14:39,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:14:39,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:14:39,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:14:39,405 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:14:39,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:14:39,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:14:39,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:14:39,463 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 08:14:39,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:14:39,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:14:40,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:14:40,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:14:41,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:14:41,647 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:14:41,660 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 08:14:41,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:14:41,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:14:41,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:14:41,756 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 08:14:41,756 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:14:41,759 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 08:14:41,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:14:42,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 08:14:42,125 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:14:42,127 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 08:14:42,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 08:14:42,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 08:14:42,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 08:14:42,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 08:14:42,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 08:14:42,211 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 08:14:42,227 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:14:42,246 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 08:14:42,275 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:14:42,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:14:42,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:14:42,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:14:42,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:14:42,357 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:14:42,371 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 08:14:42,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:14:42,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:14:42,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:14:42,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:42,436 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:14:42,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:42,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:42,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:42,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:42,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:42,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:42,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:42,596 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:42,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:42,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:14:42,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:42,670 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:42,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:42,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:42,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:42,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:42,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:42,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:42,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:42,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:14:42,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:42,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:42,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:42,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:42,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:42,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:42,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:42,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:43,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:43,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:14:43,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:43,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:43,086 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:43,104 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:43,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:43,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:43,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:43,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:43,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:14:43,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:14:43,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:14:43,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:14:43,262 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:14:43,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:14:43,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:14:43,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:14:43,331 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:14:43,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:14:43,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:43,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:14:43,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:43,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:43,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:43,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:43,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:43,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:43,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:43,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:43,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:43,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:14:43,576 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:43,596 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:43,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:43,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:43,673 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:43,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:43,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:43,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:43,744 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 08:14:43,747 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:14:43,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:14:44,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:14:44,776 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:14:44,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:14:44,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:14:44,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:14:45,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:14:45,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:14:45,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:14:45,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:14:45,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:14:45,494 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 08:14:45,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:14:45,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:14:46,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:14:47,367 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:14:47,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:14:47,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:14:47,994 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 08:14:47,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:14:48,010 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:14:48,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:14:48,089 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 08:14:48,091 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:14:48,094 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 08:14:48,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:14:48,410 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 08:14:48,476 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:14:48,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 08:14:48,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 08:14:48,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 08:14:48,517 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 08:14:48,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 08:14:48,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 08:14:48,563 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 08:14:48,579 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:14:48,599 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 08:14:48,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:14:48,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:14:48,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:14:48,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:14:48,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:14:48,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:14:48,726 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 08:14:48,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:14:48,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:14:48,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:14:48,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:48,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:14:48,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:48,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:48,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:48,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:48,897 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:48,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:48,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:48,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:14:48,984 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:49,005 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:14:49,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:49,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:49,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:49,081 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:49,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:49,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:49,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:49,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:14:49,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:49,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:14:49,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:49,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:49,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:49,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:49,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:49,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:49,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:49,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:14:49,367 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:49,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:14:49,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:49,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:49,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:49,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:49,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:49,487 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:49,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:49,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:14:49,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:14:49,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:14:49,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:14:49,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 5 queries submitted, 0 failed, 10688 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:14:49,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:14:49,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:14:49,642 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:14:49,660 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:14:49,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:14:49,693 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:14:49,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:49,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:14:49,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:49,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:49,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:49,806 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:49,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:49,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:49,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:49,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:14:49,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:49,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:14:49,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:49,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:49,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:50,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:50,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:50,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:50,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:50,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:14:50,104 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 08:14:50,108 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:14:50,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:14:50,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:14:50,493 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:14:50,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:14:50,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:14:50,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:14:50,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:14:50,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:14:50,561 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:14:50,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:14:50,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:14:50,601 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 08:14:50,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:14:50,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:14:52,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:14:52,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:14:53,227 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:14:53,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:14:53,635 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 08:14:53,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:14:53,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:14:53,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:14:53,719 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 08:14:53,720 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:14:53,723 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:14:53,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:14:54,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:14:54,095 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:14:54,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:14:54,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:14:54,127 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:14:54,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:14:54,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:14:54,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:14:54,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:14:54,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:14:54,226 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 08:14:54,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:14:54,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:14:58,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:14:59,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:15:00,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:15:00,346 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:15:00,369 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 08:15:00,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:15:00,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:15:00,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:15:00,473 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 08:15:00,473 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:15:00,476 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 08:15:00,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:15:00,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 08:15:00,865 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:15:00,867 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 08:15:00,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 08:15:00,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 08:15:00,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 08:15:00,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 08:15:00,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 08:15:00,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 08:15:00,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:15:00,980 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 08:15:01,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:15:01,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:15:01,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:15:01,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:15:01,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:15:01,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:15:01,108 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 08:15:01,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:15:01,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:15:01,137 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:15:01,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:01,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:15:01,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:01,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:01,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:01,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:01,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:01,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:01,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:01,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:01,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:01,364 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:15:01,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:01,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:01,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:01,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:01,454 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:01,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:01,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:01,507 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:01,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:01,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:15:01,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:01,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:01,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:01,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:01,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:01,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:01,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:01,686 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:01,701 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:01,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:15:01,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:01,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:01,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:01,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:01,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:01,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:01,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:01,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:01,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:01,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:15:01,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:01,920 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:01,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:01,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:01,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:01,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:02,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:02,030 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:02,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:02,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:15:02,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:02,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:02,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:02,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:02,183 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:02,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:02,211 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:02,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:02,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:02,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:15:02,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:02,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:02,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:02,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:02,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:02,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:02,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:02,392 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:02,407 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 08:15:02,410 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 08:15:02,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:15:02,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 08:15:02,808 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:15:02,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 08:15:02,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 08:15:02,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 08:15:02,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 08:15:02,867 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 08:15:02,880 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 08:15:02,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 08:15:02,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:15:02,927 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 08:15:02,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:15:02,984 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:15:03,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:15:03,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:15:03,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:15:03,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:15:03,056 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 08:15:03,058 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:15:03,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:15:03,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:15:03,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:03,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:15:03,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:03,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:03,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:03,272 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:03,295 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:03,318 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:03,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:03,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:03,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:03,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:15:03,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:03,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:03,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:03,482 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:03,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:03,517 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:03,533 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:03,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:03,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:03,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:15:03,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:03,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:03,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:03,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:03,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:03,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:03,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:03,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:03,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:03,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:15:03,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:03,790 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:03,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:03,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:03,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:03,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:03,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:03,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:03,908 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:03,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:15:03,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:03,959 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:03,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:03,994 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:04,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:04,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:04,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:04,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:04,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:04,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:15:04,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:04,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:04,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:04,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:04,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:04,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:04,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:04,239 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:04,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:04,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:15:04,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:04,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:04,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:04,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:04,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:04,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:04,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:04,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:04,437 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 08:15:04,441 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:15:04,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:15:04,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:15:04,874 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:15:04,874 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:15:04,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:15:04,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:15:04,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:15:04,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:15:04,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:15:04,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:15:05,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:15:05,028 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 08:15:05,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:15:05,073 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:15:09,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:15:10,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:15:10,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:15:10,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:15:10,828 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 08:15:10,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:15:10,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:15:10,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:15:10,885 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 08:15:10,887 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:15:10,888 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:15:10,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:15:11,227 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:15:11,302 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:15:11,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:15:11,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:15:11,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:15:11,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:15:11,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:15:11,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:15:11,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:15:11,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:15:11,454 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 08:15:11,482 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:15:11,499 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:15:13,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:15:14,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:15:14,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:15:14,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:15:14,812 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 08:15:14,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:15:14,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:15:14,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:15:14,892 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 08:15:14,892 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:15:14,895 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:15:14,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:15:15,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:15:15,322 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:15:15,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:15:15,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:15:15,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:15:15,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:15:15,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:15:15,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:15:15,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:15:15,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:15:15,502 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 08:15:15,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:15:15,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:15:19,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:15:20,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:15:20,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:15:21,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:15:21,045 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 08:15:21,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:15:21,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:15:21,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:15:21,128 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 08:15:21,128 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:15:21,131 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:15:21,224 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:15:21,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:15:21,507 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:15:21,507 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:15:21,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:15:21,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:15:21,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:15:21,566 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:15:21,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:15:21,596 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:15:21,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:15:21,637 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 08:15:21,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:15:21,688 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:15:23,897 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:15:24,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:15:25,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:15:25,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:15:25,559 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 08:15:25,561 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:15:25,579 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:15:25,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:15:25,644 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 08:15:25,644 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:15:25,647 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 08:15:25,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:15:25,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 08:15:26,046 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:15:26,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 08:15:26,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 08:15:26,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 08:15:26,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 08:15:26,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 08:15:26,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 08:15:26,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 08:15:26,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:15:26,207 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 08:15:26,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:15:26,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:15:26,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:15:26,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:15:26,397 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:15:26,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:15:26,427 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 08:15:26,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:15:26,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:15:26,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:15:26,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:26,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:15:26,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:26,543 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:26,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:26,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:26,609 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:26,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:26,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:26,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:15:26,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:26,719 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:15:26,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:26,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:26,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:26,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:26,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:26,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:26,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:26,864 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:15:26,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:26,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:15:26,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:26,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:26,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:26,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:26,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:27,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:27,058 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:27,081 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:15:27,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:27,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:15:27,137 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:27,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:27,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:27,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:27,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:27,227 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:27,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:27,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:15:27,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:27,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:15:27,318 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:27,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:27,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:27,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:27,390 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:27,405 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:27,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:27,438 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:15:27,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:27,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:15:27,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:27,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:27,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:27,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:27,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:27,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:27,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:27,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:15:27,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:27,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:15:27,660 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:27,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:27,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:27,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:27,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:27,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:27,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:27,788 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:15:27,805 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 08:15:27,805 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:15:27,808 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:15:27,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:15:28,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:15:28,293 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:15:28,295 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:15:28,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:15:28,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:15:28,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:15:28,357 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:15:28,375 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:15:28,390 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:15:28,405 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:15:29,115 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:15:29,117 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.31200003624. 2022-02-08 08:15:29,117 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:15:29,119 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:15:29,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:15:29,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:15:29,496 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:15:29,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:15:29,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:15:29,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:15:29,548 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:15:29,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:15:29,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:15:29,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:15:29,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:15:29,821 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:15:29,821 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:15:31,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:15:31,980 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:15:31,980 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:15:33,982 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:15:34,138 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:15:34,140 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:15:36,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:15:36,411 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:15:36,411 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:15:38,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:15:38,632 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:15:38,634 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:15:40,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:15:40,918 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:15:40,920 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:15:42,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:15:43,115 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:15:43,117 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:15:45,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:15:45,266 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:15:45,266 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:15:47,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:15:47,482 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:15:47,483 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:15:49,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 10689 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:15:49,634 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:15:49,634 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:15:51,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:15:51,790 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:15:51,790 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:15:53,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:15:53,944 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:15:53,944 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:15:55,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:15:56,115 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:15:56,117 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:15:58,117 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:15:58,266 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:15:58,266 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:16:00,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:00,415 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:16:00,417 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:16:02,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:02,575 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:16:02,576 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:16:04,576 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:04,740 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:16:04,740 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:16:06,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:06,908 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:16:06,910 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:16:08,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:09,078 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:16:09,079 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:16:11,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:11,259 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:16:11,259 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:16:13,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:13,417 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:16:13,417 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:16:15,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:15,584 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:16:15,585 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:16:17,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:17,790 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:16:17,790 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 48.6739997864. 2022-02-08 08:16:17,790 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:16:17,792 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:16:18,484 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:16:18,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:16:19,055 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:16:19,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:16:19,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:16:19,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:16:19,127 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:16:19,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:16:19,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:16:19,196 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:16:19,214 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:19,766 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:16:19,766 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.97600007057. 2022-02-08 08:16:19,766 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:16:19,769 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:16:20,563 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:16:20,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:16:21,052 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:16:21,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:16:21,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:16:21,132 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:16:21,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:16:21,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:16:21,295 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:16:21,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:16:21,364 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:21,776 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:16:21,776 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 08:16:23,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:23,920 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:16:23,921 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.15400004387. 2022-02-08 08:16:23,921 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:16:23,923 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:16:24,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:16:24,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:16:24,963 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:16:24,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:16:24,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:16:24,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:16:25,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:16:25,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:16:25,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:16:25,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:16:25,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:25,588 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:16:25,588 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.66799998283. 2022-02-08 08:16:25,588 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:16:25,592 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:16:26,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:16:26,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:16:26,622 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:16:26,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:16:26,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:16:26,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:16:26,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:16:26,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:16:26,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:16:26,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:16:26,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:27,301 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:16:27,302 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.71200013161. 2022-02-08 08:16:27,302 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:16:27,303 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:16:27,900 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:16:28,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:16:28,246 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:16:28,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:16:28,262 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:16:28,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:16:28,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:16:28,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:16:28,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:16:28,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:16:28,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:28,841 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:16:28,842 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.53999996185. 2022-02-08 08:16:28,842 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:16:28,844 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:16:29,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:16:29,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:16:29,743 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:16:29,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:16:29,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:16:29,778 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:16:29,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:16:29,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:16:29,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:16:29,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:16:29,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:30,313 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:16:30,315 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.47300004959. 2022-02-08 08:16:30,315 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:16:30,316 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:16:30,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:16:31,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:16:31,660 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:16:31,660 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:16:31,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:16:31,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:16:31,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:16:32,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:16:32,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:16:32,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:16:32,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:33,415 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:16:33,417 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:16:35,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:35,749 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:16:35,749 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:16:37,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:16:37,898 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:16:37,898 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.58399987221. ############################################################ # # RAM FootPrint: 4612.92578125 # ############################################################ 2022-02-08 08:16:37,936 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 08:16:37,937 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 63 seconds. Reason: Iteration:4 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 10699 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:17:40,941 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 08:17:40,941 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 08:17:45,946 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:17:46,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:17:46,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:17:46,388 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:17:46,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:17:46,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:17:46,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:17:46,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:17:46,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:17:46,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:17:46,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:17:46,510 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:17:46,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:17:46,586 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['30476 memcached\n'], []) 2022-02-08 08:17:46,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 08:17:46,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:17:46,647 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['30476 memcached\n'], []) 2022-02-08 08:17:46,648 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:17:46,651 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:17:47,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:17:47,367 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:17:47,471 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:17:47,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:17:47,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:17:47,507 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:17:47,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:17:47,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:17:47,548 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:17:47,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:17:47,591 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:17:47,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:17:47,650 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['81701 memcached\n'], []) 2022-02-08 08:17:47,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 08:17:47,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:17:47,694 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['81701 memcached\n'], []) 2022-02-08 08:17:47,694 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:17:47,697 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:17:47,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:17:48,002 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:17:48,065 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:17:48,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:17:48,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:17:48,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:17:48,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:17:48,132 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:17:48,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:17:48,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:17:48,181 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:17:48,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:17:48,236 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['64814 memcached\n'], []) 2022-02-08 08:17:48,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 08:17:48,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:17:48,286 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['64814 memcached\n'], []) 2022-02-08 08:17:48,286 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:17:48,289 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:17:48,380 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:17:48,596 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:17:48,680 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:17:48,681 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:17:48,698 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:17:48,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:17:48,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:17:48,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:17:48,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:17:48,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:17:48,783 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:17:48,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:17:48,826 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['94682 memcached\n'], []) 2022-02-08 08:17:48,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 08:17:48,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:17:48,884 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['94682 memcached\n'], []) 2022-02-08 08:17:48,884 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:17:48,898 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:17:48,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:17:49,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:17:49,257 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:17:49,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:17:49,272 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:17:49,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:17:49,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:17:49,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:17:49,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:17:49,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:17:49,364 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:17:49,364 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:17:49,415 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['88544 memcached\n'], []) 2022-02-08 08:17:49,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 08:17:49,443 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:17:49,463 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['88544 memcached\n'], []) 2022-02-08 08:17:49,463 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:17:49,467 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:17:49,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 10721 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:17:49,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:17:49,845 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:17:49,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:17:49,859 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:17:49,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:17:49,885 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:17:49,903 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:17:49,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:17:49,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:17:49,941 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:17:49,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:17:49,986 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['123514 memcached\n'], []) 2022-02-08 08:17:49,987 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 08:17:50,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:17:50,032 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['123514 memcached\n'], []) 2022-02-08 08:17:50,033 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:17:50,036 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:17:50,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:17:50,346 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:17:50,420 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:17:50,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:17:50,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:17:50,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:17:50,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:17:50,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:17:50,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:17:50,505 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:17:50,519 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:17:50,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:17:50,569 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['57453 memcached\n'], []) 2022-02-08 08:17:50,571 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 08:17:50,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:17:50,615 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['57453 memcached\n'], []) 2022-02-08 08:17:50,617 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:17:50,618 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:17:50,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:17:50,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:17:51,002 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:17:51,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:17:51,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:17:51,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:17:51,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:17:51,061 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:17:51,073 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:17:51,086 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:17:51,101 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:17:51,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:17:51,193 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['72351 memcached\n'], []) 2022-02-08 08:17:51,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 08:17:51,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:17:51,242 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['72351 memcached\n'], []) 2022-02-08 08:17:51,242 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:17:51,243 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:17:51,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:17:51,579 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:17:51,641 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:17:51,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:17:51,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:17:51,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:17:51,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:17:51,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:17:51,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:17:51,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:17:51,736 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:17:51,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:17:51,779 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['7719 memcached\n'], []) 2022-02-08 08:17:51,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 08:17:51,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:17:51,823 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['7719 memcached\n'], []) 2022-02-08 08:17:51,823 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:17:51,826 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:17:51,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:17:52,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:17:52,178 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:17:52,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:17:52,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:17:52,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:17:52,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:17:52,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:17:52,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:17:52,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:17:52,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:17:52,305 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 08:17:52,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:17:52,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:17:54,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:17:55,124 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:17:55,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:17:55,948 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:17:55,963 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 08:17:55,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:17:55,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:17:56,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:17:56,049 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 08:17:56,049 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:17:56,052 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:17:56,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:17:56,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:17:56,434 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:17:56,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:17:56,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:17:56,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:17:56,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:17:56,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:17:56,512 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:17:56,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:17:56,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:17:56,553 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 08:17:56,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:17:56,599 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:17:57,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:17:58,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:17:58,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:17:58,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:17:58,864 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 08:17:58,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:17:58,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:17:58,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:17:58,967 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 08:17:58,967 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:17:58,970 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 08:17:59,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:17:59,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 08:17:59,341 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:17:59,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 08:17:59,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 08:17:59,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 08:17:59,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 08:17:59,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 08:17:59,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 08:17:59,426 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 08:17:59,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:17:59,457 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 08:17:59,484 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:17:59,499 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:17:59,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:17:59,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:17:59,543 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:17:59,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:17:59,572 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 08:17:59,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:17:59,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:17:59,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:17:59,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:17:59,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:17:59,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:17:59,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:17:59,719 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:17:59,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:17:59,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:17:59,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:17:59,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:17:59,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:17:59,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:17:59,864 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:17:59,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:17:59,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:17:59,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:17:59,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:17:59,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:17:59,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:17:59,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:00,005 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:00,020 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:00,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:18:00,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:00,081 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:00,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:00,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:00,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:00,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:00,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:00,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:00,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:00,227 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:18:00,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:00,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:00,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:00,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:00,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:00,334 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:00,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:00,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:00,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:00,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:18:00,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:00,436 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:00,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:00,471 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:00,489 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:00,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:00,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:00,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:00,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:00,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:18:00,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:00,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:00,614 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:00,630 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:00,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:00,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:00,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:00,698 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:00,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:00,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:18:00,757 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:00,775 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:00,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:00,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:00,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:00,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:00,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:00,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:00,900 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 08:18:00,903 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:18:01,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:18:01,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:18:01,461 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:18:01,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:18:01,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:18:01,655 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:18:01,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:18:01,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:18:01,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:18:01,948 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:18:01,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:18:02,081 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 08:18:02,107 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:18:02,227 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:04,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:18:05,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:05,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:05,719 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:18:05,763 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 08:18:05,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:18:05,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:05,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:18:05,848 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 08:18:05,848 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:18:05,851 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 08:18:05,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:18:06,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 08:18:06,292 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:18:06,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 08:18:06,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 08:18:06,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 08:18:06,341 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 08:18:06,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 08:18:06,372 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 08:18:06,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 08:18:06,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:18:06,417 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 08:18:06,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:18:06,464 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:06,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:18:06,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:06,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:06,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:18:06,539 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 08:18:06,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:18:06,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:06,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:18:06,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:06,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:18:06,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:06,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:06,676 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:06,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:06,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:06,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:06,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:06,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:06,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:06,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:18:06,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:06,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:06,885 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:06,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:06,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:06,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:06,956 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:06,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:06,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:07,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:18:07,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:07,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:07,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:07,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:07,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:07,107 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:07,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:07,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:07,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:07,168 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:18:07,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:07,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:07,216 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:07,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:07,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:07,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:07,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:07,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:07,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:07,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:18:07,357 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:07,372 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:07,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:07,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:07,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:07,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:07,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:07,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:07,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:07,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:18:07,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:07,543 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:07,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:07,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:07,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:07,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:07,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:07,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:07,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:07,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:18:07,693 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:07,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:07,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:07,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:07,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:07,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:07,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:07,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:07,832 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 08:18:07,835 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:18:07,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:18:08,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:18:08,194 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:18:08,196 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:18:08,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:18:08,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:18:08,237 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:18:08,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:18:08,275 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:18:08,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:18:08,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:18:08,342 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 08:18:08,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:18:08,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:09,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:18:10,331 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:10,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:11,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:18:11,170 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 08:18:11,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:18:11,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:11,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:18:11,332 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 08:18:11,332 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:18:11,335 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:18:11,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:18:11,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:18:11,775 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:18:11,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:18:11,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:18:11,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:18:11,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:18:11,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:18:11,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:18:11,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:18:11,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:18:11,943 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 08:18:11,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:18:11,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:14,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:18:15,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:15,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:16,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:18:16,252 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 08:18:16,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:18:16,270 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:16,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:18:16,357 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 08:18:16,357 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:18:16,359 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 08:18:16,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:18:16,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 08:18:16,811 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:18:16,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 08:18:16,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 08:18:16,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 08:18:16,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 08:18:16,877 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 08:18:16,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 08:18:16,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 08:18:16,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:18:16,946 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 08:18:16,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:18:16,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:17,017 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:18:17,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:17,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:17,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:18:17,085 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 08:18:17,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:18:17,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:17,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:18:17,137 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:17,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:18:17,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:17,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:17,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:17,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:17,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:17,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:17,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:17,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:17,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:17,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:18:17,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:17,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:17,487 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:17,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:17,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:17,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:17,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:17,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:17,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:17,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:18:17,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:17,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:17,693 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:17,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:17,729 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:17,747 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:17,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:17,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:17,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:17,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:18:17,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:17,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:17,880 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:17,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:17,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:17,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:17,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:17,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:17,994 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:18,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:18:18,035 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:18,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:18,073 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:18,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:18,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:18,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:18,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:18,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:18,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:18,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:18:18,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:18,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:18,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:18,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:18,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:18,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:18,321 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:18,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:18,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:18,372 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:18:18,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:18,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:18,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:18,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:18,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:18,492 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:18,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:18,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:18,549 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 08:18:18,552 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 08:18:18,632 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:18:18,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 08:18:18,943 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:18:18,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 08:18:18,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 08:18:18,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 08:18:18,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 08:18:19,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 08:18:19,025 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 08:18:19,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 08:18:19,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:18:19,073 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 08:18:19,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:18:19,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:19,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:18:19,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:19,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:19,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:18:19,194 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 08:18:19,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:18:19,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:19,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:18:19,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:19,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:18:19,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:19,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:19,334 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:19,357 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:19,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:19,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:19,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:19,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:19,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:19,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:18:19,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:19,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:19,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:19,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:19,581 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:19,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:19,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:19,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:19,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:19,670 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:18:19,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:19,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:19,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:19,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:19,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:19,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:19,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:19,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:19,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:19,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:18:19,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:19,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:19,895 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:19,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:19,928 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:19,946 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:19,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:19,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:19,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:20,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:18:20,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:20,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:20,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:20,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:20,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:20,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:20,127 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:20,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:20,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:20,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:18:20,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:20,206 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:20,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:20,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:20,252 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:20,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:20,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:20,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:20,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:20,334 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:18:20,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:20,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:20,385 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:20,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:20,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:20,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:20,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:20,471 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:20,487 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 08:18:20,490 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:18:20,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:18:20,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:18:20,941 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:18:20,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:18:20,959 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:18:20,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:18:20,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:18:21,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:18:21,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:18:21,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:18:21,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:18:21,079 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 08:18:21,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:18:21,132 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:23,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:18:24,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:24,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:25,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:18:25,283 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 08:18:25,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:18:25,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:25,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:18:25,375 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 08:18:25,375 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:18:25,378 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:18:25,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:18:25,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:18:25,770 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:18:25,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:18:25,788 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:18:25,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:18:25,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:18:25,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:18:25,852 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:18:25,867 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:18:25,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:18:25,897 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 08:18:25,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:18:25,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:26,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:18:27,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:27,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:27,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:18:27,920 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 08:18:27,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:18:27,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:27,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:18:27,993 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 08:18:27,994 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:18:27,996 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:18:28,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:18:29,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:18:29,471 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:18:29,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:18:29,487 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:18:29,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:18:29,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:18:29,561 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:18:29,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:18:29,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:18:29,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:18:29,631 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 08:18:29,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:18:29,681 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:33,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:18:33,701 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:34,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:34,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:18:34,319 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 08:18:34,321 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:18:34,339 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:34,375 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:18:34,391 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 08:18:34,391 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:18:34,394 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:18:34,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:18:35,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:18:35,578 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:18:35,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:18:35,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:18:35,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:18:35,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:18:35,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:18:35,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:18:35,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:18:35,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:18:35,910 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 08:18:35,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:18:35,969 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:37,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:18:38,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:38,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:39,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:18:39,532 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 08:18:39,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:18:39,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:39,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:18:39,710 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 08:18:39,711 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:18:39,713 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 08:18:39,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:18:40,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 08:18:40,111 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:18:40,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 08:18:40,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 08:18:40,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 08:18:40,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 08:18:40,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 08:18:40,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 08:18:40,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 08:18:40,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:18:40,255 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 08:18:40,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:18:40,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:40,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:18:40,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:40,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:18:40,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:18:40,384 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 08:18:40,385 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:18:40,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:18:40,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:18:40,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:40,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:18:40,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:40,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:40,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:40,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:40,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:40,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:40,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:40,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:18:40,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:40,719 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:18:40,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:40,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:40,775 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:40,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:40,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:40,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:40,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:40,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:18:40,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:40,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:18:40,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:40,948 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:40,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:40,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:41,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:41,017 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:41,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:41,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:18:41,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:41,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:18:41,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:41,124 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:41,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:41,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:41,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:41,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:41,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:41,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:18:41,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:41,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:18:41,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:41,306 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:41,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:41,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:41,364 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:41,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:41,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:41,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:18:41,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:41,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:18:41,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:41,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:41,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:41,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:41,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:41,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:41,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:41,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:18:41,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:41,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:18:41,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:41,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:41,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:41,686 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:41,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:41,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:41,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:41,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:18:41,775 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 08:18:41,776 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:18:41,779 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:18:41,852 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:18:42,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:18:42,174 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:18:42,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:18:42,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:18:42,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:18:42,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:18:42,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:18:42,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:18:42,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:18:42,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:18:42,546 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:18:42,546 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.769999980927. 2022-02-08 08:18:42,546 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:18:42,549 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:18:42,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:18:43,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:18:43,263 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:18:43,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:18:43,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:18:43,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:18:43,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:18:43,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:18:43,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:18:43,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:18:43,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:18:43,891 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:18:43,891 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:18:45,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:18:46,371 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:18:46,372 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:18:48,372 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:18:48,726 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:18:48,726 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 10731 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:18:50,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:18:51,022 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:18:51,022 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:18:53,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:18:53,322 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:18:53,322 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:18:55,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:18:55,529 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:18:55,529 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:18:57,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:18:57,772 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:18:57,772 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:18:59,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:18:59,982 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:18:59,982 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:19:01,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:19:02,177 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:19:02,177 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:19:04,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:19:04,365 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:19:04,365 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:19:06,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:19:06,530 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:19:06,530 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:19:08,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:19:08,683 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:19:08,684 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:19:10,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:19:10,831 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:19:10,831 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:19:12,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:19:12,986 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:19:12,986 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:19:14,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:19:15,131 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:19:15,131 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:19:17,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:19:17,286 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:19:17,288 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:19:19,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:19:19,424 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:19:19,426 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 36.8790001869. 2022-02-08 08:19:19,426 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:19:19,427 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:19:20,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:19:21,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:19:21,117 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:19:21,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:19:21,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:19:21,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:19:21,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:19:21,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:19:21,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:19:21,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:19:21,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:19:21,823 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:19:21,823 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.39799976349. 2022-02-08 08:19:21,823 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:19:21,825 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:19:22,686 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:19:22,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:19:23,048 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:19:23,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:19:23,063 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:19:23,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:19:23,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:19:23,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:19:23,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:19:23,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:19:23,168 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:19:23,693 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:19:23,694 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.87000012398. 2022-02-08 08:19:23,694 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:19:23,696 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:19:24,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:19:24,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:19:24,765 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:19:24,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:19:24,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:19:24,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:19:24,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:19:24,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:19:24,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:19:24,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:19:24,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:19:25,404 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:19:25,404 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.71000003815. 2022-02-08 08:19:25,404 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:19:25,405 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:19:26,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:19:26,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:19:26,944 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:19:26,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:19:26,959 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:19:26,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:19:27,010 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:19:27,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:19:27,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:19:27,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:19:27,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:19:27,503 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:19:27,505 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.10100007057. 2022-02-08 08:19:27,505 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:19:27,506 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:19:28,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:19:28,278 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:19:28,348 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:19:28,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:19:28,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:19:28,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:19:28,397 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:19:28,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:19:28,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:19:28,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:19:28,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:19:28,926 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:19:28,927 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.42199993134. 2022-02-08 08:19:28,927 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:19:28,930 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:19:30,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:19:30,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:19:30,858 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:19:30,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:19:30,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:19:30,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:19:30,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:19:30,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:19:30,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:19:31,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:19:31,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:19:32,355 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:19:32,357 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.43000006676. 2022-02-08 08:19:32,357 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:19:32,358 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:19:33,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:19:33,895 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:19:34,049 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:19:34,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:19:34,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:19:34,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:19:34,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:19:34,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:19:34,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:19:34,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:19:34,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:19:35,375 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:19:35,377 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.01999998093. ############################################################ # # RAM FootPrint: 4618.5 # ############################################################ 2022-02-08 08:19:35,414 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 08:19:35,414 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 90 seconds. Reason: Iteration:5 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 10742 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 10762 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:21:05,420 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 08:21:05,420 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 08:21:10,423 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:21:10,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:10,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:21:10,865 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:10,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:21:10,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:21:10,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:21:10,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:21:10,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:21:10,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:21:10,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:21:10,961 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:21:10,961 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:21:11,046 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['31121 memcached\n'], []) 2022-02-08 08:21:11,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 08:21:11,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:21:11,095 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['31121 memcached\n'], []) 2022-02-08 08:21:11,095 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:21:11,099 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:21:11,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:11,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:21:11,986 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:11,987 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:21:12,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:21:12,017 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:21:12,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:21:12,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:21:12,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:21:12,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:21:12,105 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:21:12,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:21:12,160 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['82625 memcached\n'], []) 2022-02-08 08:21:12,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 08:21:12,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:21:12,217 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['82625 memcached\n'], []) 2022-02-08 08:21:12,217 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:21:12,220 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:21:12,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:12,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:21:12,611 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:12,612 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:21:12,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:21:12,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:21:12,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:21:12,673 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:21:12,686 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:21:12,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:21:12,714 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:21:12,716 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:21:12,757 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['65601 memcached\n'], []) 2022-02-08 08:21:12,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 08:21:12,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:21:12,806 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['65601 memcached\n'], []) 2022-02-08 08:21:12,806 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:21:12,809 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:21:12,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:13,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:21:13,252 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:13,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:21:13,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:21:13,280 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:21:13,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:21:13,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:21:13,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:21:13,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:21:13,355 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:21:13,357 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:21:13,426 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['95356 memcached\n'], []) 2022-02-08 08:21:13,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 08:21:13,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:21:13,483 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['95356 memcached\n'], []) 2022-02-08 08:21:13,483 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:21:13,486 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:21:13,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:13,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:21:13,828 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:13,829 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:21:13,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:21:13,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:21:13,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:21:13,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:21:13,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:21:13,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:21:13,927 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:21:13,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:21:13,996 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['89231 memcached\n'], []) 2022-02-08 08:21:13,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 08:21:14,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:21:14,046 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['89231 memcached\n'], []) 2022-02-08 08:21:14,046 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:21:14,049 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:21:14,135 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:14,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:21:14,424 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:14,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:21:14,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:21:14,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:21:14,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:21:14,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:21:14,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:21:14,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:21:14,526 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:21:14,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:21:14,628 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['124205 memcached\n'], []) 2022-02-08 08:21:14,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 08:21:14,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:21:14,677 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['124205 memcached\n'], []) 2022-02-08 08:21:14,677 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:21:14,680 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:21:14,793 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:15,005 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:21:15,065 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:15,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:21:15,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:21:15,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:21:15,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:21:15,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:21:15,132 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:21:15,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:21:15,158 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:21:15,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:21:15,203 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['58099 memcached\n'], []) 2022-02-08 08:21:15,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 08:21:15,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:21:15,257 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['58099 memcached\n'], []) 2022-02-08 08:21:15,257 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:21:15,259 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:21:15,349 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:15,566 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:21:15,625 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:15,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:21:15,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:21:15,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:21:15,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:21:15,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:21:15,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:21:15,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:21:15,720 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:21:15,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:21:15,783 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['73009 memcached\n'], []) 2022-02-08 08:21:15,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 08:21:15,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:21:15,835 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['73009 memcached\n'], []) 2022-02-08 08:21:15,835 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:21:15,838 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:21:15,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:16,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:21:16,210 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:16,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:21:16,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:21:16,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:21:16,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:21:16,278 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:21:16,295 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:21:16,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:21:16,326 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:21:16,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:21:16,384 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['8434 memcached\n'], []) 2022-02-08 08:21:16,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 08:21:16,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:21:16,431 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['8434 memcached\n'], []) 2022-02-08 08:21:16,431 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:21:16,434 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:21:16,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:16,752 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:21:16,815 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:16,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:21:16,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:21:16,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:21:16,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:21:16,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:21:16,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:21:16,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:21:16,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:21:16,940 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 08:21:16,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:21:16,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:18,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:21:19,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:20,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:20,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:21:20,427 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 08:21:20,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:21:20,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:20,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:21:20,516 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 08:21:20,517 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:21:20,519 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:21:20,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:20,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:21:20,865 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:20,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:21:20,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:21:20,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:21:20,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:21:20,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:21:20,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:21:20,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:21:20,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:21:20,980 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 08:21:21,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:21:21,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:23,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:21:24,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:24,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:25,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:21:25,122 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 08:21:25,124 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:21:25,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:25,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:21:25,217 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 08:21:25,217 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:21:25,220 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 08:21:25,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:25,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 08:21:25,579 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:25,581 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 08:21:25,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 08:21:25,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 08:21:25,619 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 08:21:25,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 08:21:25,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 08:21:25,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 08:21:25,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:21:25,694 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 08:21:25,724 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:21:25,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:25,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:21:25,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:25,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:25,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:21:25,816 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 08:21:25,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:21:25,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:25,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:21:25,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:25,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:21:25,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:25,926 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:25,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:25,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:25,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:26,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:26,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:26,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:26,089 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:26,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:21:26,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:26,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:26,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:26,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:26,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:26,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:26,239 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:26,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:26,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:26,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:21:26,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:26,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:26,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:26,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:26,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:26,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:26,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:26,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:26,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:26,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:21:26,499 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:26,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:26,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:26,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:26,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:26,579 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:26,596 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:26,614 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:26,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:26,655 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:21:26,681 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:26,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:26,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:26,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:26,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:26,767 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:26,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:26,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:26,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:26,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:21:26,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:26,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:26,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:26,900 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:26,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:26,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:26,948 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:26,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:26,982 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:27,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:21:27,022 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:27,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:27,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:27,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:27,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:27,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:27,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:27,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:27,164 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 08:21:27,167 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:21:27,397 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:27,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:21:28,502 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:28,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:21:28,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:21:28,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:21:29,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:21:29,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:21:29,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:21:29,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:21:29,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:21:29,950 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 08:21:29,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:21:30,168 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:31,147 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:21:32,073 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:32,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:32,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:21:32,898 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 08:21:32,900 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:21:32,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:33,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:21:33,194 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 08:21:33,194 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:21:33,197 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 08:21:33,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:33,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 08:21:33,589 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:33,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 08:21:33,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 08:21:33,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 08:21:33,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 08:21:33,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 08:21:33,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 08:21:33,681 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 08:21:33,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:21:33,711 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 08:21:33,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:21:33,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:33,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:21:33,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:33,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:33,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:21:33,828 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 08:21:33,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:21:33,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:33,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:21:33,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:33,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:21:33,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:33,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:33,959 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:33,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:34,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:34,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:34,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:34,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:34,089 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:34,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:21:34,132 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:34,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:34,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:34,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:34,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:34,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:34,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:34,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:34,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:34,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:21:34,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:34,334 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:34,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:34,375 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:34,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:34,410 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:34,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:34,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:34,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:34,494 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:21:34,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:34,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:34,551 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:34,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:34,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:34,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:34,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:34,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:34,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:34,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:21:34,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:34,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:34,716 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:34,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:34,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:34,762 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:34,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:34,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:34,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:34,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:21:34,839 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:34,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:34,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:34,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:34,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:34,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:34,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:34,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:34,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:34,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:21:35,002 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:35,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:35,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:35,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:35,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:35,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:35,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:35,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:35,140 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 08:21:35,142 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:21:35,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:35,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:21:35,516 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:35,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:21:35,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:21:35,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:21:35,563 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:21:35,581 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:21:35,594 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:21:35,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:21:35,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:21:35,638 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 08:21:35,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:21:35,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:39,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:21:40,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:40,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:40,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:21:40,947 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 08:21:40,948 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:21:40,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:41,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:21:41,138 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 08:21:41,138 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:21:41,141 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:21:41,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:41,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:21:41,542 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:41,543 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:21:41,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:21:41,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:21:41,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:21:41,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:21:41,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:21:41,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:21:41,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:21:41,690 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 08:21:41,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:21:41,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:45,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:21:45,928 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:46,331 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:46,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:21:46,720 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 08:21:46,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:21:46,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:46,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:21:46,822 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 08:21:46,822 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:21:46,825 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 08:21:46,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:47,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 08:21:47,196 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:47,196 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 08:21:47,211 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 08:21:47,224 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 08:21:47,237 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 08:21:47,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 08:21:47,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 08:21:47,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 08:21:47,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:21:47,313 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 08:21:47,339 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:21:47,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:47,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:21:47,385 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:47,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:47,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:21:47,430 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 08:21:47,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:21:47,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:47,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:21:47,489 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:47,512 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:21:47,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:47,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:47,576 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:47,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:47,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:47,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:47,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:47,686 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:47,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:47,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:21:47,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:47,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:47,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:47,788 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:47,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:47,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:47,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:47,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:47,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:47,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:21:47,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:47,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:47,946 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:47,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:47,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:47,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:48,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:48,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:48,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:48,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:21:48,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:48,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:48,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:48,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:48,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:48,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:48,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:48,196 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:48,211 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:48,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:21:48,270 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:48,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:48,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:48,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:48,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:48,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:48,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:48,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:48,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:48,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:21:48,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:48,484 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:48,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:48,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:48,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:48,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:48,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:48,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:48,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:48,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:21:48,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:48,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:48,655 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:48,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:48,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:48,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:48,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:48,739 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:48,757 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 08:21:48,760 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 08:21:48,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:49,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 08:21:49,130 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:49,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 08:21:49,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 08:21:49,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 08:21:49,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 08:21:49,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 08:21:49,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 08:21:49,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 08:21:49,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:21:49,250 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 08:21:49,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:21:49,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:49,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:21:49,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:49,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:49,362 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:21:49,377 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 08:21:49,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:21:49,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:49,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:21:49,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:49,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:21:49,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:49,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:49,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:49,538 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:49,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:49,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 5 queries submitted, 0 failed, 10780 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:21:49,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:49,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:21:49,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:49,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:21:49,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:49,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:49,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:49,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:49,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:49,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:49,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:49,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:21:49,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:49,852 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:21:49,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:49,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:49,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:49,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:49,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:49,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:49,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:50,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:21:50,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:50,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:21:50,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:50,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:50,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:50,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:50,127 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:50,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:50,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:50,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:21:50,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:50,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:21:50,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:50,239 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:50,255 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:50,270 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:50,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:50,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:50,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:50,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:21:50,349 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:50,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:21:50,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:50,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:50,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:50,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:50,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:50,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:50,571 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:50,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:21:50,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:50,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:21:50,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:50,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:50,670 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:50,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:50,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:50,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:50,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:50,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:21:50,776 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 08:21:50,779 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:21:50,852 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:51,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:21:51,190 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:51,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:21:51,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:21:51,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:21:51,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:21:51,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:21:51,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:21:51,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:21:51,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:21:51,346 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 08:21:51,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:21:51,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:55,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:21:56,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:56,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:21:57,232 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:21:57,246 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 08:21:57,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:21:57,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:21:57,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:21:57,299 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 08:21:57,299 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:21:57,303 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:21:57,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:21:57,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:21:57,683 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:21:57,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:21:57,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:21:57,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:21:57,724 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:21:57,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:21:57,757 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:21:57,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:21:57,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:21:57,801 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 08:21:57,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:21:57,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:22:00,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:22:00,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:22:00,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:22:01,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:22:01,177 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 08:22:01,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:22:01,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:22:01,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:22:01,250 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 08:22:01,252 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:22:01,253 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:22:01,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:22:02,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:22:02,165 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:22:02,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:22:02,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:22:02,239 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:22:02,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:22:02,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:22:02,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:22:02,367 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:22:02,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:22:02,398 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 08:22:02,436 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:22:02,454 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:22:02,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:22:03,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:22:03,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:22:04,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:22:04,125 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 08:22:04,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:22:04,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:22:04,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:22:04,180 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 08:22:04,180 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:22:04,183 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:22:04,272 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:22:04,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:22:04,565 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:22:04,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:22:04,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:22:04,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:22:04,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:22:04,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:22:04,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:22:04,665 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:22:04,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:22:04,707 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 08:22:04,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:22:04,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:22:08,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:22:09,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:22:09,920 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:22:10,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:22:10,334 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 08:22:10,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:22:10,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:22:10,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:22:10,418 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 08:22:10,418 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:22:10,421 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 08:22:10,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:22:10,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 08:22:10,816 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:22:10,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 08:22:10,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 08:22:10,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 08:22:10,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 08:22:10,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 08:22:10,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 08:22:10,908 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 08:22:10,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:22:10,943 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 08:22:10,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:22:10,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:22:11,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:22:11,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:22:11,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:22:11,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:22:11,069 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 08:22:11,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:22:11,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:22:11,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:22:11,114 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:22:11,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:22:11,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:22:11,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:22:11,211 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:22:11,234 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:22:11,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:22:11,280 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:22:11,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:22:11,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:22:11,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:22:11,375 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:22:11,395 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:22:11,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:22:11,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:22:11,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:22:11,487 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:22:11,507 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:22:11,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:22:11,556 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:22:11,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:22:11,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:22:11,612 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:22:11,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:22:11,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:22:11,670 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:22:11,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:22:11,706 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:22:11,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:22:11,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:22:11,762 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:22:11,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:22:11,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:22:11,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:22:11,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:22:11,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:22:11,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:22:11,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:22:11,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:22:11,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:22:11,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:22:11,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:22:11,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:22:11,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:22:12,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:22:12,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:22:12,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:22:12,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:22:12,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:22:12,094 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:22:12,114 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:22:12,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:22:12,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:22:12,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:22:12,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:22:12,216 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:22:12,232 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:22:12,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:22:12,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:22:12,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:22:12,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:22:12,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:22:12,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:22:12,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:22:12,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:22:12,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:22:12,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:22:12,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:22:12,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:22:12,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:22:12,487 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 08:22:12,490 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:22:12,493 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:22:12,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:22:12,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:22:12,910 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:22:12,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:22:12,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:22:12,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:22:12,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:22:12,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:22:12,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:22:13,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:22:13,022 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:13,434 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:22:13,434 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.945999860764. 2022-02-08 08:22:13,434 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:22:13,437 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:22:13,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:22:13,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:22:14,048 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:22:14,048 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:22:14,073 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:22:14,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:22:14,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:22:14,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:22:14,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:22:14,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:22:14,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:14,615 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:14,615 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:16,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:16,812 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:16,812 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:18,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:19,138 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:19,138 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:21,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:21,404 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:21,404 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:23,405 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:23,609 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:23,609 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:25,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:25,809 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:25,809 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:27,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:27,980 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:27,980 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:29,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:30,147 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:30,147 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:32,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:32,325 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:32,325 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:34,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:34,588 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:34,589 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:36,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:36,753 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:36,753 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:38,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:38,907 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:38,908 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:40,908 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:41,075 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:41,075 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:43,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:43,236 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:43,236 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:45,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:45,385 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:45,385 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:47,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:47,539 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:47,539 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:49,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 10780 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:22:49,688 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:49,690 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:51,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:51,841 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:51,841 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:53,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:53,986 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:53,986 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:55,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:56,130 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:22:56,130 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:22:58,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:22:58,269 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:22:58,270 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 44.8359999657. 2022-02-08 08:22:58,270 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:22:58,273 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:22:58,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:22:59,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:22:59,578 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:22:59,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:22:59,596 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:22:59,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:22:59,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:22:59,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:22:59,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:22:59,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:22:59,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:23:00,263 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:23:00,263 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.99300003052. 2022-02-08 08:23:00,265 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:23:00,266 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:23:00,839 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:23:01,135 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:23:01,236 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:23:01,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:23:01,252 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:23:01,272 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:23:01,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:23:01,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:23:01,321 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:23:01,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:23:01,367 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:23:01,901 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:23:01,904 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.63999986649. 2022-02-08 08:23:01,904 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:23:01,907 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:23:02,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:23:02,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:23:03,131 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:23:03,132 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:23:03,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:23:03,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:23:03,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:23:03,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:23:03,239 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:23:03,275 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:23:03,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:23:03,782 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:23:03,783 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.87800002098. 2022-02-08 08:23:03,783 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:23:03,786 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:23:04,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:23:04,706 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:23:04,799 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:23:04,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:23:04,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:23:04,829 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:23:04,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:23:04,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:23:04,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:23:04,903 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:23:04,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:23:05,397 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:23:05,397 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.6130001545. 2022-02-08 08:23:05,398 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:23:05,401 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:23:06,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:23:06,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:23:06,640 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:23:06,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:23:06,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:23:06,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:23:06,681 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:23:06,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:23:06,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:23:06,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:23:06,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:23:07,220 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:23:07,220 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.8220000267. 2022-02-08 08:23:07,220 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:23:07,221 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:23:07,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:23:07,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:23:08,056 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:23:08,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:23:08,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:23:08,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:23:08,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:23:08,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:23:08,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:23:08,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:23:08,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:23:08,599 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:23:08,599 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.37999987602. 2022-02-08 08:23:08,601 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:23:08,602 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:23:09,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:23:09,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:23:09,927 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:23:09,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:23:10,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:23:10,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:23:10,321 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:23:10,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:23:10,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:23:10,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:23:10,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:23:11,476 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:23:11,476 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:23:13,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:23:13,717 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:23:13,719 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:23:15,719 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:23:16,845 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:23:16,845 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 8.24500012398. ############################################################ # # RAM FootPrint: 4633.86328125 # ############################################################ 2022-02-08 08:23:16,884 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 08:23:16,885 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 113 seconds. Reason: Iteration:6 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 10803 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 10823 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:25:09,891 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 08:25:09,891 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 08:25:14,894 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:25:14,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:15,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:25:15,312 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:15,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:25:15,331 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:25:15,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:25:15,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:25:15,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:25:15,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:25:15,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:25:15,436 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:25:15,436 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:25:15,496 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['31819 memcached\n'], []) 2022-02-08 08:25:15,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 08:25:15,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:25:15,543 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['31819 memcached\n'], []) 2022-02-08 08:25:15,545 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:25:15,546 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:25:16,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:16,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:25:16,493 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:16,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:25:16,507 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:25:16,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:25:16,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:25:16,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:25:16,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:25:16,612 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:25:16,628 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:25:16,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:25:16,684 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['83468 memcached\n'], []) 2022-02-08 08:25:16,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 08:25:16,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:25:16,746 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['83468 memcached\n'], []) 2022-02-08 08:25:16,746 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:25:16,750 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:25:16,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:17,035 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:25:17,101 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:17,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:25:17,117 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:25:17,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:25:17,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:25:17,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:25:17,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:25:17,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:25:17,207 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:25:17,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:25:17,252 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['66353 memcached\n'], []) 2022-02-08 08:25:17,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 08:25:17,278 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:25:17,301 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['66353 memcached\n'], []) 2022-02-08 08:25:17,302 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:25:17,303 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:25:17,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:17,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:25:17,707 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:17,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:25:17,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:25:17,739 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:25:17,757 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:25:17,778 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:25:17,790 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:25:17,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:25:17,818 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:25:17,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:25:17,904 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['96054 memcached\n'], []) 2022-02-08 08:25:17,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 08:25:17,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:25:17,953 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['96054 memcached\n'], []) 2022-02-08 08:25:17,953 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:25:17,957 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:25:18,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:18,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:25:18,296 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:18,298 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:25:18,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:25:18,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:25:18,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:25:18,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:25:18,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:25:18,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:25:18,395 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:25:18,395 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:25:18,444 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['89887 memcached\n'], []) 2022-02-08 08:25:18,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 08:25:18,471 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:25:18,493 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['89887 memcached\n'], []) 2022-02-08 08:25:18,493 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:25:18,497 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:25:18,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:18,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:25:18,888 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:18,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:25:18,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:25:18,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:25:18,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:25:18,946 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:25:18,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:25:18,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:25:18,983 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:25:18,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:25:19,094 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['124859 memcached\n'], []) 2022-02-08 08:25:19,094 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 08:25:19,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:25:19,141 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['124859 memcached\n'], []) 2022-02-08 08:25:19,141 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:25:19,142 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:25:19,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:19,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:25:19,619 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:19,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:25:19,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:25:19,647 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:25:19,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:25:19,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:25:19,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:25:19,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:25:19,716 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:25:19,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:25:19,786 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['58802 memcached\n'], []) 2022-02-08 08:25:19,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 08:25:19,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:25:19,854 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['58802 memcached\n'], []) 2022-02-08 08:25:19,854 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:25:19,857 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:25:19,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:20,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:25:20,232 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:20,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:25:20,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:25:20,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:25:20,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:25:20,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:25:20,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:25:20,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:25:20,328 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:25:20,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:25:20,404 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['73705 memcached\n'], []) 2022-02-08 08:25:20,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 08:25:20,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:25:20,451 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['73705 memcached\n'], []) 2022-02-08 08:25:20,453 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:25:20,456 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:25:20,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:21,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:25:21,581 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:21,581 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:25:21,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:25:21,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:25:21,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:25:21,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:25:21,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:25:21,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:25:21,711 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:25:21,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:25:21,773 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['9106 memcached\n'], []) 2022-02-08 08:25:21,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 08:25:21,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:25:21,823 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['9106 memcached\n'], []) 2022-02-08 08:25:21,823 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:25:21,826 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:25:21,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:22,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:25:22,177 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:22,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:25:22,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:25:22,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:25:22,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:25:22,242 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:25:22,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:25:22,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:25:22,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:25:22,305 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 08:25:22,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:25:22,349 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:25:25,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:25:26,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:27,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:27,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:25:27,486 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 08:25:27,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:25:27,499 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:25:27,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:25:27,572 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 08:25:27,573 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:25:27,575 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:25:27,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:27,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:25:27,926 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:27,926 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:25:27,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:25:27,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:25:27,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:25:27,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:25:27,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:25:28,010 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:25:28,025 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:25:28,040 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 08:25:28,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:25:28,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:25:29,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:25:30,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:30,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:31,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:25:31,302 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 08:25:31,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:25:31,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:25:31,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:25:31,394 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 08:25:31,394 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:25:31,395 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 08:25:31,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:31,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 08:25:31,868 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:31,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 08:25:31,885 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 08:25:31,900 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 08:25:31,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 08:25:31,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 08:25:31,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 08:25:31,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 08:25:31,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:25:31,999 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 08:25:32,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:25:32,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:25:32,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:25:32,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:32,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:32,117 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:25:32,131 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 08:25:32,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:25:32,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:25:32,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:25:32,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:32,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:25:32,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:32,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:32,275 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:32,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:32,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:32,339 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:32,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:32,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:32,405 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:32,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:25:32,443 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:32,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:32,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:32,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:32,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:32,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:32,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:32,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:32,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:32,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:25:32,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:32,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:32,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:32,676 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:32,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:32,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:32,732 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:32,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:32,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:32,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:25:32,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:32,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:32,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:32,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:32,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:32,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:32,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:32,926 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:32,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:32,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:25:32,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:32,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:33,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:33,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:33,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:33,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:33,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:33,104 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:33,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:33,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:25:33,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:33,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:33,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:33,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:33,227 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:33,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:33,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:33,290 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:33,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:33,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:25:33,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:33,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:33,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:33,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:33,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:33,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:33,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:33,482 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:33,500 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 08:25:33,502 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:25:33,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:34,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:25:34,838 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:34,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:25:35,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:25:35,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:25:35,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:25:35,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:25:35,410 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:25:35,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:25:35,670 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:25:35,802 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 08:25:35,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:25:35,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:25:39,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:25:40,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:40,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:41,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:25:41,038 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 08:25:41,038 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:25:41,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:25:41,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:25:41,128 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 08:25:41,128 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:25:41,131 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 08:25:41,252 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:41,471 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 08:25:41,529 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:41,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 08:25:41,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 08:25:41,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 08:25:41,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 08:25:41,581 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 08:25:41,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 08:25:41,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 08:25:41,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:25:41,631 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 08:25:41,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:25:41,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:25:41,693 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:25:41,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:41,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:41,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:25:41,759 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 08:25:41,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:25:41,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:25:41,788 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:25:41,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:41,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:25:41,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:41,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:41,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:41,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:41,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:41,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:41,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:41,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:42,017 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:42,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:25:42,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:42,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:42,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:42,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:42,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:42,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:42,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:42,168 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:42,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:42,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:25:42,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:42,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:42,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:42,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:42,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:42,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:42,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:42,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:42,339 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:42,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:25:42,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:42,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:42,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:42,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:42,433 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:42,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:42,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:42,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:42,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:42,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:25:42,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:42,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:42,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:42,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:42,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:42,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:42,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:42,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:42,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:42,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:25:42,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:42,716 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:42,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:42,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:42,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:42,778 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:42,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:42,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:42,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:42,839 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:25:42,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:42,874 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:42,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:42,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:42,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:42,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:42,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:42,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:42,983 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 08:25:42,984 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:25:43,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:43,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:25:43,325 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:43,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:25:43,341 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:25:43,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:25:43,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:25:43,385 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:25:43,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:25:43,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:25:43,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:25:43,441 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 08:25:43,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:25:43,484 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:25:47,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:25:47,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:48,334 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:48,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:25:48,765 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 08:25:48,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:25:48,790 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:25:48,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:25:48,861 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 08:25:48,861 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:25:48,865 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:25:48,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:49,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:25:49,247 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:49,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:25:49,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:25:49,280 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:25:49,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:25:49,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:25:49,331 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:25:49,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:25:49,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:25:49,388 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 08:25:49,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:25:49,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 10840 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:25:52,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:25:52,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:53,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:53,543 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:25:53,563 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 08:25:53,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:25:53,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:25:53,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:25:53,673 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 08:25:53,674 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:25:53,676 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 08:25:53,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:54,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 08:25:54,114 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:54,114 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 08:25:54,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 08:25:54,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 08:25:54,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 08:25:54,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 08:25:54,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 08:25:54,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 08:25:54,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:25:54,263 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 08:25:54,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:25:54,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:25:54,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:25:54,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:54,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:54,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:25:54,417 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 08:25:54,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:25:54,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:25:54,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:25:54,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:54,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:25:54,517 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:54,543 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:54,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:54,596 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:54,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:54,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:54,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:54,706 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:54,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:54,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:25:54,778 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:54,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:54,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:54,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:54,867 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:54,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:54,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:54,936 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:54,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:54,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:25:55,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:55,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:55,038 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:55,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:55,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:55,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:55,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:55,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:55,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:55,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:25:55,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:55,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:55,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:55,227 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:55,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:55,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:55,280 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:55,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:55,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:55,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:25:55,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:55,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:55,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:55,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:55,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:55,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:55,436 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:55,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:55,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:55,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:25:55,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:55,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:55,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:55,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:55,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:55,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:55,594 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:55,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:55,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:55,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:25:55,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:55,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:55,701 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:55,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:55,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:55,767 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:55,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:55,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:55,822 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 08:25:55,825 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 08:25:55,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:56,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 08:25:56,176 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:56,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 08:25:56,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 08:25:56,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 08:25:56,211 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 08:25:56,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 08:25:56,239 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 08:25:56,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 08:25:56,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:25:56,279 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 08:25:56,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:25:56,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:25:56,334 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:25:56,346 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:56,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:25:56,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:25:56,384 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 08:25:56,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:25:56,395 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:25:56,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:25:56,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:56,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:25:56,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:56,492 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:56,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:56,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:56,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:56,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:56,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:56,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:25:56,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:56,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:25:56,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:56,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:56,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:56,752 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:56,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:56,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:56,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:56,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:25:56,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:56,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:25:56,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:56,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:56,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:56,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:56,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:56,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:56,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:56,999 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:25:57,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:57,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:25:57,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:57,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:57,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:57,084 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:57,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:57,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:57,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:57,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:25:57,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:57,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:25:57,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:57,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:57,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:57,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:57,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:57,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:57,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:57,278 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:25:57,290 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:57,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:25:57,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:57,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:57,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:57,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:57,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:57,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:57,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:57,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:25:57,426 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:57,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:25:57,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:57,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:57,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:57,507 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:57,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:57,538 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:57,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:57,566 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:25:57,581 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 08:25:57,585 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:25:57,686 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:25:57,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:25:58,015 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:25:58,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:25:58,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:25:58,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:25:58,063 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:25:58,084 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:25:58,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:25:58,117 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:25:58,132 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:25:58,151 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 08:25:58,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:25:58,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:26:01,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:26:01,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:26:02,216 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:26:02,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:26:02,634 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 08:26:02,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:26:02,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:26:02,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:26:02,732 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 08:26:02,732 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:26:02,734 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:26:02,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:26:03,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:26:03,197 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:26:03,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:26:03,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:26:03,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:26:03,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:26:03,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:26:03,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:26:03,339 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:26:03,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:26:03,377 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 08:26:03,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:26:03,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:26:06,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:26:07,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:26:07,665 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:26:07,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:26:07,990 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 08:26:07,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:26:08,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:26:08,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:26:08,061 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 08:26:08,062 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:26:08,063 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:26:08,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:26:08,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:26:08,874 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:26:08,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:26:08,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:26:08,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:26:08,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:26:08,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:26:08,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:26:08,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:26:08,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:26:09,010 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 08:26:09,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:26:09,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:26:10,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:26:11,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:26:11,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:26:11,933 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:26:11,947 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 08:26:11,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:26:11,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:26:11,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:26:11,994 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 08:26:11,994 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:26:11,996 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:26:12,117 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:26:12,334 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:26:12,397 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:26:12,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:26:12,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:26:12,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:26:12,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:26:12,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:26:12,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:26:12,484 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:26:12,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:26:12,516 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 08:26:12,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:26:12,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:26:16,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:26:17,321 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:26:17,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:26:18,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:26:18,161 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 08:26:18,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:26:18,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:26:18,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:26:18,233 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 08:26:18,233 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:26:18,236 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 08:26:18,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:26:18,576 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 08:26:18,638 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:26:18,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 08:26:18,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 08:26:18,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 08:26:18,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 08:26:18,698 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 08:26:18,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 08:26:18,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 08:26:18,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:26:18,763 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 08:26:18,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:26:18,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:26:18,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:26:18,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:26:18,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:26:18,880 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:26:18,894 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 08:26:18,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:26:18,908 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:26:18,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:26:18,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:26:18,961 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:26:18,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:26:19,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:26:19,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:26:19,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:26:19,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:26:19,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:26:19,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:26:19,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:26:19,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:26:19,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:26:19,206 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:26:19,221 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:26:19,237 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:26:19,270 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:26:19,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:26:19,318 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:26:19,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:26:19,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:26:19,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:26:19,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:26:19,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:26:19,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:26:19,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:26:19,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:26:19,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:26:19,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:26:19,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:26:19,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:26:19,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:26:19,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:26:19,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:26:19,581 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:26:19,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:26:19,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:26:19,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:26:19,642 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:26:19,660 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:26:19,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:26:19,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:26:19,706 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:26:19,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:26:19,739 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:26:19,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:26:19,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:26:19,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:26:19,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:26:19,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:26:19,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:26:19,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:26:19,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:26:19,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:26:19,900 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:26:19,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:26:19,933 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:26:19,948 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:26:19,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:26:19,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:26:20,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:26:20,017 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:26:20,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:26:20,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:26:20,073 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:26:20,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:26:20,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:26:20,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:26:20,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:26:20,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:26:20,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:26:20,207 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 08:26:20,207 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:26:20,209 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:26:20,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:26:20,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:26:20,687 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:26:20,688 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:26:20,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:26:20,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:26:20,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:26:20,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:26:20,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:26:20,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:26:20,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:21,184 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:26:21,186 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.979000091553. 2022-02-08 08:26:21,186 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:26:21,188 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:26:21,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:26:21,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:26:21,589 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:26:21,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:26:21,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:26:21,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:26:21,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:26:21,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:26:21,670 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:26:21,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:26:21,698 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:21,851 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:21,851 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:26:23,852 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:24,006 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:24,006 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:26:26,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:26,151 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:26,151 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:26:28,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:28,305 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:28,305 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:26:30,306 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:30,460 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:30,460 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:26:32,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:32,614 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:32,614 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:26:34,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:34,786 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:34,786 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:26:36,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:36,943 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:36,944 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:26:38,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:39,111 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:39,111 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:26:41,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:41,368 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:41,368 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:26:43,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:43,836 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:43,838 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:26:45,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:46,144 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:46,145 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:26:48,147 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:48,696 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:48,696 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 10843 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:26:50,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:50,959 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:50,960 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:26:52,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:53,224 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:53,224 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:26:55,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:55,509 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:55,509 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:26:57,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:57,721 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:57,723 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:26:59,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:26:59,977 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:26:59,979 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:27:01,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:27:02,148 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:27:02,148 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 40.9629998207. 2022-02-08 08:27:02,148 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:27:02,153 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:27:02,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:27:03,295 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:27:03,415 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:27:03,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:27:03,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:27:03,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:27:03,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:27:03,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:27:03,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:27:03,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:27:03,599 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:27:04,187 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:27:04,188 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.03999996185. 2022-02-08 08:27:04,188 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:27:04,190 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:27:04,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:27:05,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:27:05,209 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:27:05,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:27:05,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:27:05,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:27:05,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:27:05,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:27:05,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:27:05,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:27:05,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:27:05,977 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:27:05,977 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.78900003433. 2022-02-08 08:27:05,979 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:27:05,983 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:27:06,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:27:07,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:27:07,141 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:27:07,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:27:07,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:27:07,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:27:07,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:27:07,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:27:07,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:27:07,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:27:07,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:27:07,826 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:27:07,826 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.84799981117. 2022-02-08 08:27:07,828 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:27:07,828 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:27:08,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:27:08,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:27:08,954 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:27:08,956 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:27:08,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:27:09,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:27:09,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:27:09,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:27:09,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:27:09,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:27:09,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:27:09,586 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:27:09,588 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.75899982452. 2022-02-08 08:27:09,588 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:27:09,589 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:27:10,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:27:10,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:27:10,605 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:27:10,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:27:10,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:27:10,632 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:27:10,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:27:10,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:27:10,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:27:10,698 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:27:10,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:27:11,168 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:27:11,168 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.58099985123. 2022-02-08 08:27:11,170 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:27:11,171 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:27:11,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:27:12,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:27:12,516 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:27:12,517 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:27:12,538 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:27:12,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:27:12,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:27:12,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:27:12,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:27:12,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:27:12,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:27:13,145 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:27:13,145 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.97599983215. 2022-02-08 08:27:13,147 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:27:13,148 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:27:13,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:27:14,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:27:14,517 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:27:14,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:27:14,665 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:27:14,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:27:14,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:27:15,084 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:27:15,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:27:15,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:27:15,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:27:16,042 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:27:16,042 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:27:18,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:27:18,193 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:27:18,193 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.04600000381. ############################################################ # # RAM FootPrint: 4659.4921875 # ############################################################ 2022-02-08 08:27:18,227 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 08:27:18,227 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 111 seconds. Reason: Iteration:7 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 10856 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 10882 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:29:09,234 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 08:29:09,234 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 08:29:14,237 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:29:14,341 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:14,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:29:14,674 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:14,676 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:29:14,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:29:14,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:29:14,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:29:14,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:29:14,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:29:14,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:29:14,862 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:29:14,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:29:14,931 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['32549 memcached\n'], []) 2022-02-08 08:29:14,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 08:29:14,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:29:14,982 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['32549 memcached\n'], []) 2022-02-08 08:29:14,983 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:29:14,986 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:29:15,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:15,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:29:15,941 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:15,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:29:15,956 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:29:15,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:29:15,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:29:16,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:29:16,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:29:16,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:29:16,058 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:29:16,058 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:29:16,119 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['84441 memcached\n'], []) 2022-02-08 08:29:16,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 08:29:16,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:29:16,178 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['84441 memcached\n'], []) 2022-02-08 08:29:16,178 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:29:16,184 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:29:16,306 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:16,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:29:16,584 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:16,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:29:16,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:29:16,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:29:16,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:29:16,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:29:16,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:29:16,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:29:16,694 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:29:16,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:29:16,742 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['67177 memcached\n'], []) 2022-02-08 08:29:16,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 08:29:16,767 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:29:16,790 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['67177 memcached\n'], []) 2022-02-08 08:29:16,792 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:29:16,795 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:29:16,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:17,107 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:29:17,167 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:17,168 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:29:17,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:29:17,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:29:17,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:29:17,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:29:17,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:29:17,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:29:17,262 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:29:17,262 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:29:17,328 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['96783 memcached\n'], []) 2022-02-08 08:29:17,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 08:29:17,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:29:17,375 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['96783 memcached\n'], []) 2022-02-08 08:29:17,375 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:29:17,380 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:29:17,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:17,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:29:17,734 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:17,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:29:17,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:29:17,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:29:17,778 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:29:17,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:29:17,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:29:17,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:29:17,835 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:29:17,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:29:17,881 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['90616 memcached\n'], []) 2022-02-08 08:29:17,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 08:29:17,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:29:17,928 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['90616 memcached\n'], []) 2022-02-08 08:29:17,930 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:29:17,931 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:29:18,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:18,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:29:18,289 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:18,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:29:18,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:29:18,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:29:18,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:29:18,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:29:18,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:29:18,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:29:18,384 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:29:18,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:29:18,430 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['125587 memcached\n'], []) 2022-02-08 08:29:18,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 08:29:18,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:29:18,477 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['125587 memcached\n'], []) 2022-02-08 08:29:18,477 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:29:18,480 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:29:18,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:18,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:29:18,854 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:18,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:29:18,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:29:18,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:29:18,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:29:18,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:29:18,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:29:18,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:29:18,950 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:29:18,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:29:19,006 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['59535 memcached\n'], []) 2022-02-08 08:29:19,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 08:29:19,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:29:19,056 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['59535 memcached\n'], []) 2022-02-08 08:29:19,056 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:29:19,059 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:29:19,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:19,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:29:19,440 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:19,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:29:19,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:29:19,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:29:19,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:29:19,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:29:19,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:29:19,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:29:19,540 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:29:19,540 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:29:19,614 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['74438 memcached\n'], []) 2022-02-08 08:29:19,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 08:29:19,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:29:19,660 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['74438 memcached\n'], []) 2022-02-08 08:29:19,660 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:29:19,663 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:29:19,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:19,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:29:20,029 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:20,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:29:20,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:29:20,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:29:20,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:29:20,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:29:20,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:29:20,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:29:20,125 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:29:20,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:29:20,173 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['9860 memcached\n'], []) 2022-02-08 08:29:20,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 08:29:20,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:29:20,219 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['9860 memcached\n'], []) 2022-02-08 08:29:20,220 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:29:20,223 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:29:20,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:20,571 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:29:20,650 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:20,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:29:20,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:29:20,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:29:20,693 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:29:20,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:29:20,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:29:20,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:29:20,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:29:20,779 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 08:29:20,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:29:20,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:21,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:29:22,375 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:22,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:23,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:29:23,246 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 08:29:23,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:29:23,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:23,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:29:23,335 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 08:29:23,335 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:29:23,338 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:29:23,426 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:23,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:29:23,709 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:23,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:29:23,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:29:23,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:29:23,747 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:29:23,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:29:23,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:29:23,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:29:23,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:29:23,819 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 08:29:23,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:29:23,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:24,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:29:25,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:25,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:25,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:29:26,003 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 08:29:26,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:29:26,025 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:26,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:29:26,115 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 08:29:26,115 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:29:26,117 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 08:29:26,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:26,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 08:29:26,469 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:26,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 08:29:26,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 08:29:26,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 08:29:26,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 08:29:26,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 08:29:26,540 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 08:29:26,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 08:29:26,571 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:29:26,588 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 08:29:26,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:29:26,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:26,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:29:26,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:26,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:26,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:29:26,727 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 08:29:26,729 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:29:26,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:26,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:29:26,775 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:26,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:29:26,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:26,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:26,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:26,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:26,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:26,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:26,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:26,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:27,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:27,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:29:27,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:27,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:27,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:27,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:27,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:27,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:27,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:27,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:27,196 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:27,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:29:27,232 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:27,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:27,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:27,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:27,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:27,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:27,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:27,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:27,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:27,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:29:27,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:27,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:27,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:27,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:27,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:27,482 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:27,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:27,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:27,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:27,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:29:27,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:27,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:27,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:27,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:27,642 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:27,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:27,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:27,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:27,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:27,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:29:27,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:27,762 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:27,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:27,793 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:27,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:27,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:27,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:27,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:27,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:27,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:29:27,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:27,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:27,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:27,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:27,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:28,010 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:28,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:28,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:28,068 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 08:29:28,071 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:29:28,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:28,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:29:28,431 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:28,433 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:29:28,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:29:28,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:29:28,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:29:28,505 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:29:28,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:29:28,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:29:28,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:29:28,572 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 08:29:28,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:29:28,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:29,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:29:29,665 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:29,969 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:30,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:29:30,253 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 08:29:30,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:29:30,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:30,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:29:30,344 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 08:29:30,344 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:29:30,345 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 08:29:30,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:30,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 08:29:30,736 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:30,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 08:29:30,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 08:29:30,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 08:29:30,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 08:29:30,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 08:29:30,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 08:29:30,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 08:29:30,839 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:29:30,857 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 08:29:30,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:29:30,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:30,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:29:30,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:30,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:30,969 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:29:30,983 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 08:29:30,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:29:30,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:31,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:29:31,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:31,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:29:31,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:31,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:31,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:31,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:31,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:31,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:31,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:31,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:31,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:31,290 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:29:31,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:31,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:31,349 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:31,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:31,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:31,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:31,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:31,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:31,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:31,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:29:31,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:31,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:31,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:31,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:31,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:31,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:31,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:31,642 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:31,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:31,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:29:31,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:31,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:31,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:31,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:31,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:31,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:31,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:31,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:31,839 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:31,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:29:31,874 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:31,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:31,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:31,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:31,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:31,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:31,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:32,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:32,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:32,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:29:32,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:32,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:32,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:32,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:32,117 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:32,132 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:32,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:32,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:32,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:32,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:29:32,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:32,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:32,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:32,272 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:32,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:32,306 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:32,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:32,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:32,358 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 08:29:32,361 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:29:32,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:32,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:29:32,706 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:32,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:29:32,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:29:32,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:29:32,747 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:29:32,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:29:32,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:29:32,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:29:32,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:29:32,819 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 08:29:32,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:29:32,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:34,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:29:34,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:35,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:35,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:29:35,799 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 08:29:35,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:29:35,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:35,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:29:35,887 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 08:29:35,888 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:29:35,890 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:29:35,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:36,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:29:36,266 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:36,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:29:36,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:29:36,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:29:36,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:29:36,334 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:29:36,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:29:36,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:29:36,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:29:36,401 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 08:29:36,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:29:36,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:39,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:29:40,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:40,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:40,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:29:40,835 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 08:29:40,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:29:40,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:40,933 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:29:40,948 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 08:29:40,948 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:29:40,951 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 08:29:41,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:41,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 08:29:41,321 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:41,321 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 08:29:41,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 08:29:41,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 08:29:41,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 08:29:41,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 08:29:41,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 08:29:41,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 08:29:41,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:29:41,433 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 08:29:41,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:29:41,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:41,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:29:41,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:41,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:41,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:29:41,561 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 08:29:41,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:29:41,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:41,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:29:41,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:41,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:29:41,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:41,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:41,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:41,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:41,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:41,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:41,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:41,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:41,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:41,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:29:41,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:41,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:41,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:41,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:41,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:41,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:41,999 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:42,017 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:42,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:42,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:29:42,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:42,086 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:42,104 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:42,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:42,135 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:42,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:42,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:42,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:42,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:42,221 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:29:42,242 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:42,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:42,275 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:42,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:42,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:42,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:42,341 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:42,357 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:42,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:42,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:29:42,410 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:42,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:42,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:42,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:42,471 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:42,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:42,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:42,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:42,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:42,548 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:29:42,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:42,576 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:42,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:42,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:42,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:42,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:42,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:42,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:42,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:42,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:29:42,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:42,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:42,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:42,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:42,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:42,793 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:42,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:42,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:42,841 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 08:29:42,844 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 08:29:42,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:43,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 08:29:43,191 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:43,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 08:29:43,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 08:29:43,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 08:29:43,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 08:29:43,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 08:29:43,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 08:29:43,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 08:29:43,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:29:43,301 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 08:29:43,331 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:29:43,346 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:43,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:29:43,375 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:43,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:43,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:29:43,414 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 08:29:43,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:29:43,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:43,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:29:43,464 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:43,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:29:43,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:43,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:43,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:43,579 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:43,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:43,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:43,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:43,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:29:43,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:43,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:29:43,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:43,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:43,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:43,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:43,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:43,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:43,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:43,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:29:43,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:43,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:29:43,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:43,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:43,936 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:43,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:43,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:43,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:44,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:44,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:29:44,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:44,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:29:44,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:44,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:44,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:44,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:44,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:44,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:44,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:44,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:29:44,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:44,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:29:44,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:44,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:44,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:44,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:44,295 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:44,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:44,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:44,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:29:44,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:44,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:29:44,392 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:44,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:44,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:44,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:44,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:44,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:44,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:44,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:29:44,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:44,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:29:44,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:44,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:44,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:44,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:44,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:44,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:44,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:44,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:29:44,690 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 08:29:44,693 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:29:44,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:44,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:29:45,052 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:45,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:29:45,066 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:29:45,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:29:45,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:29:45,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:29:45,124 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:29:45,137 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:29:45,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:29:45,174 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 08:29:45,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:29:45,221 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:47,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:29:48,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:48,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:49,295 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:29:49,309 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 08:29:49,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:29:49,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:49,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:29:49,392 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 08:29:49,392 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:29:49,394 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:29:49,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 10895 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:29:49,716 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:29:49,779 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:49,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:29:49,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:29:49,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:29:49,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:29:49,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:29:49,859 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:29:49,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:29:49,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:29:49,907 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 08:29:49,948 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:29:49,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:52,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:29:52,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:52,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:53,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:29:53,217 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 08:29:53,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:29:53,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:53,275 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:29:53,290 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 08:29:53,290 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:29:53,292 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:29:53,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:54,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:29:54,538 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:54,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:29:54,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:29:54,599 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:29:54,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:29:54,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:29:54,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:29:54,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:29:54,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:29:54,729 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 08:29:54,757 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:29:54,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:56,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:29:57,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:57,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:29:57,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:29:57,730 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 08:29:57,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:29:57,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:29:57,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:29:57,793 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 08:29:57,793 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:29:57,796 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:29:57,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:29:58,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:29:58,286 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:29:58,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:29:58,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:29:58,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:29:58,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:29:58,357 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:29:58,372 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:29:58,395 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:29:58,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:29:58,440 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 08:29:58,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:29:58,489 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:30:03,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:30:04,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:30:04,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:30:05,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:30:05,084 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 08:30:05,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:30:05,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:30:05,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:30:05,170 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 08:30:05,171 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:30:05,173 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 08:30:05,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:30:05,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 08:30:05,559 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:30:05,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 08:30:05,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 08:30:05,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 08:30:05,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 08:30:05,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 08:30:05,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 08:30:05,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 08:30:05,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:30:05,684 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 08:30:05,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:30:05,732 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:30:05,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:30:05,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:30:05,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:30:05,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:30:05,819 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 08:30:05,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:30:05,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:30:05,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:30:05,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:30:05,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:30:05,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:30:05,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:30:05,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:30:05,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:30:06,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:30:06,048 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:30:06,073 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:30:06,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:30:06,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:30:06,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:30:06,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:30:06,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:30:06,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:30:06,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:30:06,242 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:30:06,262 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:30:06,280 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:30:06,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:30:06,318 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:30:06,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:30:06,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:30:06,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:30:06,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:30:06,405 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:30:06,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:30:06,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:30:06,459 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:30:06,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:30:06,492 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:30:06,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:30:06,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:30:06,543 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:30:06,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:30:06,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:30:06,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:30:06,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:30:06,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:30:06,642 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:30:06,658 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:30:06,676 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:30:06,691 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:30:06,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:30:06,724 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:30:06,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:30:06,757 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:30:06,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:30:06,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:30:06,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:30:06,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:30:06,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:30:06,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:30:06,880 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:30:06,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:30:06,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:30:06,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:30:06,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:30:06,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:30:06,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:30:06,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:30:07,010 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:30:07,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:30:07,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:30:07,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:30:07,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:30:07,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:30:07,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:30:07,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:30:07,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:30:07,177 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 08:30:07,177 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:30:07,180 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:30:07,278 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:30:07,520 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:30:07,591 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:30:07,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:30:07,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:30:07,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:30:07,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:30:07,658 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:30:07,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:30:07,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:30:07,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:08,105 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:30:08,107 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.929000139236. 2022-02-08 08:30:08,107 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:30:08,109 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:30:08,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:30:08,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:30:08,532 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:30:08,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:30:08,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:30:08,571 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:30:08,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:30:08,614 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:30:08,632 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:30:08,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:30:08,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:08,842 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:08,844 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:10,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:11,003 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:11,003 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:13,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:13,157 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:13,158 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:15,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:15,367 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:15,368 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:17,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:17,861 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:17,861 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:19,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:20,253 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:20,255 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:22,255 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:22,451 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:22,451 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:24,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:24,644 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:24,644 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:26,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:26,846 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:26,846 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:28,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:29,032 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:29,032 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:31,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:31,230 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:31,230 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:33,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:33,594 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:33,594 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:35,594 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:35,785 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:35,786 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:37,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:38,039 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:38,039 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:40,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:40,221 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:40,223 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:42,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:42,397 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:42,397 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:44,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:44,624 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:44,624 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:46,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:46,937 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:46,937 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:48,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:49,092 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:49,092 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 10899 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:30:51,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:51,246 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:51,246 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:53,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:53,391 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:53,391 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:55,392 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:55,546 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:55,556 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:57,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:57,707 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:30:57,707 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:30:59,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:30:59,851 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:30:59,852 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 51.7459998131. 2022-02-08 08:30:59,852 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:30:59,855 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:31:01,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:31:01,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:31:01,947 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:31:01,948 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:31:01,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:31:01,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:31:02,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:31:02,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:31:02,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:31:02,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:31:02,081 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:31:02,581 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:31:02,581 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.72800016403. 2022-02-08 08:31:02,582 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:31:02,584 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:31:03,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:31:03,380 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:31:03,460 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:31:03,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:31:03,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:31:03,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:31:03,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:31:03,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:31:03,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:31:03,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:31:03,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:31:04,069 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:31:04,069 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.4889998436. 2022-02-08 08:31:04,069 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:31:04,072 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:31:04,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:31:05,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:31:05,536 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:31:05,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:31:05,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:31:05,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:31:05,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:31:05,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:31:05,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:31:05,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:31:05,665 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:31:06,142 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:31:06,144 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.07400012016. 2022-02-08 08:31:06,144 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:31:06,145 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:31:07,104 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:31:07,357 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:31:07,438 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:31:07,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:31:07,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:31:07,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:31:07,487 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:31:07,505 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:31:07,517 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:31:07,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:31:07,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:31:07,999 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:31:08,000 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.85599994659. 2022-02-08 08:31:08,000 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:31:08,002 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:31:08,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:31:08,739 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:31:08,823 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:31:08,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:31:08,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:31:08,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:31:08,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:31:08,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:31:08,895 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:31:08,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:31:08,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:31:09,388 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:31:09,388 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.38800001144. 2022-02-08 08:31:09,388 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:31:09,390 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:31:10,224 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:31:10,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:31:10,894 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:31:10,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:31:10,982 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:31:11,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:31:11,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:31:11,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:31:11,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:31:11,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:31:11,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:31:11,858 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:31:11,858 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.47099995613. 2022-02-08 08:31:11,859 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:31:11,861 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:31:12,790 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:31:13,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:31:13,381 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:31:13,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:31:13,397 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:31:13,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:31:13,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:31:13,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:31:13,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:31:13,499 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:31:13,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:31:14,079 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:31:14,081 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.22000002861. ############################################################ # # RAM FootPrint: 4675.70703125 # ############################################################ 2022-02-08 08:31:14,111 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 08:31:14,111 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 118 seconds. Reason: Iteration:8 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 10913 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 10936 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:33:12,121 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 08:33:12,122 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 08:33:17,127 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:33:17,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:17,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:33:17,595 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:17,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:33:17,609 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:33:17,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:33:17,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:33:17,658 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:33:17,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:33:17,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:33:17,721 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:33:17,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:33:17,792 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['33276 memcached\n'], []) 2022-02-08 08:33:17,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 08:33:17,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:33:17,845 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['33276 memcached\n'], []) 2022-02-08 08:33:17,845 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:33:17,848 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:33:18,405 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:18,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:33:18,908 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:18,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:33:18,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:33:18,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:33:18,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:33:18,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:33:18,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:33:19,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:33:19,033 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:33:19,035 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:33:19,089 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['85382 memcached\n'], []) 2022-02-08 08:33:19,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 08:33:19,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:33:19,135 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['85382 memcached\n'], []) 2022-02-08 08:33:19,135 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:33:19,138 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:33:19,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:19,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:33:19,506 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:19,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:33:19,520 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:33:19,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:33:19,551 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:33:19,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:33:19,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:33:19,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:33:19,641 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:33:19,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:33:19,690 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['67996 memcached\n'], []) 2022-02-08 08:33:19,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 08:33:19,719 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:33:19,744 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['67996 memcached\n'], []) 2022-02-08 08:33:19,744 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:33:19,750 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:33:19,874 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:20,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:33:20,164 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:20,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:33:20,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:33:20,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:33:20,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:33:20,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:33:20,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:33:20,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:33:20,259 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:33:20,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:33:20,321 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['97509 memcached\n'], []) 2022-02-08 08:33:20,321 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 08:33:20,357 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:33:20,388 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['97509 memcached\n'], []) 2022-02-08 08:33:20,388 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:33:20,391 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:33:20,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:20,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:33:20,749 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:20,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:33:20,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:33:20,778 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:33:20,790 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:33:20,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:33:20,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:33:20,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:33:20,846 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:33:20,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:33:20,905 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['91343 memcached\n'], []) 2022-02-08 08:33:20,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 08:33:20,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:33:20,954 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['91343 memcached\n'], []) 2022-02-08 08:33:20,954 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:33:20,959 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:33:21,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:21,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:33:21,513 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:21,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:33:21,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:33:21,540 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:33:21,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:33:21,579 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:33:21,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:33:21,609 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:33:21,627 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:33:21,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:33:21,789 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['126313 memcached\n'], []) 2022-02-08 08:33:21,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 08:33:21,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:33:21,844 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['126313 memcached\n'], []) 2022-02-08 08:33:21,844 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:33:21,848 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:33:21,920 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:22,127 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:33:22,181 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:22,183 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:33:22,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:33:22,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:33:22,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:33:22,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:33:22,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:33:22,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:33:22,273 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:33:22,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:33:22,325 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['60261 memcached\n'], []) 2022-02-08 08:33:22,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 08:33:22,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:33:22,375 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['60261 memcached\n'], []) 2022-02-08 08:33:22,375 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:33:22,378 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:33:22,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:22,701 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:33:22,785 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:22,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:33:22,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:33:22,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:33:22,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:33:22,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:33:22,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:33:22,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:33:22,901 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:33:22,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:33:22,982 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['75164 memcached\n'], []) 2022-02-08 08:33:22,982 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 08:33:23,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:33:23,055 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['75164 memcached\n'], []) 2022-02-08 08:33:23,055 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:33:23,068 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:33:23,147 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:23,372 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:33:23,440 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:23,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:33:23,454 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:33:23,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:33:23,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:33:23,499 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:33:23,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:33:23,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:33:23,540 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:33:23,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:33:23,605 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['10596 memcached\n'], []) 2022-02-08 08:33:23,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 08:33:23,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:33:23,657 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['10596 memcached\n'], []) 2022-02-08 08:33:23,657 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:33:23,660 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:33:23,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:24,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:33:24,092 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:24,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:33:24,114 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:33:24,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:33:24,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:33:24,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:33:24,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:33:24,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:33:24,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:33:24,234 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 08:33:24,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:33:24,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:28,897 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:33:29,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:33:30,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:33:30,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:33:30,502 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 08:33:30,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:33:30,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:30,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:33:30,591 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 08:33:30,592 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:33:30,594 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:33:30,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:30,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:33:30,946 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:30,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:33:30,961 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:33:30,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:33:30,987 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:33:31,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:33:31,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:33:31,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:33:31,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:33:31,056 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 08:33:31,094 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:33:31,107 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:34,443 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:33:35,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:33:35,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:33:35,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:33:35,944 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 08:33:35,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:33:35,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:36,025 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:33:36,039 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 08:33:36,039 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:33:36,043 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 08:33:36,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:36,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 08:33:36,459 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:36,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 08:33:36,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 08:33:36,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 08:33:36,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 08:33:36,520 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 08:33:36,533 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 08:33:36,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 08:33:36,563 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:33:36,581 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 08:33:36,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:33:36,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:36,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:33:36,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:33:36,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:33:36,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:33:36,736 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 08:33:36,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:33:36,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:36,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:33:36,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:36,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:33:36,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:36,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:36,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:36,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:36,936 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:36,961 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:36,987 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:37,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:37,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:37,063 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:33:37,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:37,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:37,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:37,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:37,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:37,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:37,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:37,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:37,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:37,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:33:37,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:37,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:37,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:37,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:37,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:37,364 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:37,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:37,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:37,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:37,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:33:37,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:37,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:37,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:37,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:37,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:37,551 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:37,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:37,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:37,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:37,619 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:33:37,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:37,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:37,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:37,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:37,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:37,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:37,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:37,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:37,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:37,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:33:37,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:37,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:37,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:37,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:37,877 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:37,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:37,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:37,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:37,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:37,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:33:37,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:38,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:38,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:38,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:38,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:38,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:38,096 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:38,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:38,132 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 08:33:38,134 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:33:38,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:38,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:33:39,036 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:39,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:33:39,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:33:39,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:33:39,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:33:39,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:33:39,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:33:39,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:33:39,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:33:39,598 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 08:33:39,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:33:39,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:40,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:33:40,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:33:41,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:33:41,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:33:41,619 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 08:33:41,619 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:33:41,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:41,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:33:41,713 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 08:33:41,713 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:33:41,716 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 08:33:41,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:42,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 08:33:42,089 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:42,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 08:33:42,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 08:33:42,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 08:33:42,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 08:33:42,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 08:33:42,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 08:33:42,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 08:33:42,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:33:42,209 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 08:33:42,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:33:42,255 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:42,270 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:33:42,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:33:42,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:33:42,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:33:42,325 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 08:33:42,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:33:42,339 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:42,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:33:42,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:42,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:33:42,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:42,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:42,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:42,492 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:42,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:42,540 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:42,571 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:42,599 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:42,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:42,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:33:42,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:42,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:42,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:42,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:42,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:42,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:42,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:42,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:42,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:42,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:33:42,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:42,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:42,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:42,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:42,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:42,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:42,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:42,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:43,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:43,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:33:43,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:43,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:43,066 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:43,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:43,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:43,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:43,137 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:43,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:43,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:43,183 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:33:43,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:43,214 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:43,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:43,244 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:43,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:43,278 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:43,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:43,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:43,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:43,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:33:43,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:43,372 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:43,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:43,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:43,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:43,436 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:43,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:43,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:43,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:43,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:33:43,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:43,540 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:43,556 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:43,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:43,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:43,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:43,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:43,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:43,654 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 08:33:43,657 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:33:43,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:43,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:33:44,017 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:44,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:33:44,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:33:44,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:33:44,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:33:44,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:33:44,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:33:44,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:33:44,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:33:44,213 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 08:33:44,255 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:33:44,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:48,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:33:48,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:33:49,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 10951 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:33:49,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:33:49,752 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 08:33:49,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:33:49,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:49,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:33:49,845 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 08:33:49,846 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:33:49,849 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:33:49,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:50,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:33:50,289 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:50,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:33:50,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:33:50,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:33:50,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:33:50,364 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:33:50,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:33:50,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:33:50,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:33:50,436 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 08:33:50,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:33:50,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:53,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:33:54,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:33:54,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:33:55,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:33:55,029 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 08:33:55,030 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:33:55,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:55,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:33:55,141 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 08:33:55,141 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:33:55,144 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 08:33:55,252 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:55,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 08:33:55,582 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:55,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 08:33:55,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 08:33:55,619 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 08:33:55,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 08:33:55,660 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 08:33:55,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 08:33:55,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 08:33:55,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:33:55,736 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 08:33:55,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:33:55,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:55,806 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:33:55,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:33:55,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:33:55,859 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:33:55,881 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 08:33:55,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:33:55,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:55,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:33:55,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:55,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:33:56,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:56,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:56,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:56,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:56,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:56,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:56,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:56,232 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:56,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:56,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:33:56,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:56,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:56,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:56,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:56,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:56,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:56,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:56,459 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:56,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:56,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:33:56,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:56,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:56,579 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:56,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:56,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:56,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:56,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:56,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:56,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:56,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:33:56,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:56,775 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:56,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:56,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:56,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:56,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:56,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:56,895 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:56,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:56,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:33:56,961 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:56,984 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:57,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:57,025 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:57,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:57,066 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:57,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:57,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:57,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:57,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:33:57,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:57,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:57,227 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:57,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:57,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:57,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:57,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:57,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:57,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:57,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:33:57,390 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:57,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:57,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:57,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:57,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:57,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:57,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:57,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:57,562 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 08:33:57,565 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 08:33:57,665 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:57,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 08:33:57,951 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:33:57,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 08:33:57,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 08:33:57,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 08:33:57,994 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 08:33:58,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 08:33:58,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 08:33:58,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 08:33:58,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:33:58,075 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 08:33:58,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:33:58,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:58,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:33:58,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:33:58,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:33:58,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:33:58,204 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 08:33:58,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:33:58,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:33:58,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:33:58,252 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:58,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:33:58,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:58,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:58,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:58,392 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:58,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:58,443 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:58,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:58,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:33:58,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:58,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:33:58,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:58,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:58,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:58,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:58,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:58,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:58,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:58,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:33:58,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:58,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:33:58,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:58,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:58,790 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:58,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:58,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:58,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:58,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:58,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:33:58,903 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:58,920 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:33:58,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:58,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:58,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:58,987 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:59,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:59,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:59,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:59,058 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:33:59,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:59,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:33:59,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:59,127 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:59,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:59,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:59,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:59,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:59,206 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:59,221 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:33:59,239 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:59,255 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:33:59,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:59,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:59,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:59,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:59,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:59,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:59,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:59,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:33:59,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:59,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:33:59,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:59,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:59,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:59,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:59,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:59,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:59,538 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:59,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:33:59,573 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 08:33:59,578 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:33:59,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:33:59,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:34:00,039 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:34:00,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:34:00,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:34:00,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:34:00,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:34:00,104 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:34:00,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:34:00,137 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:34:00,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:34:00,174 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 08:34:00,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:34:00,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:34:03,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:34:03,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:34:04,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:34:04,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:34:04,859 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 08:34:04,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:34:04,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:34:04,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:34:04,954 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 08:34:04,954 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:34:04,957 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:34:05,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:34:05,306 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:34:05,382 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:34:05,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:34:05,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:34:05,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:34:05,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:34:05,459 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:34:05,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:34:05,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:34:05,538 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:34:05,556 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 08:34:05,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:34:05,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:34:06,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:34:06,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:34:07,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:34:07,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:34:07,546 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 08:34:07,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:34:07,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:34:07,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:34:07,621 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 08:34:07,621 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:34:07,624 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:34:08,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:34:08,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:34:08,529 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:34:08,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:34:08,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:34:08,566 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:34:08,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:34:08,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:34:08,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:34:08,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:34:08,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:34:08,688 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 08:34:08,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:34:08,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:34:10,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:34:11,331 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:34:11,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:34:11,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:34:11,959 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 08:34:11,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:34:11,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:34:12,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:34:12,055 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 08:34:12,055 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:34:12,056 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:34:12,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:34:12,487 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:34:12,607 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:34:12,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:34:12,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:34:12,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:34:12,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:34:12,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:34:12,729 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:34:12,747 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:34:12,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:34:12,788 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 08:34:12,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:34:12,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:34:17,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:34:17,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:34:18,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:34:19,117 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:34:19,144 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 08:34:19,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:34:19,227 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:34:19,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:34:19,328 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 08:34:19,328 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:34:19,331 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 08:34:19,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:34:19,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 08:34:19,736 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:34:19,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 08:34:19,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 08:34:19,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 08:34:19,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 08:34:19,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 08:34:19,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 08:34:19,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 08:34:19,852 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:34:19,871 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 08:34:19,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:34:19,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:34:19,936 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:34:19,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:34:19,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:34:19,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:34:19,997 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 08:34:19,999 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:34:20,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:34:20,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:34:20,048 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:34:20,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:34:20,107 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:34:20,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:34:20,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:34:20,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:34:20,234 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:34:20,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:34:20,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:34:20,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:34:20,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:34:20,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:34:20,471 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:34:20,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:34:20,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:34:20,538 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:34:20,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:34:20,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:34:20,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:34:20,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:34:20,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:34:20,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:34:20,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:34:20,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:34:20,724 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:34:20,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:34:20,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:34:20,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:34:20,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:34:20,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:34:20,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:34:20,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:34:20,877 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:34:20,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:34:20,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:34:20,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:34:20,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:34:20,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:34:20,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:34:21,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:34:21,020 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:34:21,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:34:21,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:34:21,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:34:21,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:34:21,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:34:21,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:34:21,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:34:21,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:34:21,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:34:21,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:34:21,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:34:21,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:34:21,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:34:21,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:34:21,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:34:21,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:34:21,306 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:34:21,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:34:21,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:34:21,357 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:34:21,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:34:21,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:34:21,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:34:21,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:34:21,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:34:21,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:34:21,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:34:21,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:34:21,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:34:21,546 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 08:34:21,546 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:34:21,549 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:34:21,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:34:21,897 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:34:21,970 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:34:21,971 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:34:21,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:34:22,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:34:22,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:34:22,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:34:22,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:34:22,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:34:22,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:34:22,535 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:34:22,535 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.989000082016. 2022-02-08 08:34:22,536 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:34:22,538 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:34:22,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:34:22,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:34:23,029 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:34:23,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:34:23,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:34:23,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:34:23,086 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:34:23,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:34:23,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:34:23,147 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:34:23,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:34:23,329 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:34:23,331 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:34:25,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:34:25,533 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:34:25,533 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:34:27,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:34:27,710 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:34:27,710 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:34:29,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:34:29,924 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:34:29,924 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:34:31,926 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:34:32,191 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:34:32,191 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:34:34,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:34:34,398 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:34:34,398 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:34:36,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:34:36,627 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:34:36,627 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:34:38,632 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:34:38,849 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:34:38,849 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:34:40,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:34:41,039 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:34:41,039 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:34:43,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:34:43,206 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:34:43,206 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:34:45,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:34:45,378 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:34:45,378 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:34:47,380 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:34:47,571 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:34:47,572 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:34:49,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 10956 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:34:49,851 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:34:49,852 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:34:51,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:34:52,190 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:34:52,191 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:34:54,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:34:54,346 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:34:54,348 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:34:56,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:34:56,612 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:34:56,612 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:34:58,612 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:34:58,785 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:34:58,785 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:35:00,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:35:00,938 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:35:00,940 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:35:02,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:35:03,084 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:35:03,085 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 40.5490000248. 2022-02-08 08:35:03,085 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:35:03,089 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:35:03,864 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:35:04,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:35:04,275 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:35:04,275 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:35:04,290 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:35:04,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:35:04,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:35:04,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:35:04,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:35:04,392 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:35:04,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:35:04,874 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:35:04,874 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.78900003433. 2022-02-08 08:35:04,875 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:35:04,877 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:35:05,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:35:05,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:35:05,865 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:35:05,867 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:35:05,880 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:35:05,897 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:35:05,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:35:05,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:35:05,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:35:05,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:35:05,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:35:06,539 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:35:06,539 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.66499996185. 2022-02-08 08:35:06,539 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:35:06,542 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:35:07,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:35:08,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:35:08,743 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:35:08,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:35:08,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:35:08,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:35:09,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:35:09,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:35:09,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:35:09,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:35:09,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:35:10,319 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:35:10,319 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 08:35:12,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:35:12,483 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:35:12,486 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.94600009918. 2022-02-08 08:35:12,486 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:35:12,489 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:35:13,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:35:13,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:35:13,996 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:35:13,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:35:14,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:35:14,063 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:35:14,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:35:14,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:35:14,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:35:14,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:35:14,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:35:14,976 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:35:14,976 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.49099993706. 2022-02-08 08:35:14,976 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:35:14,979 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:35:15,599 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:35:15,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:35:15,947 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:35:15,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:35:15,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:35:15,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:35:15,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:35:16,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:35:16,030 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:35:16,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:35:16,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:35:16,667 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:35:16,667 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.69099998474. 2022-02-08 08:35:16,667 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:35:16,670 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:35:17,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:35:17,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:35:17,569 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:35:17,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:35:17,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:35:17,609 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:35:17,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:35:17,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:35:17,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:35:17,688 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:35:17,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:35:18,308 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:35:18,308 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.6400001049. 2022-02-08 08:35:18,309 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:35:18,311 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:35:18,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:35:19,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:35:19,492 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:35:19,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:35:19,507 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:35:19,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:35:19,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:35:19,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:35:19,581 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:35:19,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:35:19,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:35:20,599 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:35:20,599 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.29099988937. ############################################################ # # RAM FootPrint: 4697.5 # ############################################################ 2022-02-08 08:35:20,644 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 08:35:20,644 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 109 seconds. Reason: Iteration:9 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 10975 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 10996 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:37:09,654 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 08:37:09,654 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 08:37:14,657 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:37:14,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:15,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:37:15,122 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:15,124 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:37:15,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:37:15,168 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:37:15,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:37:15,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:37:15,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:37:15,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:37:15,267 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:37:15,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:37:15,338 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['34007 memcached\n'], []) 2022-02-08 08:37:15,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 08:37:15,392 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:37:15,418 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['34007 memcached\n'], []) 2022-02-08 08:37:15,420 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:37:15,423 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:37:15,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:16,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:37:16,338 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:16,339 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:37:16,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:37:16,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:37:16,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:37:16,405 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:37:16,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:37:16,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:37:16,461 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:37:16,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:37:16,522 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['86388 memcached\n'], []) 2022-02-08 08:37:16,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 08:37:16,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:37:16,573 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['86388 memcached\n'], []) 2022-02-08 08:37:16,573 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:37:16,576 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:37:16,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:16,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:37:16,961 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:16,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:37:16,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:37:16,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:37:17,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:37:17,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:37:17,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:37:17,063 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:37:17,079 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:37:17,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:37:17,128 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['68826 memcached\n'], []) 2022-02-08 08:37:17,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 08:37:17,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:37:17,183 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['68826 memcached\n'], []) 2022-02-08 08:37:17,184 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:37:17,187 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:37:17,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:17,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:37:17,546 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:17,548 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:37:17,561 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:37:17,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:37:17,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:37:17,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:37:17,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:37:17,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:37:17,645 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:37:17,647 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:37:17,710 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['98237 memcached\n'], []) 2022-02-08 08:37:17,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 08:37:17,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:37:17,766 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['98237 memcached\n'], []) 2022-02-08 08:37:17,766 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:37:17,769 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:37:17,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:18,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:37:18,128 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:18,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:37:18,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:37:18,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:37:18,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:37:18,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:37:18,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:37:18,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:37:18,234 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:37:18,234 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:37:18,290 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['92073 memcached\n'], []) 2022-02-08 08:37:18,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 08:37:18,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:37:18,357 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['92073 memcached\n'], []) 2022-02-08 08:37:18,357 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:37:18,359 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:37:18,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:18,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:37:18,726 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:18,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:37:18,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:37:18,752 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:37:18,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:37:18,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:37:18,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:37:18,806 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:37:18,822 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:37:18,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:37:18,963 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['127041 memcached\n'], []) 2022-02-08 08:37:18,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 08:37:18,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:37:19,007 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['127041 memcached\n'], []) 2022-02-08 08:37:19,007 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:37:19,010 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:37:19,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:19,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:37:19,394 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:19,395 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:37:19,410 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:37:19,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:37:19,436 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:37:19,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:37:19,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:37:19,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:37:19,493 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:37:19,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:37:19,543 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['61001 memcached\n'], []) 2022-02-08 08:37:19,543 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 08:37:19,566 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:37:19,588 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['61001 memcached\n'], []) 2022-02-08 08:37:19,588 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:37:19,592 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:37:19,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:19,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:37:19,990 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:19,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:37:20,005 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:37:20,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:37:20,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:37:20,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:37:20,114 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:37:20,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:37:20,144 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:37:20,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:37:20,213 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['75894 memcached\n'], []) 2022-02-08 08:37:20,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 08:37:20,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:37:20,280 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['75894 memcached\n'], []) 2022-02-08 08:37:20,282 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:37:20,283 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:37:20,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:20,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:37:20,677 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:20,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:37:20,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:37:20,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:37:20,719 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:37:20,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:37:20,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:37:20,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:37:20,786 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:37:20,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:37:20,846 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['11334 memcached\n'], []) 2022-02-08 08:37:20,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 08:37:20,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:37:20,891 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['11334 memcached\n'], []) 2022-02-08 08:37:20,891 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:37:20,894 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:37:20,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:21,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:37:21,247 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:21,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:37:21,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:37:21,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:37:21,295 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:37:21,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:37:21,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:37:21,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:37:21,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:37:21,375 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 08:37:21,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:37:21,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:22,581 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:37:23,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:23,762 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:24,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:37:24,165 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 08:37:24,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:37:24,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:24,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:37:24,263 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 08:37:24,265 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:37:24,266 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:37:24,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:24,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:37:24,674 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:24,676 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:37:24,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:37:24,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:37:24,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:37:24,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:37:24,752 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:37:24,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:37:24,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:37:24,796 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 08:37:24,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:37:24,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:27,928 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:37:28,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:29,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:29,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:37:29,444 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 08:37:29,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:37:29,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:29,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:37:29,546 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 08:37:29,546 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:37:29,549 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 08:37:29,655 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:29,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 08:37:29,953 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:29,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 08:37:29,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 08:37:29,999 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 08:37:30,017 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 08:37:30,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 08:37:30,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 08:37:30,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 08:37:30,089 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:37:30,105 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 08:37:30,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:37:30,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:30,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:37:30,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:30,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:30,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:37:30,316 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 08:37:30,318 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:37:30,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:30,346 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:37:30,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:30,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:37:30,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:30,436 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:30,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:30,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:30,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:30,543 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:30,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:30,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:30,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:30,647 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:37:30,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:30,686 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:30,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:30,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:30,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:30,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:30,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:30,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:30,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:30,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:37:30,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:30,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:30,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:30,908 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:30,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:30,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:30,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:30,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:30,999 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:31,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:37:31,035 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:31,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:31,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:31,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:31,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:31,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:31,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:31,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:31,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:31,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:37:31,216 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:31,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:31,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:31,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:31,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:31,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:31,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:31,334 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:31,349 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:31,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:37:31,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:31,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:31,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:31,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:31,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:31,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:31,484 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:31,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:31,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:31,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:37:31,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:31,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:31,594 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:31,612 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:31,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:31,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:31,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:31,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:31,710 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 08:37:31,713 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:37:31,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:32,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:37:32,849 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:32,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:37:32,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:37:32,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:37:33,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:37:33,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:37:33,339 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:37:33,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:37:33,581 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:37:33,737 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 08:37:33,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:37:33,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:34,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:37:35,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:35,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:36,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:37:36,226 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 08:37:36,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:37:36,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:36,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:37:36,538 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 08:37:36,538 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:37:36,539 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 08:37:36,647 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:36,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 08:37:36,950 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:36,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 08:37:36,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 08:37:36,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 08:37:36,994 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 08:37:37,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 08:37:37,030 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 08:37:37,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 08:37:37,061 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:37:37,076 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 08:37:37,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:37:37,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:37,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:37:37,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:37,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:37,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:37:37,206 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 08:37:37,206 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:37:37,221 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:37,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:37:37,252 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:37,280 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:37:37,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:37,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:37,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:37,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:37,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:37,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:37,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:37,492 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:37,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:37,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:37:37,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:37,576 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:37,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:37,612 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:37,630 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:37,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:37,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:37,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:37,706 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:37,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:37:37,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:37,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:37,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:37,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:37,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:37,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:37,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:37,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:37,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:37,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:37:37,926 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:37,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:37,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:37,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:37,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:38,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:38,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:38,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:38,061 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:38,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:37:38,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:38,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:38,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:38,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:38,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:38,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:38,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:38,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:38,237 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:38,255 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:37:38,272 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:38,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:38,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:38,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:38,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:38,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:38,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:38,385 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:38,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:38,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:37:38,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:38,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:38,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:38,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:38,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:38,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:38,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:38,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:38,585 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 08:37:38,588 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:37:38,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:38,874 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:37:38,934 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:38,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:37:38,948 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:37:38,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:37:38,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:37:38,994 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:37:39,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:37:39,020 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:37:39,035 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:37:39,049 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 08:37:39,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:37:39,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:39,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:37:40,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:40,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:41,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:37:41,299 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 08:37:41,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:37:41,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:41,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:37:41,375 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 08:37:41,375 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:37:41,378 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:37:41,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:41,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:37:41,757 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:41,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:37:41,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:37:41,793 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:37:41,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:37:41,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:37:41,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:37:41,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:37:41,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:37:41,901 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 08:37:41,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:37:41,956 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:46,936 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:37:47,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:48,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:48,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:37:48,414 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 08:37:48,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:37:48,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:48,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:37:48,519 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 08:37:48,519 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:37:48,523 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 08:37:48,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:48,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 08:37:48,892 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:48,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 08:37:48,908 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 08:37:48,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 08:37:48,936 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 08:37:48,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 08:37:48,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 08:37:48,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 08:37:49,002 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:37:49,019 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 08:37:49,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:37:49,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:49,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:37:49,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:49,114 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:49,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:37:49,141 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 08:37:49,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:37:49,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:49,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:37:49,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:49,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:37:49,239 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:49,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:49,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:49,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:49,349 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:49,380 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:49,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:49,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:49,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:49,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:37:49,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:49,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:49,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:49,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:49,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:49,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:49,596 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:49,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:49,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:49,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 5 queries submitted, 0 failed, 11016 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:37:49,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:49,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:49,698 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:49,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:49,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:49,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:49,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:49,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:49,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:49,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:37:49,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:49,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:49,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:49,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:49,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:49,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:49,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:49,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:49,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:49,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:37:50,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:50,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:50,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:50,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:50,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:50,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:50,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:50,132 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:50,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:50,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:37:50,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:50,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:50,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:50,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:50,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:50,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:50,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:50,318 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:50,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:50,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:37:50,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:50,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:50,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:50,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:50,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:50,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:50,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:50,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:50,517 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 08:37:50,520 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 08:37:50,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:50,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 08:37:50,924 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:50,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 08:37:50,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 08:37:50,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 08:37:50,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 08:37:50,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 08:37:51,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 08:37:51,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 08:37:51,030 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:37:51,046 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 08:37:51,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:37:51,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:51,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:37:51,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:51,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:51,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:37:51,180 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 08:37:51,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:37:51,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:51,214 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:37:51,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:51,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:37:51,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:51,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:51,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:51,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:51,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:51,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:51,482 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:51,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:37:51,538 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:51,556 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:37:51,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:51,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:51,614 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:51,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:51,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:51,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:51,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:51,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:37:51,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:51,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:37:51,757 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:51,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:51,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:51,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:51,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:51,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:51,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:51,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:37:51,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:51,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:37:51,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:51,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:51,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:51,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:52,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:52,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:52,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:52,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:37:52,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:52,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:37:52,114 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:52,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:52,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:52,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:52,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:52,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:52,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:52,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:37:52,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:52,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:37:52,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:52,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:52,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:52,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:52,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:52,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:52,375 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:52,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:37:52,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:52,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:37:52,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:52,464 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:52,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:52,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:52,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:52,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:52,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:52,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:37:52,591 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 08:37:52,594 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:37:52,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:37:53,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:37:54,007 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:37:54,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:37:54,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:37:54,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:37:54,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:37:54,642 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:37:54,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:37:54,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:37:54,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:37:55,075 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 08:37:55,104 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:37:55,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:58,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:37:58,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:59,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:37:59,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:37:59,720 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 08:37:59,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:37:59,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:37:59,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:37:59,803 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 08:37:59,803 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:37:59,805 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:37:59,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:38:00,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:38:00,232 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:38:00,232 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:38:00,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:38:00,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:38:00,280 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:38:00,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:38:00,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:38:00,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:38:00,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:38:00,371 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 08:38:00,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:38:00,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:38:02,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:38:03,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:38:04,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:38:04,459 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:38:04,489 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 08:38:04,489 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:38:04,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:38:04,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:38:04,575 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 08:38:04,576 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:38:04,579 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:38:05,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:38:05,410 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:38:05,539 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:38:05,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:38:05,556 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:38:05,596 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:38:05,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:38:05,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:38:05,670 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:38:05,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:38:05,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:38:05,729 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 08:38:05,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:38:05,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:38:08,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:38:09,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:38:09,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:38:10,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:38:10,131 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 08:38:10,132 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:38:10,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:38:10,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:38:10,229 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 08:38:10,230 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:38:10,233 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:38:10,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:38:10,579 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:38:10,654 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:38:10,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:38:10,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:38:10,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:38:10,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:38:10,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:38:10,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:38:10,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:38:10,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:38:10,815 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 08:38:10,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:38:10,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:38:14,658 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:38:15,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:38:15,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:38:16,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:38:16,434 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 08:38:16,436 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:38:16,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:38:16,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:38:16,532 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 08:38:16,532 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:38:16,535 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 08:38:16,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:38:16,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 08:38:16,917 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:38:16,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 08:38:16,933 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 08:38:16,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 08:38:16,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 08:38:16,982 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 08:38:16,994 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 08:38:17,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 08:38:17,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:38:17,042 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 08:38:17,073 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:38:17,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:38:17,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:38:17,124 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:38:17,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:38:17,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:38:17,168 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 08:38:17,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:38:17,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:38:17,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:38:17,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:38:17,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:38:17,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:38:17,295 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:38:17,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:38:17,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:38:17,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:38:17,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:38:17,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:38:17,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:38:17,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:38:17,499 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:38:17,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:38:17,538 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:38:17,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:38:17,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:38:17,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:38:17,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:38:17,647 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:38:17,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:38:17,691 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:38:17,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:38:17,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:38:17,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:38:17,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:38:17,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:38:17,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:38:17,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:38:17,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:38:17,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:38:17,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:38:17,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:38:17,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:38:17,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:38:17,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:38:17,961 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:38:17,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:38:17,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:38:18,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:38:18,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:38:18,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:38:18,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:38:18,086 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:38:18,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:38:18,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:38:18,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:38:18,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:38:18,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:38:18,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:38:18,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:38:18,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:38:18,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:38:18,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:38:18,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:38:18,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:38:18,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:38:18,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:38:18,349 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:38:18,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:38:18,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:38:18,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:38:18,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:38:18,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:38:18,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:38:18,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:38:18,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:38:18,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:38:18,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:38:18,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:38:18,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:38:18,582 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 08:38:18,584 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:38:18,585 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:38:18,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:38:18,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:38:19,019 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:38:19,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:38:19,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:38:19,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:38:19,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:38:19,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:38:19,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:38:19,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:38:19,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:19,594 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:38:19,594 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.00999999046. 2022-02-08 08:38:19,594 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:38:19,595 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:38:19,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:38:20,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:38:20,295 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:38:20,295 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:38:20,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:38:20,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:38:20,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:38:20,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:38:20,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:38:20,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:38:20,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:20,584 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:38:20,584 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:38:22,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:22,746 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:38:22,747 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:38:24,747 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:24,898 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:38:24,898 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:38:26,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:27,085 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:38:27,085 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:38:29,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:29,272 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:38:29,272 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:38:31,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:31,480 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:38:31,482 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:38:33,482 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:33,803 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:38:33,803 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:38:35,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:35,969 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:38:35,970 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:38:37,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:38,187 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:38:38,187 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:38:40,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:40,479 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:38:40,480 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:38:42,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:43,009 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:38:43,010 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:38:45,010 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:45,296 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:38:45,298 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:38:47,298 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:47,506 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:38:47,506 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:38:49,507 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 11021 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:38:49,720 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:38:49,721 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:38:51,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:51,960 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:38:51,960 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:38:53,961 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:54,147 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:38:54,148 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:38:56,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:56,345 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:38:56,345 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 36.751999855. 2022-02-08 08:38:56,346 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:38:56,348 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:38:57,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:38:58,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:38:58,384 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:38:58,385 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:38:58,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:38:58,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:38:58,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:38:58,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:38:58,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:38:58,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:38:58,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:38:59,226 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:38:59,226 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 08:39:01,227 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:39:01,398 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:39:01,398 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 08:39:03,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:39:03,572 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:39:03,572 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 08:39:05,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:39:05,746 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:39:05,747 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 08:39:07,747 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:39:07,970 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:39:07,970 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 11.625. 2022-02-08 08:39:07,970 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:39:07,973 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:39:08,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:39:09,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:39:09,407 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:39:09,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:39:09,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:39:09,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:39:09,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:39:09,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:39:09,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:39:09,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:39:09,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:39:10,213 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:39:10,214 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.24399995804. 2022-02-08 08:39:10,214 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:39:10,217 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:39:10,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:39:11,426 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:39:11,575 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:39:11,576 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:39:11,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:39:11,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:39:11,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:39:11,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:39:11,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:39:11,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:39:11,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:39:12,219 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:39:12,220 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 08:39:14,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:39:14,401 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:39:14,401 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.18600010872. 2022-02-08 08:39:14,401 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:39:14,403 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:39:15,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:39:15,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:39:15,969 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:39:15,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:39:15,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:39:16,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:39:16,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:39:16,086 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:39:16,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:39:16,135 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:39:16,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:39:16,792 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:39:16,793 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 08:39:18,793 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:39:19,052 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:39:19,052 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 08:39:21,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:39:21,250 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:39:21,250 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 08:39:23,252 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:39:23,513 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:39:23,515 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 9.11400008202. 2022-02-08 08:39:23,515 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:39:23,516 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:39:24,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:39:24,806 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:39:24,996 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:39:24,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:39:25,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:39:25,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:39:25,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:39:25,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:39:25,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:39:25,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:39:25,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:39:25,946 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:39:25,946 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.43099999428. 2022-02-08 08:39:25,946 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:39:25,948 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:39:26,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:39:27,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:39:27,473 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:39:27,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:39:27,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:39:27,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:39:27,571 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:39:27,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:39:27,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:39:27,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:39:27,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:39:28,483 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:39:28,484 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.53899979591. 2022-02-08 08:39:28,484 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:39:28,486 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:39:29,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:39:29,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:39:30,191 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:39:30,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:39:30,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:39:30,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:39:30,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:39:30,290 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:39:30,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:39:30,410 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:39:30,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:39:32,101 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:39:32,101 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.61600017548. ############################################################ # # RAM FootPrint: 4626.31640625 # ############################################################ 2022-02-08 08:39:32,145 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 08:39:32,145 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 60 seconds. Reason: Iteration:10 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 11033 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:40:32,151 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 08:40:32,151 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 08:40:37,154 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:40:37,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:40:38,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:40:38,549 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:40:38,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:40:38,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:40:38,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:40:38,706 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:40:38,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:40:38,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:40:38,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:40:38,969 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:40:38,969 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:40:39,040 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['34733 memcached\n'], []) 2022-02-08 08:40:39,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 08:40:39,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:40:39,151 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['34733 memcached\n'], []) 2022-02-08 08:40:39,151 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:40:39,154 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:40:40,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:40:40,540 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:40:40,733 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:40:40,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:40:40,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:40:40,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:40:40,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:40:40,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:40:40,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:40:40,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:40:40,970 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:40:40,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:40:41,036 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['87328 memcached\n'], []) 2022-02-08 08:40:41,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 08:40:41,084 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:40:41,108 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['87328 memcached\n'], []) 2022-02-08 08:40:41,109 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:40:41,111 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:40:41,706 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:40:42,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:40:42,230 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:40:42,232 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:40:42,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:40:42,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:40:42,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:40:42,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:40:42,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:40:42,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:40:42,545 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:40:42,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:40:42,641 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['69644 memcached\n'], []) 2022-02-08 08:40:42,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 08:40:42,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:40:42,710 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['69644 memcached\n'], []) 2022-02-08 08:40:42,711 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:40:42,714 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:40:43,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:40:44,252 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:40:44,358 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:40:44,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:40:44,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:40:44,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:40:44,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:40:44,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:40:44,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:40:44,551 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:40:44,579 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:40:44,579 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:40:44,648 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['98959 memcached\n'], []) 2022-02-08 08:40:44,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 08:40:44,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:40:44,706 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['98959 memcached\n'], []) 2022-02-08 08:40:44,707 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:40:44,710 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:40:46,331 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:40:46,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:40:46,828 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:40:46,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:40:46,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:40:46,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:40:46,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:40:46,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:40:46,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:40:46,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:40:46,967 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:40:46,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:40:47,039 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['92810 memcached\n'], []) 2022-02-08 08:40:47,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 08:40:47,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:40:47,086 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['92810 memcached\n'], []) 2022-02-08 08:40:47,086 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:40:47,088 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:40:48,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:40:48,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:40:48,723 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:40:48,724 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:40:48,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:40:48,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:40:48,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:40:48,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:40:48,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:40:48,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:40:48,867 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:40:48,867 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:40:48,944 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['127766 memcached\n'], []) 2022-02-08 08:40:48,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 08:40:48,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:40:49,012 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['127766 memcached\n'], []) 2022-02-08 08:40:49,012 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:40:49,013 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 5 queries submitted, 0 failed, 11057 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:40:49,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:40:50,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:40:50,506 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:40:50,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:40:50,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:40:50,540 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:40:50,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:40:50,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:40:50,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:40:50,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:40:50,641 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:40:50,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:40:50,694 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['61723 memcached\n'], []) 2022-02-08 08:40:50,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 08:40:50,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:40:50,744 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['61723 memcached\n'], []) 2022-02-08 08:40:50,744 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:40:50,746 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:40:51,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:40:51,956 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:40:52,053 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:40:52,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:40:52,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:40:52,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:40:52,114 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:40:52,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:40:52,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:40:52,183 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:40:52,198 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:40:52,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:40:52,263 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['76619 memcached\n'], []) 2022-02-08 08:40:52,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 08:40:52,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:40:52,322 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['76619 memcached\n'], []) 2022-02-08 08:40:52,322 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:40:52,325 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:40:53,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:40:53,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:40:53,782 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:40:53,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:40:53,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:40:53,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:40:53,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:40:53,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:40:53,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:40:53,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:40:53,911 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:40:53,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:40:53,984 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['12061 memcached\n'], []) 2022-02-08 08:40:53,984 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 08:40:54,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:40:54,036 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['12061 memcached\n'], []) 2022-02-08 08:40:54,038 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:40:54,039 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:40:54,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:40:54,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:40:54,398 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:40:54,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:40:54,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:40:54,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:40:54,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:40:54,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:40:54,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:40:54,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:40:54,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:40:54,509 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 08:40:54,533 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:40:54,551 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:00,729 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:41:01,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:01,948 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:02,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:41:02,372 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 08:41:02,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:41:02,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:02,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:41:02,463 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 08:41:02,464 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:41:02,466 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:41:02,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:41:02,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:41:02,823 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:41:02,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:41:02,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:41:02,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:41:02,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:41:02,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:41:02,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:41:02,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:41:02,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:41:02,931 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 08:41:02,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:41:02,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:09,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:41:10,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:10,487 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:10,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:41:10,874 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 08:41:10,874 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:41:10,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:10,969 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:41:10,983 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 08:41:10,983 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:41:10,986 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 08:41:11,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:41:11,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 08:41:11,355 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:41:11,357 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 08:41:11,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 08:41:11,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 08:41:11,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 08:41:11,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 08:41:11,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 08:41:11,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 08:41:11,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:41:11,476 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 08:41:11,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:41:11,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:11,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:41:11,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:11,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:11,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:41:11,624 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 08:41:11,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:41:11,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:11,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:41:11,670 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:11,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:41:11,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:11,767 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:11,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:11,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:11,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:11,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:11,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:11,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:11,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:11,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:41:12,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:12,035 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:12,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:12,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:12,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:12,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:12,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:12,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:12,168 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:12,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:41:12,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:12,224 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:12,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:12,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:12,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:12,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:12,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:12,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:12,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:12,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:41:12,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:12,395 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:12,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:12,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:12,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:12,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:12,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:12,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:12,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:12,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:41:12,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:12,563 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:12,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:12,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:12,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:12,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:12,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:12,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:12,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:12,698 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:41:12,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:12,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:12,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:12,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:12,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:12,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:12,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:12,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:12,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:12,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:41:12,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:12,903 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:12,920 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:12,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:12,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:12,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:12,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:13,005 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:13,022 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 08:41:13,023 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:41:13,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:41:13,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:41:13,974 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:41:13,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:41:14,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:41:14,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:41:14,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:41:14,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:41:14,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:41:14,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:41:14,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:41:14,694 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 08:41:14,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:41:14,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:19,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:41:19,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:19,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:20,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:41:20,210 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 08:41:20,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:41:20,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:20,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:41:20,306 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 08:41:20,306 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:41:20,309 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 08:41:20,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:41:20,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 08:41:20,694 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:41:20,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 08:41:20,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 08:41:20,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 08:41:20,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 08:41:20,757 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 08:41:20,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 08:41:20,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 08:41:20,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:41:20,815 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 08:41:20,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:41:20,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:20,877 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:41:20,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:20,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:20,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:41:20,937 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 08:41:20,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:41:20,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:20,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:41:20,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:21,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:41:21,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:21,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:21,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:21,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:21,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:21,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:21,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:21,252 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:21,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:21,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:41:21,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:21,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:21,362 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:21,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:21,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:21,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:21,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:21,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:21,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:21,494 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:41:21,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:21,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:21,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:21,566 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:21,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:21,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:21,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:21,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:21,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:21,673 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:41:21,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:21,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:21,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:21,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:21,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:21,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:21,790 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:21,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:21,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:21,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:41:21,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:21,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:21,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:21,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:21,926 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:21,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:21,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:21,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:21,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:22,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:41:22,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:22,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:22,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:22,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:22,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:22,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:22,117 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:22,132 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:22,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:22,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:41:22,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:22,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:22,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:22,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:22,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:22,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:22,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:22,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:22,325 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 08:41:22,328 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:41:22,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:41:22,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:41:22,690 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:41:22,691 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:41:22,706 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:41:22,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:41:22,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:41:22,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:41:22,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:41:22,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:41:22,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:41:22,819 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 08:41:22,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:41:22,867 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:28,599 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:41:29,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:30,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:30,579 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:41:30,598 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 08:41:30,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:41:30,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:30,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:41:30,694 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 08:41:30,696 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:41:30,697 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:41:30,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:41:31,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:41:31,285 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:41:31,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:41:31,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:41:31,334 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:41:31,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:41:31,392 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:41:31,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:41:31,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:41:31,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:41:31,500 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 08:41:31,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:41:31,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:37,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:41:38,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:39,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:39,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:41:39,667 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 08:41:39,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:41:39,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:39,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:41:39,911 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 08:41:39,911 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:41:39,914 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 08:41:40,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:41:40,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 08:41:40,375 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:41:40,375 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 08:41:40,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 08:41:40,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 08:41:40,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 08:41:40,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 08:41:40,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 08:41:40,499 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 08:41:40,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:41:40,540 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 08:41:40,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:41:40,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:40,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:41:40,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:40,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:40,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:41:40,686 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 08:41:40,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:41:40,706 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:40,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:41:40,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:40,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:41:40,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:40,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:40,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:40,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:40,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:41,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:41,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:41,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:41,117 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:41,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:41:41,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:41,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:41,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:41,234 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:41,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:41,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:41,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:41,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:41,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:41,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:41:41,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:41,426 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:41,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:41,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:41,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:41,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:41,538 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:41,561 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:41,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:41,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:41:41,630 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:41,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:41,673 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:41,693 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:41,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:41,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:41,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:41,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:41,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:41,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:41:41,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:41,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:41,895 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:41,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:41,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:41,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:41,982 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:42,002 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:42,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:42,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:41:42,063 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:42,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:42,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:42,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:42,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:42,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:42,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:42,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:42,242 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:42,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:41:42,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:42,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:42,339 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:42,362 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:42,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:42,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:42,426 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:42,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:42,470 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 08:41:42,473 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 08:41:42,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:41:42,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 08:41:42,854 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:41:42,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 08:41:42,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 08:41:42,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 08:41:42,897 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 08:41:42,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 08:41:42,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 08:41:42,946 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 08:41:42,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:41:42,977 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 08:41:43,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:41:43,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:43,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:41:43,066 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:43,081 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:43,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:41:43,111 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 08:41:43,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:41:43,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:43,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:41:43,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:43,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:41:43,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:43,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:43,290 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:43,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:43,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:43,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:43,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:43,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:41:43,482 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:43,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:41:43,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:43,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:43,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:43,576 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:43,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:43,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:43,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:43,647 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:41:43,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:43,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:41:43,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:43,716 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:43,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:43,752 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:43,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:43,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:43,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:43,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:41:43,867 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:43,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:41:43,903 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:43,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:43,936 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:43,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:43,971 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:43,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:44,005 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:44,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:41:44,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:44,066 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:41:44,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:44,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:44,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:44,135 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:44,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:44,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:44,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:44,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:41:44,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:44,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:41:44,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:44,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:44,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:44,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:44,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:44,341 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:44,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:44,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:41:44,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:44,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:41:44,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:44,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:44,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:44,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:44,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:44,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:44,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:44,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:41:44,568 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 08:41:44,571 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:41:44,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:41:45,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:41:45,351 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:41:45,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:41:45,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:41:45,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:41:45,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:41:45,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:41:45,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:41:45,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:41:45,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:41:45,548 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 08:41:45,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:41:45,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 11079 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:41:51,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:41:52,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:52,660 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:53,089 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:41:53,177 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 08:41:53,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:41:53,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:53,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:41:53,306 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 08:41:53,308 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:41:53,311 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:41:53,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:41:53,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:41:53,782 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:41:53,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:41:53,806 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:41:53,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:41:53,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:41:53,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:41:53,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:41:53,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:41:53,936 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:41:53,960 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 08:41:54,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:41:54,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:41:58,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:41:59,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:41:59,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:42:00,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:42:00,174 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 08:42:00,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:42:00,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:42:00,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:42:00,259 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 08:42:00,259 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:42:00,263 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:42:00,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:42:01,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:42:01,371 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:42:01,372 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:42:01,390 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:42:01,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:42:01,426 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:42:01,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:42:01,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:42:01,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:42:01,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:42:01,523 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 08:42:01,632 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:42:01,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:42:06,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:42:07,237 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:42:07,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:42:07,895 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:42:07,924 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 08:42:07,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:42:07,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:42:07,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:42:08,016 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 08:42:08,016 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:42:08,019 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:42:09,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:42:09,614 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:42:09,875 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:42:09,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:42:09,920 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:42:09,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:42:10,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:42:10,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:42:10,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:42:10,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:42:10,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:42:10,184 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 08:42:10,224 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:42:10,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:42:17,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:42:18,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:42:19,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:42:19,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:42:19,776 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 08:42:19,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:42:19,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:42:19,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:42:19,996 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 08:42:19,997 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:42:20,000 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 08:42:20,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:42:20,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 08:42:20,401 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:42:20,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 08:42:20,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 08:42:20,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 08:42:20,459 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 08:42:20,484 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 08:42:20,499 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 08:42:20,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 08:42:20,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:42:20,549 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 08:42:20,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:42:20,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:42:20,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:42:20,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:42:20,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:42:20,701 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:42:20,716 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 08:42:20,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:42:20,732 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:42:20,747 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:42:20,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:42:20,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:42:20,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:42:20,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:42:20,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:42:20,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:42:20,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:42:20,982 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:42:21,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:42:21,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:42:21,081 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:42:21,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:42:21,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:42:21,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:42:21,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:42:21,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:42:21,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:42:21,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:42:21,237 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:42:21,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:42:21,278 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:42:21,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:42:21,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:42:21,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:42:21,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:42:21,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:42:21,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:42:21,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:42:21,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:42:21,443 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:42:21,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:42:21,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:42:21,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:42:21,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:42:21,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:42:21,551 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:42:21,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:42:21,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:42:21,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:42:21,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:42:21,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:42:21,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:42:21,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:42:21,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:42:21,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:42:21,739 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:42:21,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:42:21,775 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:42:21,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:42:21,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:42:21,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:42:21,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:42:21,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:42:21,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:42:21,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:42:21,928 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:42:21,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:42:21,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:42:21,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:42:21,999 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:42:22,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:42:22,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:42:22,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:42:22,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:42:22,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:42:22,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:42:22,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:42:22,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:42:22,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:42:22,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:42:22,236 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 08:42:22,236 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:42:22,240 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:42:22,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:42:22,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:42:22,641 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:42:22,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:42:22,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:42:22,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:42:22,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:42:22,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:42:22,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:42:22,752 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:42:22,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:42:23,286 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:42:23,286 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.0490000248. 2022-02-08 08:42:23,286 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:42:23,289 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:42:23,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:42:24,364 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:42:24,453 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:42:24,454 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:42:24,471 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:42:24,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:42:24,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:42:24,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:42:24,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:42:24,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:42:24,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:42:25,342 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:42:25,344 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.05700016022. 2022-02-08 08:42:25,344 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:42:25,349 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:42:26,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:42:26,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:42:26,815 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:42:26,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:42:26,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:42:26,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:42:26,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:42:26,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:42:26,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:42:27,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:42:27,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:42:27,618 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:42:27,619 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.27599978447. 2022-02-08 08:42:27,619 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:42:27,622 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:42:28,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:42:28,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:42:28,842 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:42:28,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:42:28,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:42:28,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:42:28,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:42:28,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:42:28,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:42:29,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:42:29,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:42:29,648 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:42:29,648 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.02900004387. 2022-02-08 08:42:29,648 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:42:29,651 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:42:30,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:42:30,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:42:31,061 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:42:31,061 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:42:31,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:42:31,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:42:31,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:42:31,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:42:31,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:42:31,224 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:42:31,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:42:31,894 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:42:31,894 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.2460000515. 2022-02-08 08:42:31,894 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:42:31,897 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:42:32,061 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:42:32,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:42:32,476 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:42:32,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:42:32,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:42:32,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:42:32,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:42:32,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:42:32,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:42:32,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:42:32,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:42:33,394 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:42:33,394 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.49900007248. 2022-02-08 08:42:33,394 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:42:33,398 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:42:34,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:42:34,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:42:34,763 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:42:34,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:42:34,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:42:34,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:42:34,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:42:34,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:42:34,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:42:34,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:42:34,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:42:35,391 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:42:35,392 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.99799990654. 2022-02-08 08:42:35,392 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:42:35,394 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:42:36,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:42:36,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:42:36,953 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:42:36,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:42:36,984 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:42:37,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:42:37,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:42:37,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:42:37,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:42:37,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:42:37,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:42:38,311 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:42:38,311 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 08:42:40,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:42:40,492 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:42:40,493 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.10000014305. 2022-02-08 08:42:40,493 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:42:40,494 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:42:40,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:42:41,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:42:41,842 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:42:41,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:42:41,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:42:42,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:42:42,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:42:42,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:42:42,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:42:42,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:42:42,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:42:43,571 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:42:43,571 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:42:45,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:42:46,003 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:42:46,003 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:42:48,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:42:48,448 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:42:48,450 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 5 queries submitted, 0 failed, 11107 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:42:50,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:42:50,868 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:42:50,868 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:42:52,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:42:53,286 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:42:53,286 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:42:55,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:42:55,938 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:42:55,938 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:42:57,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:42:58,345 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:42:58,346 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:43:00,346 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:43:00,904 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:43:00,904 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:43:02,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:43:03,131 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:43:03,131 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:43:05,132 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:43:05,334 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:43:05,335 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 24.8420000076. ############################################################ # # RAM FootPrint: 4240.0625 # ############################################################ 2022-02-08 08:43:05,371 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 08:43:05,371 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 103 seconds. Reason: Iteration:11 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 11133 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:44:48,377 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 08:44:48,378 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 5 queries submitted, 0 failed, 11157 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:44:53,381 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:44:54,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:44:54,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:44:55,216 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:44:55,216 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:44:55,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:44:55,341 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:44:55,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:44:55,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:44:55,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:44:55,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:44:55,576 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:44:55,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:44:55,684 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['35399 memcached\n'], []) 2022-02-08 08:44:55,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 08:44:55,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:44:55,760 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['35399 memcached\n'], []) 2022-02-08 08:44:55,760 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:44:55,763 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:44:56,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:44:57,135 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:44:57,328 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:44:57,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:44:57,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:44:57,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:44:57,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:44:57,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:44:57,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:44:57,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:44:57,519 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:44:57,520 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:44:57,586 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['88217 memcached\n'], []) 2022-02-08 08:44:57,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 08:44:57,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:44:57,654 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['88217 memcached\n'], []) 2022-02-08 08:44:57,654 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:44:57,655 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:44:58,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:44:59,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:44:59,272 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:44:59,272 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:44:59,295 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:44:59,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:44:59,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:44:59,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:44:59,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:44:59,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:44:59,480 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:44:59,482 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:44:59,559 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['70496 memcached\n'], []) 2022-02-08 08:44:59,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 08:44:59,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:44:59,628 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['70496 memcached\n'], []) 2022-02-08 08:44:59,630 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:44:59,631 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:45:01,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:45:02,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:45:02,220 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:45:02,221 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:45:02,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:45:02,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:45:02,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:45:02,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:45:02,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:45:02,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:45:02,388 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:45:02,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:45:02,448 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['99611 memcached\n'], []) 2022-02-08 08:45:02,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 08:45:02,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:45:02,509 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['99611 memcached\n'], []) 2022-02-08 08:45:02,510 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:45:02,513 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:45:04,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:45:05,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:45:05,513 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:45:05,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:45:05,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:45:05,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:45:05,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:45:05,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:45:05,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:45:05,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:45:05,667 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:45:05,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:45:05,727 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['93519 memcached\n'], []) 2022-02-08 08:45:05,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 08:45:05,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:45:05,780 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['93519 memcached\n'], []) 2022-02-08 08:45:05,780 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:45:05,783 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:45:06,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:45:08,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:45:08,451 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:45:08,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:45:08,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:45:08,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:45:08,505 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:45:08,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:45:08,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:45:08,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:45:08,589 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:45:08,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:45:08,650 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['128504 memcached\n'], []) 2022-02-08 08:45:08,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 08:45:08,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:45:08,707 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['128504 memcached\n'], []) 2022-02-08 08:45:08,707 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:45:08,710 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:45:09,581 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:45:10,168 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:45:10,311 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:45:10,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:45:10,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:45:10,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:45:10,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:45:10,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:45:10,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:45:10,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:45:10,450 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:45:10,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:45:10,496 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['62399 memcached\n'], []) 2022-02-08 08:45:10,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 08:45:10,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:45:10,543 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['62399 memcached\n'], []) 2022-02-08 08:45:10,545 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:45:10,546 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:45:11,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:45:11,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:45:12,010 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:45:12,010 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:45:12,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:45:12,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:45:12,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:45:12,104 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:45:12,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:45:12,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:45:12,171 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:45:12,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:45:12,233 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['77278 memcached\n'], []) 2022-02-08 08:45:12,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 08:45:12,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:45:12,301 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['77278 memcached\n'], []) 2022-02-08 08:45:12,302 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:45:12,305 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:45:13,081 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:45:13,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:45:14,191 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:45:14,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:45:14,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:45:14,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:45:14,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:45:14,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:45:14,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:45:14,346 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:45:14,365 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:45:14,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:45:14,427 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['12773 memcached\n'], []) 2022-02-08 08:45:14,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 08:45:14,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:45:14,479 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['12773 memcached\n'], []) 2022-02-08 08:45:14,479 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:45:14,480 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:45:14,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:45:14,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:45:14,901 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:45:14,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:45:14,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:45:14,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:45:14,959 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:45:14,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:45:14,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:45:15,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:45:15,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:45:15,042 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 08:45:15,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:45:15,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:45:20,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:45:21,499 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:45:21,959 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:45:22,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:45:22,421 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 08:45:22,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:45:22,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:45:22,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:45:22,516 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 08:45:22,516 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:45:22,519 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:45:22,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:45:22,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:45:22,920 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:45:22,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:45:22,936 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:45:22,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:45:22,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:45:22,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:45:22,999 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:45:23,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:45:23,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:45:23,046 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 08:45:23,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:45:23,096 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:45:29,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:45:30,395 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:45:30,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:45:31,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:45:31,184 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 08:45:31,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:45:31,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:45:31,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:45:31,279 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 08:45:31,279 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:45:31,285 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 08:45:31,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:45:31,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 08:45:31,697 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:45:31,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 08:45:31,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 08:45:31,724 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 08:45:31,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 08:45:31,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 08:45:31,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 08:45:31,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 08:45:31,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:45:31,815 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 08:45:31,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:45:31,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:45:31,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:45:31,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:45:31,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:45:31,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:45:31,933 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 08:45:31,933 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:45:31,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:45:31,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:45:31,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:32,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:45:32,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:32,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:32,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:32,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:32,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:32,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:32,232 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:32,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:32,295 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:32,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:45:32,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:32,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:32,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:32,385 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:32,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:32,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:32,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:32,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:32,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:32,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:45:32,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:32,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:32,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:32,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:32,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:32,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:32,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:32,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:32,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:32,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:45:32,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:32,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:32,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:32,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:32,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:32,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:32,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:32,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:32,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:32,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:45:32,880 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:32,897 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:32,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:32,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:32,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:32,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:32,982 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:32,999 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:33,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:33,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:45:33,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:33,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:33,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:33,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:33,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:33,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:33,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:33,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:33,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:33,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:45:33,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:33,239 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:33,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:33,275 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:33,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:33,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:33,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:33,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:33,367 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 08:45:33,369 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:45:33,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:45:33,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:45:34,227 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:45:34,227 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:45:34,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:45:34,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:45:34,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:45:34,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:45:34,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:45:34,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:45:34,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:45:35,019 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 08:45:35,048 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:45:35,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:45:39,596 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:45:40,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:45:40,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:45:40,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:45:40,798 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 08:45:40,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:45:40,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:45:40,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:45:40,874 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 08:45:40,874 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:45:40,877 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 08:45:40,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:45:41,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 08:45:41,273 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:45:41,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 08:45:41,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 08:45:41,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 08:45:41,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 08:45:41,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 08:45:41,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 08:45:41,362 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 08:45:41,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:45:41,394 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 08:45:41,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:45:41,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:45:41,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:45:41,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:45:41,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:45:41,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:45:41,520 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 08:45:41,520 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:45:41,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:45:41,551 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:45:41,566 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:41,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:45:41,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:41,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:41,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:41,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:41,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:41,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:41,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:41,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:45:41,903 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:41,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:45:41,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:41,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:41,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:41,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:42,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:42,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:42,048 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:42,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:45:42,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:42,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:45:42,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:42,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:42,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:42,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:42,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:42,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:42,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:42,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:45:42,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:42,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:45:42,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:42,318 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:42,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:42,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:42,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:42,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:42,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:42,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:45:42,438 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:42,454 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:45:42,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:42,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:42,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:42,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:42,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:42,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:42,561 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:42,576 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:45:42,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:42,614 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:45:42,630 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:42,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:42,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:42,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:42,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:42,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:42,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:42,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:45:42,762 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:42,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:45:42,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:42,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:42,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:42,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:42,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:42,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:42,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:42,920 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:45:42,937 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 08:45:42,940 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:45:43,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:45:43,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:45:43,349 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:45:43,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:45:43,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:45:43,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:45:43,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:45:43,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:45:43,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:45:43,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:45:43,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:45:43,483 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 08:45:43,512 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:45:43,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 11180 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:45:49,900 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:45:50,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:45:51,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:45:51,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:45:51,523 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 08:45:51,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:45:51,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:45:51,596 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:45:51,612 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 08:45:51,612 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:45:51,615 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:45:51,767 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:45:52,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:45:52,153 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:45:52,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:45:52,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:45:52,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:45:52,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:45:52,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:45:52,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:45:52,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:45:52,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:45:52,536 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 08:45:52,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:45:52,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:45:58,920 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:45:59,729 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:46:00,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:46:00,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:46:00,802 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 08:46:00,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:46:00,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:46:00,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:46:00,941 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 08:46:00,943 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:46:00,946 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 08:46:01,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:46:01,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 08:46:01,385 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:46:01,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 08:46:01,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 08:46:01,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 08:46:01,438 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 08:46:01,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 08:46:01,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 08:46:01,494 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 08:46:01,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:46:01,529 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 08:46:01,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:46:01,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:46:01,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:46:01,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:46:01,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:46:01,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:46:01,674 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 08:46:01,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:46:01,691 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:46:01,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:46:01,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:01,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:46:01,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:01,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:01,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:01,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:01,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:01,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:02,020 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:02,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:02,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:02,114 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:46:02,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:02,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:02,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:02,224 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:02,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:02,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:02,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:02,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:02,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:02,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:46:02,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:02,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:02,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:02,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:02,471 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:02,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:02,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:02,533 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:02,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:02,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:46:02,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:02,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:02,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:02,660 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:02,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:02,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:02,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:02,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:02,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:02,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:46:02,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:02,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:02,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:02,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:02,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:02,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:02,933 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:02,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:02,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:02,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:46:03,017 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:03,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:03,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:03,081 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:03,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:03,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:03,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:03,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:03,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:03,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:46:03,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:03,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:03,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:03,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:03,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:03,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:03,364 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:03,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:03,408 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 08:46:03,411 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 08:46:03,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:46:03,724 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 08:46:03,789 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:46:03,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 08:46:03,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 08:46:03,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 08:46:03,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 08:46:03,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 08:46:03,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 08:46:03,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 08:46:03,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:46:03,934 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 08:46:03,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:46:03,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:46:03,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:46:04,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:46:04,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:46:04,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:46:04,056 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 08:46:04,058 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:46:04,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:46:04,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:46:04,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:04,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:46:04,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:04,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:04,239 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:04,270 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:04,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:04,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:04,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:04,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:04,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:04,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:46:04,484 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:04,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:04,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:04,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:04,556 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:04,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:04,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:04,609 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:04,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:04,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:46:04,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:04,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:04,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:04,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:04,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:04,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:04,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:04,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:04,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:04,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:46:04,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:04,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:04,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:04,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:04,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:04,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:04,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:04,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:04,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:04,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:46:04,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:05,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:05,025 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:05,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:05,058 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:05,073 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:05,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:05,107 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:05,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:05,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:46:05,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:05,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:05,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:05,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:05,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:05,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:05,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:05,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:05,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:05,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:46:05,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:05,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:05,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:05,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:05,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:05,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:05,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:05,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:05,505 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 08:46:05,507 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:46:06,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:46:06,775 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:46:07,036 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:46:07,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:46:07,073 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:46:07,104 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:46:07,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:46:07,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:46:07,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:46:07,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:46:07,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:46:07,315 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 08:46:07,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:46:07,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:46:13,494 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:46:14,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:46:14,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:46:15,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:46:15,266 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 08:46:15,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:46:15,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:46:15,390 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:46:15,407 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 08:46:15,407 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:46:15,408 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:46:15,489 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:46:15,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:46:15,805 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:46:15,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:46:15,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:46:15,839 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:46:15,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:46:15,877 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:46:15,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:46:15,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:46:15,926 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:46:15,944 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 08:46:15,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:46:15,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:46:21,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:46:21,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:46:22,084 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:46:22,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:46:22,460 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 08:46:22,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:46:22,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:46:22,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:46:22,552 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 08:46:22,553 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:46:22,558 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:46:23,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:46:23,556 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:46:23,684 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:46:23,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:46:23,698 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:46:23,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:46:23,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:46:23,752 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:46:23,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:46:23,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:46:23,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:46:23,831 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 08:46:23,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:46:23,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:46:28,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:46:29,094 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:46:29,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:46:29,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:46:29,753 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 08:46:29,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:46:29,767 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:46:29,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:46:29,828 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 08:46:29,828 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:46:29,829 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:46:29,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:46:30,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:46:30,233 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:46:30,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:46:30,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:46:30,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:46:30,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:46:30,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:46:30,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:46:30,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:46:30,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:46:30,388 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 08:46:30,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:46:30,436 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:46:38,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:46:39,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:46:40,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:46:41,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:46:41,654 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 08:46:41,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:46:41,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:46:41,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:46:42,009 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 08:46:42,009 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:46:42,012 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 08:46:42,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:46:42,349 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 08:46:42,418 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:46:42,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 08:46:42,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 08:46:42,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 08:46:42,464 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 08:46:42,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 08:46:42,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 08:46:42,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 08:46:42,540 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:46:42,559 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 08:46:42,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:46:42,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:46:42,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:46:42,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:46:42,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:46:42,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:46:42,687 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 08:46:42,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:46:42,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:46:42,719 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:46:42,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:42,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:46:42,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:42,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:42,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:42,900 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:42,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:42,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:43,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:43,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:46:43,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:43,089 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:46:43,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:43,127 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:43,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:43,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:43,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:43,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:43,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:43,242 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:46:43,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:43,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:46:43,306 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:43,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:43,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:43,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:43,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:43,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:43,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:43,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:46:43,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:43,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:46:43,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:43,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:43,517 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:43,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:43,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:43,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:43,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:43,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:46:43,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:43,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:46:43,660 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:43,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:43,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:43,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:43,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:43,747 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:43,767 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:43,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:46:43,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:43,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:46:43,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:43,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:43,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:43,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:43,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:43,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:43,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:43,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:46:43,969 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:43,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:46:44,005 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:44,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:44,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:44,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:44,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:44,094 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:44,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:44,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:46:44,148 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 08:46:44,148 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:46:44,151 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:46:44,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:46:44,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:46:44,671 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:46:44,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:46:44,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:46:44,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:46:44,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:46:44,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:46:44,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:46:44,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:46:44,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:46:45,292 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:46:45,292 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.14499998093. 2022-02-08 08:46:45,293 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:46:45,295 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:46:46,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:46:46,665 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:46:46,786 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:46:46,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:46:46,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:46:46,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:46:46,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:46:46,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:46:46,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:46:46,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:46:46,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:46:47,653 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:46:47,653 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.36000013351. 2022-02-08 08:46:47,654 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:46:47,654 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:46:48,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:46:48,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:46:49,101 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:46:49,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:46:49,124 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:46:49,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:46:49,168 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:46:49,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:46:49,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:46:49,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:46:49,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 11205 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:46:49,940 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:46:49,940 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.28600001335. 2022-02-08 08:46:49,940 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:46:49,941 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:46:51,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:46:51,676 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:46:51,862 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:46:51,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:46:51,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:46:51,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:46:51,933 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:46:51,961 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:46:51,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:46:52,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:46:52,048 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:46:52,786 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:46:52,786 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.84599995613. 2022-02-08 08:46:52,786 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:46:52,789 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:46:53,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:46:53,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:46:54,098 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:46:54,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:46:54,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:46:54,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:46:54,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:46:54,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:46:54,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:46:54,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:46:54,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:46:54,884 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:46:54,885 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.09899997711. 2022-02-08 08:46:54,885 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:46:54,888 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:46:55,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:46:55,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:46:56,267 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:46:56,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:46:56,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:46:56,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:46:56,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:46:56,443 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:46:56,492 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:46:56,512 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:46:56,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:46:57,703 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:46:57,703 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 08:46:59,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:47:00,246 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:47:00,246 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 08:47:02,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:47:02,559 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:47:02,559 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 08:47:04,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:47:04,780 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:47:04,782 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 08:47:06,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:47:06,967 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:47:06,967 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 08:47:08,969 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:47:09,357 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:47:09,357 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 08:47:11,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:47:11,642 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:47:11,642 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 08:47:13,642 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:47:14,141 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:47:14,142 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 08:47:16,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:47:16,342 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:47:16,344 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 08:47:18,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:47:18,523 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:47:18,523 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 23.6380000114. 2022-02-08 08:47:18,525 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:47:18,526 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:47:19,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:47:19,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:47:19,937 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:47:19,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:47:19,959 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:47:19,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:47:20,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:47:20,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:47:20,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:47:20,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:47:20,089 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:47:20,598 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:47:20,598 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.07299995422. 2022-02-08 08:47:20,598 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:47:20,599 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:47:21,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:47:21,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:47:21,746 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:47:21,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:47:21,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:47:21,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:47:21,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:47:21,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:47:21,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:47:21,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:47:21,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:47:22,522 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:47:22,523 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.9240000248. 2022-02-08 08:47:22,523 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:47:22,525 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:47:23,729 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:47:24,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:47:24,803 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:47:24,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:47:24,859 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:47:24,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:47:25,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:47:25,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:47:25,280 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:47:25,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:47:25,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:47:26,243 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:47:26,243 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:47:28,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:47:28,413 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:47:28,414 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.89100003242. ############################################################ # # RAM FootPrint: 4446.83984375 # ############################################################ 2022-02-08 08:47:28,450 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 08:47:28,451 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 120 seconds. Reason: Iteration:12 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 11229 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 11250 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:49:28,457 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 08:49:28,457 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 08:49:33,459 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:49:34,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:49:34,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:49:35,142 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:49:35,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:49:35,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:49:35,244 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:49:35,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:49:35,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:49:35,392 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:49:35,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:49:35,500 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:49:35,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:49:35,651 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['36140 memcached\n'], []) 2022-02-08 08:49:35,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 08:49:35,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:49:35,779 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['36140 memcached\n'], []) 2022-02-08 08:49:35,779 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:49:35,782 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:49:36,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:49:37,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:49:37,505 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:49:37,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:49:37,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:49:37,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:49:37,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:49:37,596 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:49:37,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:49:37,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:49:37,683 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:49:37,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:49:37,796 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['88960 memcached\n'], []) 2022-02-08 08:49:37,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 08:49:37,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:49:37,864 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['88960 memcached\n'], []) 2022-02-08 08:49:37,865 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:49:37,867 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:49:38,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:49:39,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:49:39,976 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:49:39,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:49:39,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:49:40,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:49:40,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:49:40,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:49:40,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:49:40,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:49:40,196 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:49:40,196 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:49:40,276 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['71333 memcached\n'], []) 2022-02-08 08:49:40,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 08:49:40,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:49:40,342 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['71333 memcached\n'], []) 2022-02-08 08:49:40,342 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:49:40,345 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:49:41,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:49:42,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:49:42,532 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:49:42,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:49:42,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:49:42,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:49:42,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:49:42,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:49:42,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:49:42,673 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:49:42,688 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:49:42,688 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:49:42,769 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['100364 memcached\n'], []) 2022-02-08 08:49:42,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 08:49:42,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:49:42,825 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['100364 memcached\n'], []) 2022-02-08 08:49:42,825 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:49:42,826 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:49:44,020 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:49:44,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:49:45,203 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:49:45,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:49:45,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:49:45,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:49:45,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:49:45,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:49:45,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:49:45,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:49:45,407 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:49:45,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:49:45,474 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['94263 memcached\n'], []) 2022-02-08 08:49:45,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 08:49:45,505 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:49:45,526 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['94263 memcached\n'], []) 2022-02-08 08:49:45,526 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:49:45,529 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:49:46,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:49:47,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:49:47,974 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:49:47,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:49:47,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:49:48,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:49:48,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:49:48,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:49:48,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:49:48,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:49:48,101 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:49:48,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:49:48,211 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['129249 memcached\n'], []) 2022-02-08 08:49:48,211 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 08:49:48,272 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:49:48,293 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['129249 memcached\n'], []) 2022-02-08 08:49:48,293 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:49:48,296 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:49:49,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 11275 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:49:49,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:49:50,012 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:49:50,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:49:50,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:49:50,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:49:50,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:49:50,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:49:50,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:49:50,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:49:50,144 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:49:50,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:49:50,196 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['63173 memcached\n'], []) 2022-02-08 08:49:50,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 08:49:50,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:49:50,283 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['63173 memcached\n'], []) 2022-02-08 08:49:50,285 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:49:50,286 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:49:51,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:49:51,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:49:51,711 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:49:51,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:49:51,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:49:51,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:49:51,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:49:51,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:49:51,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:49:51,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:49:51,845 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:49:51,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:49:51,904 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['78080 memcached\n'], []) 2022-02-08 08:49:51,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 08:49:51,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:49:51,956 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['78080 memcached\n'], []) 2022-02-08 08:49:51,957 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:49:51,960 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:49:52,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:49:53,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:49:53,859 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:49:53,859 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:49:53,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:49:53,897 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:49:53,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:49:53,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:49:53,969 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:49:53,994 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:49:54,009 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:49:54,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:49:54,076 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['13625 memcached\n'], []) 2022-02-08 08:49:54,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 08:49:54,107 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:49:54,128 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['13625 memcached\n'], []) 2022-02-08 08:49:54,128 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:49:54,131 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:49:54,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:49:54,451 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:49:54,513 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:49:54,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:49:54,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:49:54,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:49:54,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:49:54,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:49:54,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:49:54,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:49:54,609 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:49:54,625 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 08:49:54,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:49:54,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:00,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:50:01,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:01,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:02,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:50:02,325 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 08:50:02,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:50:02,339 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:02,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:50:02,407 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 08:50:02,408 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:50:02,411 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:50:02,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:50:02,716 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:50:02,796 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:50:02,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:50:02,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:50:02,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:50:02,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:50:02,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:50:02,864 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:50:02,877 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:50:02,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:50:02,904 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 08:50:02,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:50:02,946 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:09,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:50:10,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:10,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:11,357 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:50:11,372 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 08:50:11,372 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:50:11,390 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:11,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:50:11,473 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 08:50:11,473 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:50:11,476 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 08:50:11,566 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:50:11,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 08:50:11,890 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:50:11,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 08:50:11,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 08:50:11,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 08:50:11,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 08:50:11,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 08:50:11,969 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 08:50:11,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 08:50:12,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:50:12,019 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 08:50:12,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:50:12,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:12,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:50:12,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:12,107 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:12,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:50:12,135 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 08:50:12,137 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:50:12,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:12,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:50:12,183 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:12,214 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:50:12,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:12,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:12,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:12,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:12,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:12,433 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:12,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:12,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:12,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:12,551 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:50:12,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:12,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:12,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:12,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:12,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:12,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:12,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:12,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:12,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:12,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:50:12,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:12,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:12,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:12,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:12,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:12,852 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:12,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:12,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:12,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:12,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:50:12,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:12,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:12,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:12,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:13,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:13,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:13,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:13,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:13,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:13,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:50:13,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:13,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:13,147 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:13,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:13,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:13,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:13,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:13,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:13,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:13,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:50:13,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:13,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:13,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:13,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:13,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:13,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:13,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:13,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:13,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:13,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:50:13,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:13,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:13,487 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:13,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:13,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:13,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:13,561 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:13,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:13,595 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 08:50:13,598 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:50:13,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:50:14,471 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:50:15,059 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:50:15,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:50:15,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:50:15,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:50:15,306 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:50:15,533 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:50:15,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:50:15,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:50:15,903 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:50:15,993 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 08:50:16,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:50:16,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:20,658 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:50:21,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:21,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:22,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:50:22,533 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 08:50:22,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:50:22,609 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:22,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:50:22,891 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 08:50:22,892 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:50:22,894 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 08:50:22,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:50:23,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 08:50:23,270 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:50:23,272 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 08:50:23,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 08:50:23,298 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 08:50:23,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 08:50:23,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 08:50:23,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 08:50:23,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 08:50:23,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:50:23,377 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 08:50:23,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:50:23,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:23,433 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:50:23,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:23,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:23,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:50:23,489 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 08:50:23,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:50:23,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:23,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:50:23,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:23,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:50:23,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:23,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:23,676 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:23,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:23,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:23,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:23,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:23,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:23,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:23,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:50:23,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:23,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:23,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:23,971 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:23,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:24,005 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:24,022 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:24,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:24,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:24,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:50:24,094 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:24,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:24,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:24,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:24,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:24,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:24,196 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:24,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:24,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:24,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:50:24,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:24,280 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:24,298 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:24,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:24,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:24,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:24,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:24,380 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:24,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:24,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:50:24,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:24,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:24,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:24,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:24,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:24,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:24,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:24,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:24,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:24,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:50:24,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:24,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:24,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:24,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:24,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:24,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:24,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:24,716 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:24,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:24,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:50:24,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:24,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:24,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:24,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:24,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:24,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:24,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:24,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:24,905 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 08:50:24,908 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:50:25,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:50:25,272 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:50:25,532 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:50:25,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:50:25,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:50:25,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:50:25,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:50:25,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:50:25,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:50:25,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:50:25,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:50:25,930 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 08:50:25,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:50:25,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:31,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:50:32,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:33,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:33,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:50:33,690 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 08:50:33,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:50:33,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:33,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:50:33,789 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 08:50:33,789 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:50:33,792 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:50:33,885 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:50:34,168 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:50:34,249 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:50:34,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:50:34,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:50:34,290 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:50:34,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:50:34,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:50:34,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:50:34,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:50:34,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:50:34,499 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 08:50:34,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:50:34,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:40,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:50:41,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:42,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:42,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:50:42,530 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 08:50:42,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:50:42,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:42,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:50:42,654 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 08:50:42,655 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:50:42,657 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 08:50:42,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:50:43,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 08:50:43,105 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:50:43,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 08:50:43,124 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 08:50:43,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 08:50:43,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 08:50:43,183 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 08:50:43,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 08:50:43,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 08:50:43,239 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:50:43,259 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 08:50:43,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:50:43,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:43,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:50:43,346 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:43,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:43,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:50:43,394 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 08:50:43,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:50:43,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:43,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:50:43,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:43,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:50:43,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:43,548 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:43,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:43,619 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:43,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:43,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:43,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:43,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:43,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:43,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:50:43,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:43,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:43,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:43,897 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:43,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:43,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:43,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:43,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:43,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:44,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:50:44,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:44,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:44,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:44,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:44,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:44,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:44,135 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:44,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:44,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:44,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:50:44,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:44,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:44,239 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:44,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:44,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:44,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:44,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:44,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:44,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:44,375 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:50:44,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:44,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:44,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:44,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:44,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:44,489 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:44,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:44,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:44,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:44,563 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:50:44,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:44,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:44,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:44,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:44,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:44,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:44,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:44,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:44,729 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:44,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:50:44,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:44,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:44,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:44,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:44,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:44,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:44,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:44,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:44,957 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 08:50:44,960 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 08:50:45,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:50:45,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 08:50:45,336 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:50:45,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 08:50:45,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 08:50:45,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 08:50:45,380 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 08:50:45,395 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 08:50:45,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 08:50:45,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 08:50:45,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:50:45,463 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 08:50:45,487 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:50:45,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:45,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:50:45,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:45,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:45,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:50:45,585 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 08:50:45,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:50:45,599 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:45,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:50:45,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:45,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:50:45,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:45,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:45,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:45,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:45,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:45,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:45,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:45,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:50:45,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:46,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:50:46,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:46,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:46,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:46,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:46,104 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:46,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:46,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:46,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:50:46,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:46,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:50:46,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:46,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:46,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:46,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:46,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:46,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:46,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:46,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:50:46,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:46,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:50:46,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:46,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:46,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:46,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:46,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:46,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:46,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:46,482 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:50:46,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:46,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:50:46,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:46,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:46,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:46,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:46,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:46,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:46,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:46,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:50:46,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:46,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:50:46,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:46,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:46,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:46,757 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:46,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:46,788 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:46,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:46,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:50:46,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:46,852 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:50:46,874 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:46,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:46,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:46,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:46,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:46,961 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:46,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:46,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:50:47,015 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 08:50:47,017 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:50:47,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:50:47,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:50:47,556 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:50:47,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:50:47,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:50:47,594 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:50:47,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:50:47,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:50:47,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:50:47,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:50:47,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:50:47,799 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 08:50:47,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:50:47,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 11297 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:50:53,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:50:54,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:54,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:50:55,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:50:55,270 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 08:50:55,272 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:50:55,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:50:55,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:50:55,362 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 08:50:55,362 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:50:55,365 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:50:55,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:50:55,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:50:56,016 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:50:56,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:50:56,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:50:56,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:50:56,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:50:56,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:50:56,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:50:56,127 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:50:56,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:50:56,180 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 08:50:56,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:50:56,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:51:01,081 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:51:01,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:51:02,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:51:02,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:51:02,726 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 08:51:02,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:51:02,757 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:51:02,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:51:02,845 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 08:51:02,845 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:51:02,848 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:51:03,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:51:03,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:51:04,105 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:51:04,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:51:04,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:51:04,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:51:04,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:51:04,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:51:04,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:51:04,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:51:04,270 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:51:04,286 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 08:51:04,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:51:04,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:51:08,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:51:09,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:51:09,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:51:10,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:51:10,140 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 08:51:10,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:51:10,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:51:10,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:51:10,246 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 08:51:10,246 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:51:10,250 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:51:10,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:51:10,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:51:10,724 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:51:10,724 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:51:10,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:51:10,762 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:51:10,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:51:10,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:51:10,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:51:10,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:51:10,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:51:10,872 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 08:51:10,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:51:10,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:51:17,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:51:18,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:51:19,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:51:19,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:51:19,713 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 08:51:19,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:51:19,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:51:19,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:51:19,795 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 08:51:19,795 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:51:19,796 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 08:51:19,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:51:20,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 08:51:20,191 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:51:20,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 08:51:20,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 08:51:20,221 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 08:51:20,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 08:51:20,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 08:51:20,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 08:51:20,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 08:51:20,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:51:20,319 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 08:51:20,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:51:20,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:51:20,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:51:20,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:51:20,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:51:20,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:51:20,438 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 08:51:20,438 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:51:20,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:51:20,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:51:20,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:51:20,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:51:20,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:51:20,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:51:20,642 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:51:20,686 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:51:20,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:51:20,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:51:20,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:51:20,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:51:20,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:51:20,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:51:20,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:51:20,928 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:51:20,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:51:20,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:51:20,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:51:20,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:51:21,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:51:21,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:51:21,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:51:21,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:51:21,086 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:51:21,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:51:21,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:51:21,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:51:21,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:51:21,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:51:21,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:51:21,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:51:21,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:51:21,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:51:21,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:51:21,275 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:51:21,298 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:51:21,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:51:21,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:51:21,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:51:21,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:51:21,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:51:21,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:51:21,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:51:21,438 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:51:21,454 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:51:21,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:51:21,489 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:51:21,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:51:21,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:51:21,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:51:21,566 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:51:21,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:51:21,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:51:21,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:51:21,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:51:21,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:51:21,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:51:21,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:51:21,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:51:21,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:51:21,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:51:21,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:51:21,778 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:51:21,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:51:21,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:51:21,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:51:21,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:51:21,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:51:21,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:51:21,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:51:21,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:51:21,934 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 08:51:21,937 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:51:21,938 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:51:22,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:51:22,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:51:22,348 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:51:22,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:51:22,362 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:51:22,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:51:22,392 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:51:22,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:51:22,433 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:51:22,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:51:22,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:51:23,033 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:51:23,033 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.09699988365. 2022-02-08 08:51:23,035 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:51:23,036 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:51:24,168 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:51:24,507 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:51:24,676 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:51:24,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:51:24,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:51:24,732 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:51:24,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:51:24,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:51:24,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:51:24,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:51:24,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:51:25,569 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:51:25,569 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:51:27,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:51:27,759 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:51:27,759 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:51:29,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:51:29,930 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:51:29,930 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:51:31,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:51:32,165 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:51:32,167 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:51:34,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:51:34,398 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:51:34,398 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:51:36,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:51:36,618 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:51:36,619 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:51:38,619 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:51:38,878 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:51:38,878 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 08:51:40,880 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:51:41,109 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:51:41,109 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 18.0759999752. 2022-02-08 08:51:41,111 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:51:41,111 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:51:42,048 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:51:42,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:51:42,668 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:51:42,670 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:51:42,688 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:51:42,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:51:42,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:51:42,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:51:42,936 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:51:42,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:51:42,987 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:51:43,871 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:51:43,871 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 08:51:45,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:51:46,095 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:51:46,095 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.9849998951. 2022-02-08 08:51:46,096 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:51:46,098 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:51:47,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:51:47,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:51:47,946 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:51:47,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:51:47,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:51:47,987 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:51:48,017 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:51:48,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:51:48,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:51:48,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:51:48,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:51:48,795 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:51:48,796 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.69900012016. 2022-02-08 08:51:48,796 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:51:48,798 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 5 queries submitted, 0 failed, 11321 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:51:49,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:51:50,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:51:50,601 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:51:50,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:51:50,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:51:50,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:51:50,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:51:50,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:51:50,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:51:50,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:51:50,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:51:51,405 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:51:51,405 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.6099998951. 2022-02-08 08:51:51,407 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:51:51,408 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:51:52,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:51:52,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:51:52,529 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:51:52,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:51:52,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:51:52,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:51:52,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:51:52,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:51:52,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:51:52,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:51:52,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:51:53,394 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:51:53,394 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.98799991608. 2022-02-08 08:51:53,395 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:51:53,398 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:51:53,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:51:54,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:51:54,421 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:51:54,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:51:54,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:51:54,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:51:54,482 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:51:54,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:51:54,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:51:54,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:51:54,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:51:55,108 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:51:55,108 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.71300005913. 2022-02-08 08:51:55,108 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:51:55,111 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:51:55,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:51:56,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:51:56,437 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:51:56,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:51:56,459 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:51:56,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:51:56,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:51:56,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:51:56,548 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:51:56,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:51:56,614 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:51:57,523 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:51:57,523 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.41499996185. 2022-02-08 08:51:57,525 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:51:57,526 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:51:57,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:51:58,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:51:58,408 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:51:58,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:51:58,489 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:51:58,571 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:51:58,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:51:58,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:51:58,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:51:58,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:51:58,885 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:52:00,140 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:52:00,141 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:52:02,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:52:02,632 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:52:02,632 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:52:04,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:52:04,882 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:52:04,882 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:52:06,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:52:07,765 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:52:07,765 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:52:09,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:52:10,160 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:52:10,160 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:52:12,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:52:12,331 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:52:12,332 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 14.8069999218. ############################################################ # # RAM FootPrint: 4538.08203125 # ############################################################ 2022-02-08 08:52:12,368 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 08:52:12,368 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 101 seconds. Reason: Iteration:13 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 11348 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 11370 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:53:53,377 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 08:53:53,377 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 08:53:58,381 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:53:59,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:53:59,946 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:54:00,299 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:54:00,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:54:00,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:54:00,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:54:00,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:54:00,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:54:00,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:54:00,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:54:00,467 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:54:00,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:54:00,523 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['36932 memcached\n'], []) 2022-02-08 08:54:00,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 08:54:00,658 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:54:00,683 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['36932 memcached\n'], []) 2022-02-08 08:54:00,684 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:54:00,686 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:54:01,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:54:02,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:54:02,427 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:54:02,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:54:02,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:54:02,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:54:02,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:54:02,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:54:02,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:54:02,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:54:02,783 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:54:02,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:54:02,857 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['89763 memcached\n'], []) 2022-02-08 08:54:02,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 08:54:02,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:54:02,927 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['89763 memcached\n'], []) 2022-02-08 08:54:02,928 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:54:02,930 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:54:03,903 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:54:04,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:54:05,016 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:54:05,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:54:05,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:54:05,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:54:05,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:54:05,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:54:05,147 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:54:05,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:54:05,203 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:54:05,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:54:05,279 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['72183 memcached\n'], []) 2022-02-08 08:54:05,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 08:54:05,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:54:05,354 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['72183 memcached\n'], []) 2022-02-08 08:54:05,354 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:54:05,357 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:54:06,252 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:54:06,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:54:06,928 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:54:06,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:54:06,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:54:06,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:54:07,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:54:07,030 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:54:07,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:54:07,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:54:07,111 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:54:07,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:54:07,181 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['101154 memcached\n'], []) 2022-02-08 08:54:07,183 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 08:54:07,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:54:07,240 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['101154 memcached\n'], []) 2022-02-08 08:54:07,240 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:54:07,243 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:54:08,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:54:08,999 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:54:09,198 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:54:09,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:54:09,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:54:09,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:54:09,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:54:09,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:54:09,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:54:09,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:54:09,381 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:54:09,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:54:09,447 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['95024 memcached\n'], []) 2022-02-08 08:54:09,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 08:54:09,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:54:09,506 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['95024 memcached\n'], []) 2022-02-08 08:54:09,506 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:54:09,507 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:54:10,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:54:11,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:54:11,398 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:54:11,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:54:11,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:54:11,436 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:54:11,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:54:11,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:54:11,487 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:54:11,520 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:54:11,538 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:54:11,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:54:11,602 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['130157 memcached\n'], []) 2022-02-08 08:54:11,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 08:54:11,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:54:11,661 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['130157 memcached\n'], []) 2022-02-08 08:54:11,663 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:54:11,664 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:54:12,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:54:13,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:54:13,319 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:54:13,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:54:13,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:54:13,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:54:13,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:54:13,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:54:13,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:54:13,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:54:13,585 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:54:13,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:54:13,631 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['63937 memcached\n'], []) 2022-02-08 08:54:13,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 08:54:13,655 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:54:13,678 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['63937 memcached\n'], []) 2022-02-08 08:54:13,680 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:54:13,684 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:54:14,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:54:14,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:54:14,769 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:54:14,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:54:14,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:54:14,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:54:14,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:54:14,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:54:14,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:54:14,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:54:14,927 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:54:14,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:54:14,993 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['78838 memcached\n'], []) 2022-02-08 08:54:14,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 08:54:15,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:54:15,046 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['78838 memcached\n'], []) 2022-02-08 08:54:15,046 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:54:15,049 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:54:15,706 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:54:16,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:54:16,618 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:54:16,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:54:16,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:54:16,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:54:16,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:54:16,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:54:16,716 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:54:16,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:54:16,762 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:54:16,762 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:54:16,822 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['14430 memcached\n'], []) 2022-02-08 08:54:16,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 08:54:16,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:54:16,868 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['14430 memcached\n'], []) 2022-02-08 08:54:16,869 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:54:16,871 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:54:16,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:54:17,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:54:17,282 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:54:17,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:54:17,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:54:17,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:54:17,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:54:17,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:54:17,357 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:54:17,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:54:17,385 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:54:17,401 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 08:54:17,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:54:17,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:54:23,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:54:23,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:54:24,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:54:24,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:54:24,780 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 08:54:24,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:54:24,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:54:24,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:54:24,872 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 08:54:24,872 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:54:24,875 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:54:24,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:54:25,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:54:25,250 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:54:25,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:54:25,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:54:25,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:54:25,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:54:25,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:54:25,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:54:25,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:54:25,341 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:54:25,355 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 08:54:25,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:54:25,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:54:31,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:54:32,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:54:32,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:54:33,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:54:33,186 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 08:54:33,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:54:33,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:54:33,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:54:33,279 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 08:54:33,279 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:54:33,280 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 08:54:33,367 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:54:33,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 08:54:33,657 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:54:33,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 08:54:33,673 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 08:54:33,686 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 08:54:33,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 08:54:33,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 08:54:33,732 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 08:54:33,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 08:54:33,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:54:33,776 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 08:54:33,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:54:33,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:54:33,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:54:33,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:54:33,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:54:33,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:54:33,894 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 08:54:33,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:54:33,908 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:54:33,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:54:33,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:33,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:54:34,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:34,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:34,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:34,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:34,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:34,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:34,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:34,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:34,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:34,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:54:34,346 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:34,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:34,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:34,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:34,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:34,443 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:34,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:34,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:34,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:34,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:54:34,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:34,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:34,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:34,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:34,609 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:34,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:34,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:34,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:34,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:34,698 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:54:34,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:34,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:34,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:34,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:34,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:34,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:34,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:34,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:34,852 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:34,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:54:34,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:34,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:34,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:34,948 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:34,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:34,982 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:34,999 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:35,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:35,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:35,058 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:54:35,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:35,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:35,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:35,124 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:35,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:35,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:35,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:35,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:35,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:35,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:54:35,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:35,262 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:35,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:35,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:35,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:35,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:35,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:35,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:35,388 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 08:54:35,391 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:54:35,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:54:36,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:54:36,387 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:54:36,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:54:36,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:54:36,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:54:36,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:54:36,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:54:36,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:54:36,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:54:36,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:54:37,052 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 08:54:37,081 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:54:37,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:54:41,614 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:54:42,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:54:42,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:54:42,739 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:54:42,753 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 08:54:42,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:54:42,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:54:42,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:54:42,872 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 08:54:42,874 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:54:42,875 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 08:54:42,969 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:54:43,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 08:54:43,269 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:54:43,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 08:54:43,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 08:54:43,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 08:54:43,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 08:54:43,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 08:54:43,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 08:54:43,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 08:54:43,397 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:54:43,414 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 08:54:43,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:54:43,459 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:54:43,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:54:43,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:54:43,512 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:54:43,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:54:43,539 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 08:54:43,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:54:43,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:54:43,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:54:43,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:43,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:54:43,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:43,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:43,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:43,757 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:43,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:43,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:43,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:43,908 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:54:43,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:43,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:54:43,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:43,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:44,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:44,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:44,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:44,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:44,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:44,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:54:44,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:44,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:54:44,147 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:44,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:44,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:44,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:44,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:44,232 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:44,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:44,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:54:44,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:44,318 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:54:44,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:44,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:44,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:44,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:44,405 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:44,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:44,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:44,451 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:54:44,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:44,482 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:54:44,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:44,512 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:44,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:44,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:44,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:44,571 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:44,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:44,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:54:44,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:44,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:54:44,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:44,665 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:44,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:44,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:44,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:44,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:44,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:44,757 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:54:44,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:44,790 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:54:44,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:44,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:44,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:44,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:44,874 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:44,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:44,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:44,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:54:44,940 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 08:54:44,941 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:54:45,038 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:54:45,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:54:45,414 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:54:45,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:54:45,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:54:45,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:54:45,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:54:45,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:54:45,494 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:54:45,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:54:45,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:54:45,545 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 08:54:45,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:54:45,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 11396 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:54:51,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:54:52,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:54:52,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:54:53,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:54:53,418 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 08:54:53,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:54:53,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:54:53,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:54:53,506 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 08:54:53,506 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:54:53,509 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:54:53,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:54:54,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:54:55,213 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:54:55,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:54:55,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:54:55,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:54:55,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:54:55,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:54:55,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:54:55,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:54:55,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:54:55,667 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 08:54:55,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:54:55,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:55:02,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:55:02,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:55:03,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:55:03,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:55:03,839 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 08:55:03,839 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:55:03,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:55:03,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:55:03,953 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 08:55:03,953 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:55:03,956 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 08:55:04,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:55:04,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 08:55:04,390 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:55:04,390 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 08:55:04,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 08:55:04,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 08:55:04,451 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 08:55:04,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 08:55:04,489 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 08:55:04,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 08:55:04,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:55:04,546 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 08:55:04,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:55:04,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:55:04,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:55:04,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:55:04,655 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:55:04,676 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:55:04,696 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 08:55:04,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:55:04,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:55:04,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:55:04,747 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:04,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:55:04,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:04,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:04,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:04,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:04,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:05,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:05,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:05,117 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:05,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:05,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:55:05,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:05,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:05,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:05,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:05,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:05,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:05,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:05,362 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:05,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:05,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:55:05,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:05,443 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:05,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:05,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:05,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:05,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:05,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:05,563 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:05,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:05,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:55:05,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:05,642 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:05,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:05,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:05,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:05,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:05,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:05,747 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:05,767 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:05,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:55:05,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:05,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:05,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:05,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:05,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:05,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:05,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:05,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:05,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:05,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:55:05,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:06,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:06,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:06,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:06,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:06,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:06,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:06,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:06,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:06,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:55:06,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:06,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:06,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:06,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:06,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:06,278 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:06,298 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:06,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:06,342 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 08:55:06,345 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 08:55:06,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:55:06,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 08:55:06,720 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:55:06,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 08:55:06,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 08:55:06,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 08:55:06,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 08:55:06,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 08:55:06,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 08:55:06,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 08:55:06,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:55:06,849 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 08:55:06,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:55:06,897 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:55:06,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:55:06,928 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:55:06,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:55:06,956 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:55:06,970 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 08:55:06,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:55:06,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:55:07,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:55:07,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:07,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:55:07,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:07,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:07,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:07,206 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:07,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:07,278 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:07,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:07,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:07,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:07,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:55:07,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:07,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:07,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:07,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:07,489 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:07,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:07,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:07,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:07,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:07,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:55:07,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:07,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:07,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:07,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:07,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:07,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:07,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:07,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:07,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:07,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:55:07,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:07,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:07,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:07,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:07,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:07,852 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:07,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:07,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:07,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:07,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:55:07,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:07,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:07,969 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:07,984 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:08,002 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:08,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:08,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:08,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:08,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:08,086 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:55:08,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:08,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:08,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:08,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:08,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:08,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:08,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:08,237 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:08,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:08,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:55:08,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:08,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:08,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:08,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:08,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:08,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:08,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:08,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:08,433 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 08:55:08,434 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:55:08,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:55:08,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:55:09,069 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:55:09,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:55:09,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:55:09,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:55:09,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:55:09,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:55:09,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:55:09,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:55:09,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:55:09,216 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 08:55:09,244 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:55:09,275 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:55:15,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:55:16,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:55:16,612 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:55:17,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:55:17,079 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 08:55:17,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:55:17,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:55:17,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:55:17,201 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 08:55:17,201 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:55:17,203 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:55:17,298 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:55:17,561 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:55:17,651 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:55:17,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:55:17,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:55:17,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:55:17,719 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:55:17,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:55:17,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:55:17,778 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:55:17,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:55:17,818 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 08:55:17,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:55:17,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:55:22,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:55:23,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:55:23,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:55:23,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:55:23,973 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 08:55:23,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:55:23,994 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:55:24,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:55:24,068 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 08:55:24,069 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:55:24,072 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:55:24,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:55:25,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:55:25,328 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:55:25,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:55:25,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:55:25,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:55:25,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:55:25,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:55:25,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:55:25,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:55:25,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:55:25,486 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 08:55:25,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:55:25,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:55:30,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:55:30,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:55:31,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:55:31,438 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:55:31,464 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 08:55:31,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:55:31,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:55:31,538 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:55:31,552 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 08:55:31,553 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:55:31,555 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:55:31,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:55:31,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:55:31,950 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:55:31,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:55:31,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:55:31,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:55:31,999 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:55:32,017 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:55:32,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:55:32,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:55:32,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:55:32,086 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 08:55:32,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:55:32,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:55:40,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:55:41,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:55:41,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:55:42,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:55:42,450 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 08:55:42,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:55:42,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:55:42,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:55:42,641 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 08:55:42,641 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:55:42,644 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 08:55:42,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:55:42,956 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 08:55:43,056 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:55:43,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 08:55:43,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 08:55:43,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 08:55:43,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 08:55:43,124 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 08:55:43,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 08:55:43,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 08:55:43,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:55:43,187 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 08:55:43,216 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:55:43,232 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:55:43,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:55:43,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:55:43,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:55:43,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:55:43,308 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 08:55:43,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:55:43,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:55:43,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:55:43,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:43,390 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:55:43,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:43,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:43,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:43,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:43,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:43,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:43,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:43,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:55:43,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:43,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:55:43,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:43,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:43,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:43,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:43,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:43,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:43,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:43,920 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:55:43,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:43,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:55:43,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:43,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:44,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:44,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:44,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:44,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:44,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:44,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:55:44,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:44,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:55:44,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:44,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:44,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:44,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:44,234 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:44,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:44,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:44,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:55:44,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:44,318 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:55:44,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:44,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:44,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:44,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:44,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:44,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:44,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:44,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:55:44,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:44,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:55:44,507 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:44,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:44,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:44,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:44,599 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:44,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:44,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:44,660 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:55:44,676 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:44,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:55:44,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:44,732 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:44,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:44,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:44,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:44,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:44,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:44,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:55:44,862 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 08:55:44,864 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:55:44,865 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:55:45,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:55:45,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:55:45,371 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:55:45,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:55:45,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:55:45,405 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:55:45,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:55:45,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:55:45,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:55:45,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:55:45,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:55:46,096 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:55:46,098 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.23399996758. 2022-02-08 08:55:46,098 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:55:46,099 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:55:46,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:55:47,459 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:55:47,566 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:55:47,566 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:55:47,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:55:47,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:55:47,632 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:55:47,655 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:55:47,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:55:47,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:55:47,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:55:48,431 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:55:48,433 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.33500003815. 2022-02-08 08:55:48,433 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:55:48,434 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:55:49,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:55:49,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 5 queries submitted, 0 failed, 11420 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:55:49,792 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:55:49,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:55:49,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:55:49,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:55:49,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:55:50,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:55:50,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:55:50,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:55:50,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:55:50,911 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:55:50,911 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 08:55:52,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:55:53,111 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:55:53,111 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.67899990082. 2022-02-08 08:55:53,111 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:55:53,114 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:55:54,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:55:54,655 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:55:54,822 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:55:54,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:55:54,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:55:54,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:55:54,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:55:54,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:55:54,994 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:55:55,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:55:55,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:55:55,869 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:55:55,871 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.75900006294. 2022-02-08 08:55:55,871 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:55:55,872 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:55:56,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:55:56,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:55:56,875 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:55:56,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:55:56,895 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:55:56,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:55:56,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:55:56,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:55:56,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:55:57,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:55:57,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:55:57,638 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:55:57,640 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.76900005341. 2022-02-08 08:55:57,640 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:55:57,641 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:55:58,900 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:55:59,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:55:59,394 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:55:59,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:55:59,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:55:59,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:55:59,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:55:59,507 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:55:59,540 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:55:59,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:55:59,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:00,302 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:56:00,302 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 08:56:02,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:02,461 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:56:02,461 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.82199978828. 2022-02-08 08:56:02,463 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:56:02,464 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:56:03,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:56:03,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:56:03,740 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:56:03,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:56:03,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:56:03,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:56:03,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:56:03,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:56:03,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:56:03,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:56:03,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:04,440 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:56:04,441 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.97899985313. 2022-02-08 08:56:04,441 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:56:04,444 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:56:04,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:56:04,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:56:05,388 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:56:05,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:56:05,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:56:05,596 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:56:05,686 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:56:05,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:56:05,877 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:56:06,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:56:06,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:08,203 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:56:08,203 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 08:56:10,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:10,592 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:56:10,592 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 08:56:12,594 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:13,226 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:56:13,227 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 08:56:15,227 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:15,483 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:56:15,483 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 08:56:17,484 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:18,016 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:56:18,016 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 08:56:20,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:20,427 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:56:20,427 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 08:56:22,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:22,749 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:56:22,750 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 08:56:24,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:24,948 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:56:24,950 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 08:56:26,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:27,198 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:56:27,200 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 22.7580001354. 2022-02-08 08:56:27,200 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:56:27,201 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:56:27,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:56:28,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:56:28,642 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:56:28,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:56:28,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:56:28,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:56:28,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:56:29,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:56:29,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:56:29,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:56:29,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:30,571 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:56:30,571 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:56:32,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:33,052 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:56:33,053 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:56:35,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:35,342 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:56:35,342 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:56:37,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:37,664 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:56:37,664 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:56:39,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:39,967 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:56:39,967 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 08:56:41,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:56:42,144 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:56:42,144 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 14.9440000057. ############################################################ # # RAM FootPrint: 4591.3125 # ############################################################ 2022-02-08 08:56:42,191 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 08:56:42,191 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 67 seconds. Reason: Iteration:14 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 11442 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:57:49,197 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 08:57:49,197 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 5 queries submitted, 0 failed, 11465 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:57:54,200 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:57:55,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:57:55,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:57:56,005 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:57:56,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:57:56,025 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:57:56,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:57:56,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:57:56,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:57:56,242 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:57:56,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:57:56,375 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:57:56,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:57:56,424 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['37724 memcached\n'], []) 2022-02-08 08:57:56,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 08:57:56,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 08:57:56,505 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['37724 memcached\n'], []) 2022-02-08 08:57:56,505 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:57:56,506 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:57:57,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:57:58,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:57:58,233 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:57:58,234 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:57:58,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:57:58,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:57:58,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:57:58,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:57:58,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:57:58,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:57:58,473 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:57:58,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:57:58,552 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['90597 memcached\n'], []) 2022-02-08 08:57:58,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 08:57:58,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 08:57:58,627 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['90597 memcached\n'], []) 2022-02-08 08:57:58,627 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:57:58,628 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:57:59,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:58:00,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:58:00,556 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:58:00,556 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:58:00,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:58:00,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:58:00,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:58:00,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:58:00,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:58:00,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:58:00,721 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:58:00,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:58:00,785 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['73093 memcached\n'], []) 2022-02-08 08:58:00,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 08:58:00,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 08:58:00,888 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['73093 memcached\n'], []) 2022-02-08 08:58:00,888 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:58:00,891 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:58:01,716 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:58:02,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:58:02,668 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:58:02,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:58:02,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:58:02,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:58:02,729 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:58:02,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:58:02,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:58:02,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:58:02,844 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:58:02,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:58:02,907 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['101935 memcached\n'], []) 2022-02-08 08:58:02,908 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 08:58:02,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 08:58:02,967 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['101935 memcached\n'], []) 2022-02-08 08:58:02,967 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:58:02,970 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:58:03,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:58:04,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:58:04,677 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:58:04,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:58:04,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:58:04,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:58:04,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:58:04,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:58:04,767 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:58:04,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:58:04,838 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:58:04,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:58:04,900 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['95816 memcached\n'], []) 2022-02-08 08:58:04,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 08:58:04,928 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 08:58:04,954 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['95816 memcached\n'], []) 2022-02-08 08:58:04,956 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:58:04,957 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:58:06,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:58:06,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:58:06,783 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:58:06,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:58:06,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:58:06,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:58:06,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:58:06,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:58:06,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:58:06,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:58:06,931 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:58:06,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:58:07,038 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['130905 memcached\n'], []) 2022-02-08 08:58:07,038 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 08:58:07,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 08:58:07,085 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['130905 memcached\n'], []) 2022-02-08 08:58:07,085 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:58:07,088 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:58:08,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:58:08,658 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:58:08,848 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:58:08,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:58:08,864 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:58:08,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:58:08,946 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:58:08,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:58:08,982 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:58:09,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:58:09,023 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:58:09,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:58:09,075 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['64732 memcached\n'], []) 2022-02-08 08:58:09,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 08:58:09,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 08:58:09,121 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['64732 memcached\n'], []) 2022-02-08 08:58:09,121 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:58:09,124 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:58:09,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:58:10,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:58:10,467 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:58:10,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:58:10,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:58:10,499 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:58:10,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:58:10,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:58:10,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:58:10,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:58:10,589 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:58:10,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:58:10,647 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['79616 memcached\n'], []) 2022-02-08 08:58:10,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 08:58:10,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 08:58:10,733 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['79616 memcached\n'], []) 2022-02-08 08:58:10,733 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:58:10,736 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:58:11,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:58:11,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:58:11,992 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:58:11,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:58:12,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:58:12,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:58:12,058 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:58:12,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:58:12,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:58:12,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:58:12,163 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 08:58:12,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:58:12,221 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['15253 memcached\n'], []) 2022-02-08 08:58:12,221 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 08:58:12,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 08:58:12,275 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['15253 memcached\n'], []) 2022-02-08 08:58:12,275 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:58:12,278 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 08:58:12,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:58:12,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 08:58:12,642 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:58:12,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 08:58:12,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 08:58:12,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 08:58:12,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 08:58:12,701 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 08:58:12,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 08:58:12,729 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 08:58:12,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:58:12,757 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 08:58:12,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:58:12,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:58:18,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:58:19,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:58:20,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:58:20,581 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:58:20,595 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 08:58:20,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:58:20,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:58:20,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:58:20,684 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 08:58:20,686 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 08:58:20,687 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:58:20,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:58:20,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:58:21,059 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:58:21,061 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:58:21,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:58:21,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:58:21,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:58:21,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:58:21,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:58:21,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:58:21,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:58:21,186 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 08:58:21,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:58:21,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:58:27,517 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:58:28,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:58:28,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:58:29,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:58:29,101 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 08:58:29,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:58:29,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:58:29,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:58:29,200 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 08:58:29,200 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:58:29,201 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 08:58:29,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:58:29,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 08:58:29,578 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:58:29,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 08:58:29,594 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 08:58:29,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 08:58:29,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 08:58:29,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 08:58:29,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 08:58:29,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 08:58:29,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:58:29,700 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 08:58:29,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:58:29,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:58:29,778 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:58:29,793 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:58:29,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:58:29,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:58:29,838 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 08:58:29,839 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:58:29,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:58:29,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:58:29,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:29,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:58:29,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:30,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:30,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:30,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:30,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:30,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:30,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:30,232 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:30,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:30,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:58:30,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:30,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:30,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:30,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:30,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:30,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:30,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:30,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:30,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:30,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:58:30,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:30,512 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:30,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:30,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:30,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:30,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:30,614 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:30,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:30,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:30,665 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:58:30,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:30,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:30,716 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:30,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:30,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:30,767 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:30,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:30,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:30,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:30,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:58:30,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:30,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:30,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:30,908 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:30,926 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:30,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:30,959 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:30,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:30,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:31,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:58:31,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:31,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:31,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:31,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:31,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:31,107 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:31,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:31,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:31,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:31,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:58:31,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:31,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:31,224 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:31,242 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:31,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:31,278 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:31,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:31,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:31,335 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 08:58:31,338 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:58:31,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:58:32,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:58:32,647 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:58:32,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:58:32,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:58:32,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:58:33,081 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:58:33,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:58:33,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:58:33,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:58:33,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:58:33,960 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 08:58:34,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:58:34,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:58:38,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:58:39,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:58:39,551 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:58:39,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:58:39,852 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 08:58:39,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:58:39,877 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:58:39,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:58:39,946 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 08:58:39,947 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:58:39,948 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 08:58:40,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:58:40,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 08:58:40,302 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:58:40,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 08:58:40,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 08:58:40,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 08:58:40,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 08:58:40,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 08:58:40,375 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 08:58:40,390 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 08:58:40,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:58:40,421 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 08:58:40,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:58:40,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:58:40,484 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:58:40,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:58:40,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:58:40,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:58:40,542 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 08:58:40,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:58:40,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:58:40,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:58:40,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:40,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:58:40,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:40,630 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:40,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:40,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:40,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:40,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:40,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:40,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:58:40,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:40,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:58:40,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:40,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:40,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:40,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:40,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:40,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:40,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:40,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:58:40,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:40,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:58:40,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:40,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:40,959 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:40,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:40,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:41,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:41,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:41,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:58:41,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:41,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:58:41,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:41,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:41,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:41,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:41,168 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:41,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:41,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:41,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:58:41,234 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:41,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:58:41,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:41,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:41,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:41,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:41,334 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:41,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:41,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:41,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:58:41,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:41,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:58:41,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:41,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:41,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:41,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:41,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:41,520 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:41,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:41,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:58:41,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:41,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:58:41,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:41,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:41,632 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:41,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:41,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:41,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:41,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:41,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:58:41,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:58:41,747 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 08:58:41,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:58:41,788 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:58:41,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:58:41,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:58:41,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:58:41,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:58:41,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:58:41,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:58:41,917 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 08:58:41,917 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 08:58:42,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:58:42,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:58:42,358 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:58:42,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:58:42,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:58:42,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:58:42,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:58:42,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:58:42,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:58:42,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:58:42,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:58:42,492 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 08:58:42,517 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:58:42,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:58:49,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 5 queries submitted, 0 failed, 11487 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:58:50,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:58:50,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:58:51,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:58:51,444 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 08:58:51,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:58:51,464 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:58:51,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:58:51,565 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 08:58:51,565 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:58:51,568 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:58:51,655 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:58:51,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:58:52,006 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:58:52,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:58:52,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:58:52,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:58:52,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:58:52,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:58:52,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:58:52,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:58:52,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:58:52,163 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 08:58:52,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:58:52,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:58:58,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:58:59,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:59:00,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:59:00,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:59:00,467 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 08:59:00,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:59:00,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:59:00,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:59:00,582 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 08:59:00,582 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:59:00,585 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 08:59:00,681 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:59:00,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 08:59:01,002 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:59:01,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 08:59:01,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 08:59:01,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 08:59:01,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 08:59:01,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 08:59:01,084 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 08:59:01,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 08:59:01,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:59:01,138 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 08:59:01,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:59:01,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:59:01,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:59:01,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:59:01,234 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:59:01,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:59:01,269 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 08:59:01,270 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:59:01,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:59:01,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:59:01,334 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:01,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:59:01,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:01,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:01,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:01,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:01,443 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:01,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:01,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:01,494 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:01,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:01,533 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:59:01,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:01,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:01,596 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:01,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:01,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:01,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:01,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:01,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:01,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:01,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:59:01,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:01,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:01,806 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:01,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:01,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:01,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:01,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:01,908 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:01,928 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:01,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:59:01,971 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:01,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:02,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:02,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:02,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:02,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:02,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:02,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:02,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:02,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:59:02,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:02,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:02,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:02,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:02,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:02,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:02,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:02,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:02,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:02,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:59:02,380 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:02,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:02,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:02,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:02,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:02,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:02,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:02,517 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:02,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:02,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:59:02,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:02,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:02,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:02,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:02,658 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:02,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:02,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:02,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:02,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:02,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 08:59:02,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:02,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:02,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:02,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:02,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:02,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:02,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:02,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:02,937 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 08:59:02,940 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 08:59:03,022 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:59:03,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 08:59:03,318 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:59:03,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 08:59:03,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 08:59:03,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 08:59:03,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 08:59:03,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 08:59:03,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 08:59:03,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 08:59:03,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:59:03,443 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 08:59:03,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:59:03,494 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:59:03,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:59:03,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:59:03,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:59:03,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:59:03,569 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 08:59:03,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:59:03,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:59:03,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:59:03,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:03,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:59:03,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:03,660 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:03,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:03,693 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:03,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:03,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:03,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:03,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:03,775 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:03,793 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:59:03,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:03,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:03,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:03,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:03,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:03,900 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:03,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:03,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:03,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:03,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:59:03,987 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:04,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:04,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:04,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:04,086 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:04,107 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:04,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:04,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:04,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:04,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:59:04,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:04,214 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:04,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:04,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:04,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:04,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:04,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:04,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:04,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:04,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:59:04,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:04,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:04,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:04,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:04,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:04,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:04,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:04,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:04,512 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:04,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:59:04,543 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:04,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:04,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:04,594 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:04,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:04,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:04,642 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:04,658 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:04,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:04,688 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:59:04,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:04,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:04,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:04,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:04,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:04,788 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:04,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:04,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:04,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:04,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 08:59:04,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:04,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:04,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:04,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:04,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:04,959 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:04,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:04,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:05,010 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 08:59:05,013 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:59:05,224 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:59:05,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:59:05,680 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:59:05,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:59:05,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:59:05,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:59:05,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:59:05,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:59:05,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:59:05,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:59:05,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:59:05,917 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 08:59:05,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:59:05,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:59:12,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:59:13,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:59:13,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:59:14,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:59:14,026 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 08:59:14,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:59:14,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:59:14,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:59:14,157 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 08:59:14,157 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:59:14,158 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:59:14,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:59:14,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:59:14,581 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:59:14,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:59:14,599 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:59:14,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:59:14,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:59:14,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:59:14,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:59:14,691 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:59:14,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:59:14,733 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 08:59:14,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:59:14,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:59:19,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:59:20,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:59:20,793 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:59:21,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:59:21,173 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 08:59:21,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:59:21,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:59:21,244 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:59:21,262 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 08:59:21,263 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 08:59:21,265 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:59:21,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:59:22,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:59:22,428 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:59:22,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:59:22,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:59:22,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:59:22,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:59:22,494 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:59:22,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:59:22,538 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:59:22,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:59:22,571 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 08:59:22,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:59:22,619 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:59:27,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:59:27,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:59:28,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:59:28,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:59:28,634 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 08:59:28,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:59:28,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:59:28,698 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:59:28,713 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 08:59:28,713 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:59:28,714 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:59:29,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:59:29,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:59:29,530 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:59:29,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:59:29,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:59:29,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:59:29,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:59:29,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:59:29,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:59:29,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:59:29,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:59:29,740 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 08:59:29,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:59:29,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:59:37,086 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:59:38,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:59:38,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:59:38,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:59:39,051 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 08:59:39,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:59:39,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:59:39,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:59:39,203 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 08:59:39,203 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:59:39,206 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 08:59:39,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:59:39,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 08:59:39,579 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:59:39,579 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 08:59:39,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 08:59:39,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 08:59:39,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 08:59:39,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 08:59:39,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 08:59:39,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 08:59:39,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 08:59:39,700 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 08:59:39,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 08:59:39,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:59:39,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 08:59:39,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:59:39,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 08:59:39,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 08:59:39,821 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 08:59:39,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 08:59:39,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 08:59:39,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 08:59:39,874 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:39,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 08:59:39,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:39,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:39,936 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:39,961 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:39,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:39,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:40,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:40,022 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 08:59:40,038 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:40,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 08:59:40,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:40,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:40,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:40,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:40,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:40,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:40,183 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:40,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 08:59:40,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:40,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 08:59:40,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:40,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:40,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:40,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:40,331 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:40,349 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:40,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:40,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 08:59:40,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:40,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 08:59:40,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:40,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:40,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:40,492 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:40,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:40,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:40,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:40,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 08:59:40,581 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:40,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 08:59:40,619 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:40,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:40,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:40,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:40,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:40,706 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:40,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:40,739 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 08:59:40,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:40,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 08:59:40,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:40,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:40,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:40,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:40,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:40,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:40,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:40,920 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 08:59:40,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:40,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 08:59:40,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:40,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:41,005 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:41,020 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:41,038 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:41,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:41,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:41,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 08:59:41,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:41,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 08:59:41,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:41,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:41,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:41,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:41,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:41,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:41,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:41,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 08:59:41,282 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 08:59:41,282 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:59:41,286 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 08:59:41,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:59:41,642 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 08:59:41,726 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:59:41,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 08:59:41,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 08:59:41,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 08:59:41,775 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 08:59:41,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 08:59:41,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 08:59:41,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 08:59:41,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:59:42,400 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 08:59:42,400 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.117000103. 2022-02-08 08:59:42,400 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:59:42,401 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 08:59:43,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:59:43,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 08:59:43,464 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:59:43,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 08:59:43,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 08:59:43,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 08:59:43,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 08:59:43,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 08:59:43,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 08:59:43,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 08:59:43,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:59:44,408 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 08:59:44,408 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.00899982452. 2022-02-08 08:59:44,410 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:59:44,411 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 08:59:45,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:59:45,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 08:59:46,022 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:59:46,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 08:59:46,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 08:59:46,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 08:59:46,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 08:59:46,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 08:59:46,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 08:59:46,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 08:59:46,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:59:46,792 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 08:59:46,792 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.38199996948. 2022-02-08 08:59:46,792 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:59:46,795 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 08:59:47,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:59:48,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 08:59:48,526 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:59:48,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 08:59:48,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 08:59:48,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 08:59:48,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 08:59:48,612 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 08:59:48,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 08:59:48,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 08:59:48,698 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:59:49,640 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 08:59:49,640 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.84800004959. 2022-02-08 08:59:49,640 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:59:49,641 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 5 queries submitted, 0 failed, 11511 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 08:59:51,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:59:51,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 08:59:51,910 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:59:51,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 08:59:51,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 08:59:51,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 08:59:51,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 08:59:52,005 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 08:59:52,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 08:59:52,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 08:59:52,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:59:52,661 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 08:59:52,661 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.02199983597. 2022-02-08 08:59:52,661 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:59:52,664 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 08:59:53,278 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:59:53,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 08:59:53,730 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:59:53,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 08:59:53,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 08:59:53,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 08:59:53,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 08:59:53,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 08:59:53,877 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 08:59:53,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 08:59:53,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:59:54,562 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 08:59:54,563 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.90199995041. 2022-02-08 08:59:54,563 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:59:54,565 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 08:59:55,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:59:55,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 08:59:55,558 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:59:55,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 08:59:55,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 08:59:55,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 08:59:55,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 08:59:55,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 08:59:55,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 08:59:55,676 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 08:59:55,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:59:56,259 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 08:59:56,259 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.69600009918. 2022-02-08 08:59:56,259 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 08:59:56,262 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 08:59:56,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 08:59:57,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 08:59:57,813 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 08:59:57,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 08:59:57,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 08:59:58,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 08:59:58,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 08:59:58,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 08:59:58,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 08:59:58,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 08:59:58,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 08:59:59,591 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 08:59:59,591 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 09:00:01,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:00:01,931 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:00:01,931 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 09:00:03,933 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:00:04,319 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:00:04,319 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 09:00:06,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:00:06,869 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:00:06,869 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 09:00:08,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:00:09,096 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:00:09,096 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 09:00:11,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:00:11,289 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:00:11,290 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 15.0309998989. 2022-02-08 09:00:11,292 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:00:11,292 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:00:11,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:00:11,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:00:12,436 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:00:12,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:00:12,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:00:12,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:00:12,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:00:12,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:00:12,959 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:00:13,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:00:13,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:00:13,477 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:00:13,479 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:00:15,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:00:15,921 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:00:15,923 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:00:17,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:00:18,341 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:00:18,342 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:00:20,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:00:20,759 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:00:20,759 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:00:22,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:00:23,469 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:00:23,470 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:00:25,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:00:25,811 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:00:25,811 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:00:27,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:00:28,154 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:00:28,155 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:00:30,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:00:30,355 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:00:30,355 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 19.0640001297. ############################################################ # # RAM FootPrint: 4504.71484375 # ############################################################ 2022-02-08 09:00:30,397 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 09:00:30,397 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 65 seconds. Reason: Iteration:15 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 11536 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:01:35,403 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 09:01:35,403 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 09:01:40,405 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:01:40,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:01:41,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:01:41,446 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:01:41,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:01:41,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:01:41,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:01:41,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:01:41,512 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:01:41,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:01:41,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:01:41,572 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:01:41,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 09:01:41,634 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['38403 memcached\n'], []) 2022-02-08 09:01:41,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 09:01:41,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 09:01:41,684 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['38403 memcached\n'], []) 2022-02-08 09:01:41,684 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:01:41,687 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:01:42,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:01:42,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:01:42,957 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:01:42,959 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:01:42,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:01:43,010 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:01:43,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:01:43,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:01:43,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:01:43,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:01:43,173 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:01:43,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 09:01:43,246 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['91322 memcached\n'], []) 2022-02-08 09:01:43,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 09:01:43,278 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 09:01:43,309 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['91322 memcached\n'], []) 2022-02-08 09:01:43,309 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:01:43,312 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:01:43,987 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:01:44,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:01:44,908 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:01:44,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:01:44,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:01:44,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:01:44,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:01:45,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:01:45,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:01:45,066 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:01:45,088 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:01:45,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 09:01:45,210 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['73922 memcached\n'], []) 2022-02-08 09:01:45,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 09:01:45,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 09:01:45,290 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['73922 memcached\n'], []) 2022-02-08 09:01:45,290 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:01:45,292 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:01:46,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:01:46,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:01:47,145 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:01:47,147 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:01:47,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:01:47,196 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:01:47,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:01:47,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:01:47,290 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:01:47,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:01:47,351 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:01:47,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 09:01:47,411 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['102626 memcached\n'], []) 2022-02-08 09:01:47,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 09:01:47,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 09:01:47,470 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['102626 memcached\n'], []) 2022-02-08 09:01:47,470 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:01:47,471 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:01:48,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:01:48,691 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:01:48,910 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:01:48,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:01:48,926 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:01:48,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:01:48,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:01:49,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:01:49,022 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:01:49,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:01:49,075 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:01:49,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 09:01:49,138 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['96539 memcached\n'], []) 2022-02-08 09:01:49,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 09:01:49,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 09:01:49,190 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['96539 memcached\n'], []) 2022-02-08 09:01:49,191 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:01:49,194 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 5 queries submitted, 0 failed, 11561 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:01:50,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:01:50,681 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:01:50,835 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:01:50,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:01:50,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:01:50,867 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:01:50,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:01:50,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:01:50,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:01:50,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:01:50,993 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:01:50,994 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 09:01:51,052 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['930 memcached\n'], []) 2022-02-08 09:01:51,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 09:01:51,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 09:01:51,101 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['930 memcached\n'], []) 2022-02-08 09:01:51,101 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:01:51,105 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:01:51,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:01:52,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:01:52,569 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:01:52,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:01:52,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:01:52,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:01:52,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:01:52,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:01:52,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:01:52,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:01:52,709 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:01:52,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 09:01:52,770 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['65415 memcached\n'], []) 2022-02-08 09:01:52,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 09:01:52,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 09:01:52,822 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['65415 memcached\n'], []) 2022-02-08 09:01:52,822 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:01:52,825 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:01:53,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:01:53,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 09:01:53,818 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:01:53,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:01:53,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:01:53,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:01:53,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:01:53,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:01:53,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 09:01:53,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:01:53,940 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:01:53,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 09:01:54,000 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['80438 memcached\n'], []) 2022-02-08 09:01:54,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 09:01:54,038 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 09:01:54,062 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['80438 memcached\n'], []) 2022-02-08 09:01:54,062 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:01:54,065 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:01:54,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:01:54,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:01:55,107 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:01:55,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:01:55,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:01:55,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:01:55,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:01:55,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:01:55,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:01:55,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:01:55,246 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:01:55,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 09:01:55,318 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['16036 memcached\n'], []) 2022-02-08 09:01:55,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 09:01:55,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 09:01:55,394 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['16036 memcached\n'], []) 2022-02-08 09:01:55,394 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:01:55,401 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:01:55,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:01:55,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:01:55,778 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:01:55,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:01:55,793 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:01:55,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:01:55,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:01:55,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:01:55,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:01:55,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:01:55,885 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:01:55,901 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 09:01:55,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:01:55,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:00,885 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:02:01,655 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:02,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:02,599 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:02:02,615 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 09:02:02,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:02:02,630 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:02,701 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:02:02,729 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 09:02:02,730 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:02:02,732 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:02:02,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:02:03,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:02:03,124 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:02:03,124 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:02:03,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:02:03,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:02:03,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:02:03,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:02:03,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:02:03,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:02:03,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:02:03,249 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 09:02:03,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:02:03,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:09,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:02:10,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:10,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:10,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:02:10,930 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 09:02:10,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:02:10,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:11,022 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:02:11,039 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 09:02:11,039 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:02:11,042 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 09:02:11,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:02:11,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 09:02:11,434 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:02:11,436 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 09:02:11,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 09:02:11,464 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 09:02:11,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 09:02:11,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 09:02:11,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 09:02:11,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 09:02:11,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:02:11,562 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 09:02:11,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:02:11,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:11,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:02:11,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:11,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:11,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:02:11,680 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 09:02:11,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:02:11,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:11,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:02:11,729 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:11,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:02:11,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:11,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:11,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:11,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:11,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:11,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:11,864 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:11,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:11,897 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:11,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:02:11,936 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:11,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:11,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:11,987 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:12,005 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:12,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:12,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:12,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:12,081 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:12,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:02:12,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:12,135 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:12,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:12,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:12,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:12,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:12,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:12,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:12,278 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:12,295 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:02:12,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:12,331 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:12,346 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:12,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:12,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:12,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:12,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:12,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:12,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:12,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:02:12,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:12,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:12,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:12,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:12,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:12,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:12,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:12,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:12,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:12,660 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:02:12,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:12,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:12,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:12,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:12,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:12,762 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:12,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:12,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:12,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:12,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:02:12,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:12,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:12,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:12,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:12,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:12,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:12,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:12,961 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:12,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:12,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:02:13,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:13,030 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:13,048 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:13,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:13,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:13,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:13,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:13,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:13,181 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 09:02:13,187 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:02:13,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:02:13,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:02:13,960 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:02:13,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:02:14,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:02:14,270 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:02:14,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:02:14,492 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:02:14,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:02:14,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:02:14,670 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:02:14,700 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 09:02:14,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:02:14,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:19,489 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:02:20,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:20,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:21,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:02:21,269 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 09:02:21,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:02:21,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:21,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:02:21,624 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 09:02:21,625 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:02:21,627 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 09:02:21,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:02:21,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 09:02:22,009 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:02:22,010 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 09:02:22,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 09:02:22,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 09:02:22,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 09:02:22,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 09:02:22,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 09:02:22,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 09:02:22,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:02:22,134 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 09:02:22,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:02:22,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:22,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:02:22,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:22,216 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:22,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:02:22,243 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 09:02:22,244 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:02:22,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:22,270 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:02:22,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:22,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:02:22,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:22,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:22,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:22,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:22,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:22,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:22,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:22,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:22,464 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:22,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:02:22,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:22,543 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:22,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:22,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:22,612 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:22,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:22,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:22,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:22,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:22,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:02:22,719 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:22,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:22,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:22,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:22,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:22,806 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:22,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:22,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:22,859 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:22,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:02:22,895 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:22,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:22,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:22,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:22,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:22,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:23,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:23,035 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:23,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:23,066 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:02:23,084 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:23,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:23,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:23,135 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:23,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:23,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:23,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:23,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:23,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:23,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:02:23,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:23,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:23,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:23,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:23,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:23,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:23,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:23,367 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:23,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:23,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:02:23,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:23,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:23,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:23,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:23,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:23,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:23,517 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:23,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:23,548 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:23,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:02:23,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:23,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:23,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:23,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:23,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:23,676 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:23,691 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:23,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:23,727 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 09:02:23,730 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:02:23,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:02:24,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:02:24,122 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:02:24,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:02:24,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:02:24,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:02:24,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:02:24,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:02:24,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:02:24,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:02:24,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:02:24,250 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 09:02:24,275 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:02:24,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:30,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:02:31,484 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:31,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:32,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:02:32,380 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 09:02:32,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:02:32,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:32,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:02:32,463 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 09:02:32,463 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:02:32,466 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:02:32,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:02:33,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:02:33,526 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:02:33,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:02:33,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:02:33,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:02:33,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:02:33,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:02:33,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:02:33,673 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:02:33,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:02:33,709 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 09:02:33,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:02:33,752 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:40,729 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:02:41,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:41,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:42,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:02:42,289 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 09:02:42,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:02:42,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:42,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:02:42,459 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 09:02:42,459 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:02:42,460 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 09:02:42,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:02:42,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 09:02:42,888 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:02:42,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 09:02:42,908 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 09:02:42,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 09:02:42,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 09:02:42,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 09:02:42,971 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 09:02:42,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 09:02:43,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:02:43,020 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 09:02:43,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:02:43,061 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:43,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:02:43,089 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:43,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:43,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:02:43,128 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 09:02:43,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:02:43,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:43,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:02:43,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:43,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:02:43,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:43,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:43,244 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:43,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:43,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:43,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:43,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:43,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:43,349 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:43,367 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:02:43,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:43,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:43,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:43,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:43,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:43,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:43,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:43,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:43,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:43,551 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:02:43,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:43,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:43,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:43,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:43,642 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:43,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:43,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:43,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:43,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:43,729 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:02:43,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:43,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:43,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:43,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:43,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:43,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:43,859 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:43,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:43,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:43,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:02:43,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:43,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:43,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:43,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:44,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:44,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:44,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:44,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:44,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:44,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:02:44,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:44,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:44,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:44,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:44,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:44,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:44,224 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:44,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:44,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:44,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:02:44,290 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:44,306 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:44,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:44,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:44,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:44,375 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:44,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:44,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:44,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:44,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:02:44,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:44,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:44,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:44,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:44,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:44,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:44,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:44,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:44,630 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 09:02:44,632 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 09:02:44,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:02:44,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 09:02:45,045 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:02:45,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 09:02:45,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 09:02:45,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 09:02:45,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 09:02:45,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 09:02:45,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 09:02:45,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 09:02:45,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:02:45,176 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 09:02:45,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:02:45,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:45,237 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:02:45,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:45,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:45,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:02:45,298 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 09:02:45,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:02:45,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:45,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:02:45,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:45,364 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:02:45,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:45,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:45,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:45,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:45,489 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:45,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:45,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:45,540 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:02:45,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:45,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:02:45,596 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:45,614 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:45,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:45,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:45,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:45,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:45,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:45,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:02:45,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:45,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:02:45,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:45,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:45,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:45,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:45,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:45,864 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:45,880 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:45,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:02:45,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:45,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:02:45,948 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:45,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:45,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:46,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:46,017 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:46,035 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:46,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:46,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:02:46,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:46,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:02:46,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:46,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:46,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:46,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:46,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:46,206 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:46,221 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:46,237 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:02:46,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:46,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:02:46,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:46,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:46,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:46,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:46,349 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:46,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:46,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:46,395 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:02:46,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:46,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:02:46,443 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:46,459 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:46,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:46,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:46,505 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:46,520 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:46,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:46,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:02:46,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:46,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:02:46,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:46,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:46,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:46,658 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:46,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:46,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:46,716 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:46,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:02:46,753 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 09:02:46,756 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:02:47,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:02:47,829 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:02:47,997 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:02:47,999 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:02:48,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:02:48,038 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:02:48,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:02:48,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:02:48,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:02:48,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:02:48,137 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:02:48,155 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 09:02:48,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:02:48,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 11581 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:02:54,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:02:55,025 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:55,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:02:55,908 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:02:55,937 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 09:02:55,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:02:55,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:02:56,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:02:56,045 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 09:02:56,046 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:02:56,048 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:02:56,137 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:02:56,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 09:02:56,500 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:02:56,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:02:56,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:02:56,540 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:02:56,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:02:56,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:02:56,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 09:02:56,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:02:56,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:02:56,674 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 09:02:56,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:02:56,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:03:01,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:03:02,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:03:02,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:03:03,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:03:03,240 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 09:03:03,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:03:03,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:03:03,306 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:03:03,326 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 09:03:03,326 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:03:03,329 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:03:04,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:03:04,367 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:03:04,467 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:03:04,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:03:04,482 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:03:04,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:03:04,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:03:04,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:03:04,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:03:04,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:03:04,614 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:03:04,630 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 09:03:04,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:03:04,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:03:09,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:03:10,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:03:10,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:03:10,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:03:10,694 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 09:03:10,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:03:10,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:03:10,757 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:03:10,773 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 09:03:10,773 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:03:10,776 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 09:03:10,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:03:11,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:03:11,476 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:03:11,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:03:11,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:03:11,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:03:11,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:03:11,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:03:11,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:03:11,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:03:11,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:03:11,717 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 09:03:11,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:03:11,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:03:19,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:03:20,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:03:21,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:03:21,790 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:03:21,858 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 09:03:21,859 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:03:21,880 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:03:21,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:03:22,046 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 09:03:22,046 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:03:22,049 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 09:03:22,135 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:03:22,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 09:03:22,440 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:03:22,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 09:03:22,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 09:03:22,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 09:03:22,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 09:03:22,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 09:03:22,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 09:03:22,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 09:03:22,548 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:03:22,563 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 09:03:22,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:03:22,673 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:03:22,691 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:03:22,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:03:22,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:03:22,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:03:22,753 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 09:03:22,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:03:22,778 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:03:22,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:03:22,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:03:22,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:03:22,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:03:22,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:03:22,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:03:22,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:03:22,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:03:22,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:03:22,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:03:22,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:03:23,005 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:03:23,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:03:23,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:03:23,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:03:23,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:03:23,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:03:23,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:03:23,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:03:23,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:03:23,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:03:23,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:03:23,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:03:23,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:03:23,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:03:23,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:03:23,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:03:23,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:03:23,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:03:23,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:03:23,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:03:23,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:03:23,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:03:23,405 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:03:23,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:03:23,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:03:23,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:03:23,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:03:23,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:03:23,517 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:03:23,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:03:23,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:03:23,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:03:23,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:03:23,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:03:23,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:03:23,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:03:23,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:03:23,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:03:23,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:03:23,732 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:03:23,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:03:23,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:03:23,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:03:23,806 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:03:23,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:03:23,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:03:23,859 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:03:23,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:03:23,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:03:23,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:03:23,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:03:23,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:03:23,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:03:23,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:03:23,994 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:03:24,010 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:03:24,030 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:03:24,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:03:24,063 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:03:24,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:03:24,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:03:24,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:03:24,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:03:24,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:03:24,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:03:24,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:03:24,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:03:24,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:03:24,244 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:03:24,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:03:24,286 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 09:03:24,286 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:03:24,289 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:03:24,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:03:24,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:03:24,710 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:03:24,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:03:24,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:03:24,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:03:24,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:03:24,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:03:24,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:03:24,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:03:24,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:03:25,338 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:03:25,339 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.0529999733. 2022-02-08 09:03:25,339 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:03:25,342 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:03:26,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:03:26,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:03:26,903 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:03:26,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:03:26,926 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:03:26,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:03:27,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:03:27,038 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:03:27,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:03:27,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:03:27,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:03:27,759 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:03:27,759 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.42000007629. 2022-02-08 09:03:27,759 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:03:27,762 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:03:28,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:03:29,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:03:29,438 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:03:29,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:03:29,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:03:29,494 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:03:29,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:03:29,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:03:29,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:03:29,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:03:29,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:03:30,309 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:03:30,309 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.54999995232. 2022-02-08 09:03:30,309 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:03:30,312 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:03:31,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:03:31,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:03:32,069 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:03:32,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:03:32,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:03:32,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:03:32,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:03:32,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:03:32,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:03:32,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:03:32,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:03:32,943 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:03:32,943 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.63300013542. 2022-02-08 09:03:32,944 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:03:32,944 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:03:33,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:03:34,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:03:34,183 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:03:34,183 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:03:34,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:03:34,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:03:34,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:03:34,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:03:34,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:03:34,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:03:34,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:03:35,105 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:03:35,105 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.127 2022-02-08 09:03:37,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:03:37,288 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:03:37,288 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.34400010109. 2022-02-08 09:03:37,288 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:03:37,289 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 09:03:37,946 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:03:38,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:03:38,569 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:03:38,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:03:38,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:03:38,630 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:03:38,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:03:38,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:03:38,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:03:38,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:03:38,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:03:40,043 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:03:40,043 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 09:03:42,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:03:42,230 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:03:42,230 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 09:03:44,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:03:44,430 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:03:44,430 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.14199995995. 2022-02-08 09:03:44,430 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:03:44,431 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:03:45,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:03:45,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:03:45,786 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:03:45,788 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:03:45,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:03:45,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:03:45,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:03:45,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:03:45,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:03:45,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:03:45,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:03:46,441 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:03:46,443 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.01300001144. 2022-02-08 09:03:46,443 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:03:46,444 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:03:47,252 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:03:47,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 09:03:47,811 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:03:47,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:03:47,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:03:48,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:03:48,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:03:48,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:03:48,331 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 09:03:48,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:03:48,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:03:49,275 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:03:49,275 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 11608 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:03:51,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:03:51,566 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:03:51,566 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 09:03:53,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:03:53,769 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:03:53,770 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.32799983025. 2022-02-08 09:03:53,770 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:03:53,775 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:03:54,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:03:55,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:03:55,773 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:03:55,775 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:03:55,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:03:55,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:03:55,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:03:55,900 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:03:55,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:03:55,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:03:55,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:03:56,776 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:03:56,778 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.0070002079. ############################################################ # # RAM FootPrint: 4223.66796875 # ############################################################ 2022-02-08 09:03:56,825 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 09:03:56,825 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 92 seconds. Reason: Iteration:16 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 11631 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:05:28,832 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 09:05:28,832 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 09:05:33,835 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:05:34,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:05:35,451 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:05:35,677 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:05:35,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:05:35,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:05:35,762 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:05:35,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:05:35,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:05:35,900 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:05:35,928 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:05:35,954 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:05:35,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 09:05:36,012 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['39127 memcached\n'], []) 2022-02-08 09:05:36,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 09:05:36,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 09:05:36,062 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['39127 memcached\n'], []) 2022-02-08 09:05:36,062 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:05:36,066 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:05:36,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:05:36,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:05:36,882 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:05:36,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:05:36,903 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:05:36,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:05:36,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:05:36,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:05:36,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:05:37,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:05:37,029 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:05:37,030 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 09:05:37,112 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['92048 memcached\n'], []) 2022-02-08 09:05:37,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 09:05:37,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 09:05:37,216 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['92048 memcached\n'], []) 2022-02-08 09:05:37,217 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:05:37,219 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:05:38,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:05:38,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:05:38,776 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:05:38,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:05:38,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:05:38,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:05:38,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:05:38,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:05:38,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:05:38,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:05:38,963 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:05:38,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 09:05:39,118 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['74699 memcached\n'], []) 2022-02-08 09:05:39,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 09:05:39,183 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 09:05:39,214 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['74699 memcached\n'], []) 2022-02-08 09:05:39,216 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:05:39,219 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:05:40,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:05:41,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:05:41,242 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:05:41,242 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:05:41,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:05:41,290 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:05:41,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:05:41,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:05:41,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:05:41,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:05:41,420 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:05:41,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 09:05:41,483 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['103348 memcached\n'], []) 2022-02-08 09:05:41,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 09:05:41,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 09:05:41,540 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['103348 memcached\n'], []) 2022-02-08 09:05:41,540 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:05:41,542 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:05:42,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:05:42,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:05:43,165 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:05:43,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:05:43,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:05:43,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:05:43,362 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:05:43,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:05:43,405 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:05:43,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:05:43,466 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:05:43,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 09:05:43,532 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['97219 memcached\n'], []) 2022-02-08 09:05:43,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 09:05:43,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 09:05:43,585 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['97219 memcached\n'], []) 2022-02-08 09:05:43,585 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:05:43,588 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 09:05:45,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:05:45,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:05:46,039 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:05:46,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:05:46,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:05:46,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:05:46,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:05:46,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:05:46,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:05:46,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:05:46,161 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:05:46,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 09:05:46,221 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['1666 memcached\n'], []) 2022-02-08 09:05:46,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 09:05:46,262 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 09:05:46,283 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['1666 memcached\n'], []) 2022-02-08 09:05:46,285 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:05:46,286 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:05:47,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:05:47,719 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:05:47,924 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:05:47,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:05:47,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:05:47,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:05:47,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:05:48,002 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:05:48,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:05:48,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:05:48,059 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:05:48,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 09:05:48,105 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['66143 memcached\n'], []) 2022-02-08 09:05:48,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 09:05:48,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 09:05:48,157 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['66143 memcached\n'], []) 2022-02-08 09:05:48,157 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:05:48,160 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:05:49,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:05:49,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 5 queries submitted, 0 failed, 11656 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:05:49,762 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:05:49,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:05:49,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:05:49,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:05:49,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:05:49,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:05:49,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 09:05:49,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:05:49,931 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:05:49,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 09:05:50,029 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['81191 memcached\n'], []) 2022-02-08 09:05:50,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 09:05:50,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 09:05:50,084 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['81191 memcached\n'], []) 2022-02-08 09:05:50,085 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:05:50,085 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:05:51,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:05:52,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:05:52,457 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:05:52,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:05:52,471 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:05:52,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:05:52,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:05:52,609 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:05:52,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:05:52,655 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:05:52,671 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:05:52,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 09:05:52,783 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['16847 memcached\n'], []) 2022-02-08 09:05:52,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 09:05:52,823 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 09:05:52,845 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['16847 memcached\n'], []) 2022-02-08 09:05:52,845 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:05:52,846 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:05:52,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:05:53,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:05:53,247 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:05:53,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:05:53,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:05:53,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:05:53,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:05:53,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:05:53,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:05:53,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:05:53,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:05:53,368 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 09:05:53,392 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:05:53,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:00,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:06:01,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:01,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:02,005 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:06:02,019 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 09:06:02,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:06:02,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:02,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:06:02,111 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 09:06:02,111 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:06:02,114 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:06:02,221 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:06:02,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:06:02,520 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:06:02,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:06:02,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:06:02,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:06:02,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:06:02,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:06:02,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:06:02,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:06:02,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:06:02,631 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 09:06:02,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:06:02,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:09,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:06:09,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:10,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:10,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:06:10,716 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 09:06:10,716 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:06:10,729 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:10,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:06:10,815 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 09:06:10,815 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:06:10,818 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 09:06:10,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:06:11,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 09:06:11,207 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:06:11,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 09:06:11,221 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 09:06:11,234 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 09:06:11,252 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 09:06:11,270 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 09:06:11,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 09:06:11,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 09:06:11,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:06:11,329 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 09:06:11,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:06:11,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:11,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:06:11,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:11,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:11,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:06:11,444 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 09:06:11,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:06:11,459 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:11,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:06:11,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:11,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:06:11,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:11,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:11,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:11,581 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:11,599 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:11,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:11,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:11,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:11,670 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:11,686 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:06:11,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:11,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:11,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:11,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:11,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:11,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:11,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:11,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:11,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:11,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:06:11,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:11,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:11,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:11,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:11,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:11,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:11,987 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:12,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:12,025 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:12,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:06:12,061 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:12,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:12,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:12,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:12,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:12,147 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:12,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:12,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:12,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:12,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:06:12,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:12,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:12,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:12,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:12,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:12,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:12,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:12,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:12,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:12,385 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:06:12,405 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:12,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:12,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:12,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:12,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:12,494 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:12,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:12,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:12,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:12,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:06:12,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:12,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:12,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:12,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:12,673 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:12,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:12,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:12,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:12,747 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:12,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:06:12,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:12,806 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:12,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:12,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:12,864 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:12,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:12,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:12,928 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:12,946 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 09:06:12,947 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:06:13,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:06:13,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:06:13,615 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:06:13,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:06:13,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:06:13,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:06:13,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:06:13,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:06:13,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:06:13,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:06:14,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:06:14,105 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 09:06:14,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:06:14,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:19,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:06:19,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:20,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:20,520 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:06:20,589 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 09:06:20,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:06:20,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:20,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:06:20,743 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 09:06:20,743 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:06:20,746 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 09:06:20,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:06:21,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 09:06:21,144 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:06:21,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 09:06:21,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 09:06:21,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 09:06:21,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 09:06:21,206 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 09:06:21,221 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 09:06:21,237 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 09:06:21,252 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:06:21,269 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 09:06:21,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:06:21,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:21,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:06:21,339 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:21,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:21,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:06:21,381 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 09:06:21,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:06:21,395 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:21,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:06:21,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:21,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:06:21,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:21,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:21,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:21,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:21,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:21,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:21,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:21,619 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:21,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:21,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:06:21,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:21,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:21,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:21,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:21,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:21,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:21,778 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:21,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:21,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:21,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:06:21,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:21,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:21,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:21,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:21,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:21,948 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:21,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:21,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:22,002 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:22,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:06:22,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:22,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:22,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:22,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:22,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:22,124 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:22,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:22,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:22,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:22,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:06:22,211 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:22,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:22,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:22,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:22,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:22,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:22,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:22,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:22,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:22,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:06:22,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:22,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:22,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:22,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:22,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:22,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:22,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:22,505 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:22,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:22,538 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:06:22,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:22,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:22,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:22,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:22,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:22,647 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:22,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:22,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:22,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:22,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:06:22,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:22,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:22,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:22,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:22,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:22,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:22,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:22,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:22,894 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 09:06:22,895 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:06:22,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:06:23,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:06:23,269 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:06:23,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:06:23,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:06:23,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:06:23,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:06:23,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:06:23,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:06:23,367 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:06:23,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:06:23,401 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 09:06:23,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:06:23,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:29,221 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:06:30,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:31,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:31,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:06:31,770 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 09:06:31,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:06:31,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:31,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:06:32,019 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 09:06:32,020 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:06:32,023 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:06:32,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:06:32,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:06:32,551 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:06:32,551 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:06:32,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:06:32,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:06:32,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:06:32,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:06:32,647 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:06:32,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:06:32,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:06:32,709 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 09:06:32,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:06:32,752 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:39,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:06:39,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:40,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:40,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:06:40,776 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 09:06:40,778 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:06:40,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:40,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:06:40,936 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 09:06:40,937 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:06:40,938 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 09:06:41,048 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:06:41,290 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 09:06:41,355 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:06:41,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 09:06:41,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 09:06:41,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 09:06:41,395 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 09:06:41,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 09:06:41,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 09:06:41,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 09:06:41,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:06:41,469 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 09:06:41,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:06:41,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:41,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:06:41,543 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:41,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:41,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:06:41,589 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 09:06:41,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:06:41,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:41,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:06:41,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:41,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:06:41,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:41,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:41,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:41,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:41,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:41,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:41,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:41,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:41,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:41,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:06:41,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:41,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:41,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:41,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:41,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:42,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:42,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:42,058 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:42,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:42,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:06:42,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:42,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:42,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:42,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:42,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:42,227 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:42,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:42,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:42,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:42,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:06:42,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:42,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:42,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:42,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:42,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:42,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:42,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:42,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:42,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:42,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:06:42,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:42,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:42,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:42,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:42,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:42,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:42,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:42,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:42,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:42,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:06:42,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:42,724 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:42,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:42,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:42,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:42,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:42,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:42,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:42,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:42,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:06:42,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:42,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:42,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:42,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:42,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:42,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:43,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:43,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:43,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:43,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:06:43,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:43,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:43,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:43,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:43,147 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:43,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:43,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:43,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:43,226 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 09:06:43,227 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 09:06:43,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:06:43,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 09:06:43,586 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:06:43,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 09:06:43,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 09:06:43,619 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 09:06:43,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 09:06:43,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 09:06:43,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 09:06:43,686 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 09:06:43,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:06:43,717 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 09:06:43,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:06:43,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:43,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:06:43,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:43,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:43,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:06:43,832 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 09:06:43,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:06:43,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:43,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:06:43,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:43,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:06:43,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:43,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:43,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:43,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:44,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:44,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:44,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:44,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:06:44,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:44,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:06:44,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:44,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:44,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:44,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:44,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:44,214 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:44,232 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:44,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:06:44,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:44,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:06:44,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:44,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:44,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:44,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:44,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:44,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:44,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:44,433 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:06:44,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:44,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:06:44,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:44,505 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:44,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:44,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:44,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:44,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:44,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:44,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:06:44,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:44,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:06:44,658 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:44,676 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:44,691 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:44,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:44,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:44,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:44,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:44,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:06:44,788 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:44,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:06:44,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:44,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:44,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:44,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:44,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:44,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:44,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:44,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:06:44,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:44,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:06:45,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:45,020 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:45,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:45,066 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:45,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:45,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:45,114 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:45,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:06:45,147 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:45,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:06:45,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:45,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:45,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:45,242 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:45,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:45,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:45,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:45,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:06:45,326 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 09:06:45,329 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:06:45,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:06:45,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:06:45,884 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:06:45,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:06:45,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:06:45,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:06:45,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:06:45,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:06:45,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:06:45,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:06:46,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:06:46,029 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 09:06:46,058 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:06:46,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 11682 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:06:51,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:06:52,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:53,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:06:53,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:06:53,859 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 09:06:53,859 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:06:53,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:53,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:06:53,960 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 09:06:53,960 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:06:53,963 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:06:54,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:06:54,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 09:06:54,387 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:06:54,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:06:54,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:06:54,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:06:54,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:06:54,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:06:54,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 09:06:54,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:06:54,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:06:54,539 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 09:06:54,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:06:54,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:06:59,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:07:00,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:07:00,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:07:00,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:07:00,957 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 09:07:00,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:07:00,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:07:01,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:07:01,059 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 09:07:01,059 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:07:01,062 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:07:01,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:07:02,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:07:02,249 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:07:02,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:07:02,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:07:02,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:07:02,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:07:02,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:07:02,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:07:02,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:07:02,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:07:02,398 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 09:07:02,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:07:02,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:07:07,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:07:07,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:07:08,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:07:08,612 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:07:08,642 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 09:07:08,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:07:08,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:07:08,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:07:08,730 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 09:07:08,730 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:07:08,733 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 09:07:08,880 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:07:09,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:07:09,298 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:07:09,298 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:07:09,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:07:09,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:07:09,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:07:09,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:07:09,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:07:09,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:07:09,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:07:09,483 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 09:07:09,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:07:09,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:07:17,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:07:18,211 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:07:18,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:07:19,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:07:19,322 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 09:07:19,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:07:19,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:07:19,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:07:19,420 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 09:07:19,421 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:07:19,423 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 09:07:19,522 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:07:19,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 09:07:19,834 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:07:19,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 09:07:19,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 09:07:19,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 09:07:19,877 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 09:07:19,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 09:07:19,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 09:07:19,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 09:07:19,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:07:19,956 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 09:07:19,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:07:19,999 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:07:20,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:07:20,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:07:20,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:07:20,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:07:20,075 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 09:07:20,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:07:20,089 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:07:20,107 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:07:20,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:07:20,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:07:20,168 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:07:20,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:07:20,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:07:20,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:07:20,255 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:07:20,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:07:20,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:07:20,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:07:20,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:07:20,362 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:07:20,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:07:20,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:07:20,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:07:20,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:07:20,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:07:20,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:07:20,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:07:20,507 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:07:20,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:07:20,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:07:20,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:07:20,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:07:20,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:07:20,619 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:07:20,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:07:20,655 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:07:20,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:07:20,693 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:07:20,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:07:20,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:07:20,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:07:20,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:07:20,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:07:20,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:07:20,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:07:20,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:07:20,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:07:20,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:07:20,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:07:20,903 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:07:20,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:07:20,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:07:20,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:07:20,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:07:20,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:07:21,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:07:21,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:07:21,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:07:21,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:07:21,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:07:21,107 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:07:21,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:07:21,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:07:21,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:07:21,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:07:21,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:07:21,211 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:07:21,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:07:21,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:07:21,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:07:21,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:07:21,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:07:21,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:07:21,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:07:21,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:07:21,362 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:07:21,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:07:21,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:07:21,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:07:21,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:07:21,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:07:21,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:07:21,492 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:07:21,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:07:21,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:07:21,548 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:07:21,566 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:07:21,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:07:21,602 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 09:07:21,602 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:07:21,605 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:07:21,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:07:21,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:07:21,990 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:07:21,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:07:22,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:07:22,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:07:22,048 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:07:22,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:07:22,089 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:07:22,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:07:22,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:07:22,638 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:07:22,638 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.03500008583. 2022-02-08 09:07:22,638 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:07:22,640 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:07:23,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:07:23,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:07:23,697 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:07:23,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:07:23,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:07:23,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:07:23,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:07:23,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:07:23,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:07:23,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:07:23,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:07:24,509 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:07:24,510 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.87299990654. 2022-02-08 09:07:24,510 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:07:24,512 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:07:25,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:07:25,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:07:25,951 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:07:25,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:07:25,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:07:26,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:07:26,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:07:26,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:07:26,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:07:26,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:07:26,132 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:07:26,802 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:07:26,803 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.29299998283. 2022-02-08 09:07:26,803 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:07:26,805 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:07:27,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:07:28,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:07:28,296 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:07:28,298 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:07:28,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:07:28,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:07:28,390 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:07:28,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:07:28,443 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:07:28,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:07:28,520 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:07:29,266 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:07:29,266 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 09:07:31,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:07:31,450 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:07:31,450 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.64700007439. 2022-02-08 09:07:31,450 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:07:31,453 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:07:32,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:07:32,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:07:32,694 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:07:32,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:07:32,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:07:32,747 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:07:32,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:07:32,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:07:32,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:07:32,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:07:32,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:07:33,500 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:07:33,500 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.0490000248. 2022-02-08 09:07:33,500 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:07:33,503 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 09:07:34,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:07:34,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:07:34,690 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:07:34,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:07:34,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:07:34,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:07:34,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:07:34,778 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:07:34,793 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:07:34,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:07:34,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:07:35,431 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:07:35,431 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.93099999428. 2022-02-08 09:07:35,433 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:07:35,434 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:07:36,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:07:36,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:07:36,752 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:07:36,752 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:07:36,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:07:36,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:07:36,806 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:07:36,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:07:36,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:07:36,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:07:36,885 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:07:37,415 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:07:37,417 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.98399996758. 2022-02-08 09:07:37,417 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:07:37,417 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:07:37,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:07:37,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 09:07:37,917 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:07:37,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:07:37,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:07:37,971 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:07:38,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:07:38,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:07:38,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 09:07:38,073 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:07:38,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:07:39,167 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:07:39,167 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 09:07:41,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:07:41,428 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:07:41,430 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 09:07:43,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:07:43,694 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:07:43,696 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 09:07:45,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:07:46,328 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:07:46,328 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 09:07:48,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:07:48,631 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:07:48,631 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 5 queries submitted, 0 failed, 11707 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:07:50,632 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:07:50,956 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:07:50,956 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 09:07:52,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:07:53,460 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:07:53,460 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 09:07:55,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:07:55,802 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:07:55,802 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 09:07:57,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:07:58,092 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:07:58,092 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 09:08:00,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:08:00,293 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:08:00,295 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 22.878000021. 2022-02-08 09:08:00,295 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:08:00,296 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:08:01,168 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:08:01,688 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:08:02,111 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:08:02,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:08:02,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:08:02,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:08:02,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:08:02,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:08:02,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:08:02,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:08:02,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:08:03,960 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:08:03,960 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:08:05,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:08:06,197 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:08:06,197 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:08:08,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:08:08,638 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:08:08,638 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:08:10,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:08:10,940 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:08:10,940 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:08:12,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:08:14,769 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:08:14,769 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 14.4739999771. ############################################################ # # RAM FootPrint: 4482.5546875 # ############################################################ 2022-02-08 09:08:14,802 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 09:08:14,803 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 98 seconds. Reason: Iteration:17 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 11730 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 11750 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:09:52,809 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 09:09:52,809 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 09:09:57,812 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:09:58,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:09:59,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:09:59,545 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:09:59,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:09:59,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:09:59,665 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:09:59,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:09:59,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:09:59,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:09:59,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:09:59,920 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:09:59,920 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 09:09:59,980 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['39849 memcached\n'], []) 2022-02-08 09:09:59,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 09:10:00,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 09:10:00,075 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['39849 memcached\n'], []) 2022-02-08 09:10:00,075 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:10:00,078 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:10:01,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:10:01,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:10:02,098 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:10:02,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:10:02,124 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:10:02,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:10:02,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:10:02,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:10:02,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:10:02,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:10:02,361 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:10:02,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 09:10:02,506 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['92721 memcached\n'], []) 2022-02-08 09:10:02,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 09:10:02,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 09:10:02,568 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['92721 memcached\n'], []) 2022-02-08 09:10:02,569 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:10:02,571 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:10:03,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:10:04,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:10:04,407 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:10:04,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:10:04,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:10:04,464 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:10:04,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:10:04,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:10:04,538 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:10:04,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:10:04,653 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:10:04,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 09:10:04,721 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['75515 memcached\n'], []) 2022-02-08 09:10:04,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 09:10:04,762 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 09:10:04,792 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['75515 memcached\n'], []) 2022-02-08 09:10:04,792 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:10:04,795 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:10:05,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:10:06,505 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:10:06,743 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:10:06,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:10:06,775 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:10:06,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:10:06,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:10:06,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:10:06,852 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:10:06,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:10:06,901 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:10:06,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 09:10:06,973 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['104067 memcached\n'], []) 2022-02-08 09:10:06,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 09:10:07,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 09:10:07,029 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['104067 memcached\n'], []) 2022-02-08 09:10:07,029 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:10:07,032 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:10:07,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:10:08,742 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:10:08,921 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:10:08,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:10:08,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:10:08,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:10:08,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:10:09,017 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:10:09,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:10:09,061 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:10:09,078 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:10:09,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 09:10:09,150 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['97954 memcached\n'], []) 2022-02-08 09:10:09,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 09:10:09,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 09:10:09,203 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['97954 memcached\n'], []) 2022-02-08 09:10:09,203 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:10:09,206 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 09:10:09,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:10:10,867 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:10:11,062 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:10:11,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:10:11,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:10:11,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:10:11,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:10:11,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:10:11,196 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:10:11,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:10:11,234 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:10:11,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 09:10:11,309 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['2374 memcached\n'], []) 2022-02-08 09:10:11,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 09:10:11,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 09:10:11,358 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['2374 memcached\n'], []) 2022-02-08 09:10:11,358 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:10:11,361 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:10:11,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:10:12,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:10:12,512 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:10:12,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:10:12,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:10:12,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:10:12,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:10:12,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:10:12,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:10:12,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:10:12,644 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:10:12,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 09:10:12,713 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['66862 memcached\n'], []) 2022-02-08 09:10:12,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 09:10:12,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 09:10:12,772 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['66862 memcached\n'], []) 2022-02-08 09:10:12,773 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:10:12,775 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:10:13,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:10:13,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 09:10:14,109 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:10:14,109 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:10:14,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:10:14,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:10:14,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:10:14,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:10:14,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 09:10:14,214 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:10:14,232 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:10:14,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 09:10:14,285 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['81940 memcached\n'], []) 2022-02-08 09:10:14,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 09:10:14,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 09:10:14,334 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['81940 memcached\n'], []) 2022-02-08 09:10:14,335 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:10:14,336 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:10:14,612 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:10:14,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:10:15,105 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:10:15,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:10:15,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:10:15,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:10:15,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:10:15,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:10:15,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:10:15,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:10:15,463 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:10:15,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 09:10:15,598 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['17572 memcached\n'], []) 2022-02-08 09:10:15,599 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 09:10:15,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 09:10:15,654 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['17572 memcached\n'], []) 2022-02-08 09:10:15,654 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:10:15,657 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:10:15,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:10:15,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:10:16,049 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:10:16,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:10:16,063 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:10:16,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:10:16,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:10:16,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:10:16,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:10:16,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:10:16,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:10:16,164 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 09:10:16,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:10:16,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:10:22,035 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:10:22,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:10:23,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:10:23,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:10:23,739 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 09:10:23,739 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:10:23,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:10:23,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:10:23,835 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 09:10:23,835 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:10:23,838 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:10:23,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:10:24,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:10:24,197 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:10:24,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:10:24,211 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:10:24,224 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:10:24,237 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:10:24,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:10:24,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:10:24,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:10:24,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:10:24,309 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 09:10:24,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:10:24,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:10:30,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:10:31,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:10:32,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:10:32,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:10:32,496 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 09:10:32,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:10:32,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:10:32,579 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:10:32,592 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 09:10:32,594 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:10:32,596 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 09:10:32,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:10:32,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 09:10:32,986 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:10:32,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 09:10:33,002 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 09:10:33,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 09:10:33,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 09:10:33,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 09:10:33,061 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 09:10:33,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 09:10:33,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:10:33,111 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 09:10:33,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:10:33,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:10:33,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:10:33,216 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:10:33,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:10:33,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:10:33,256 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 09:10:33,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:10:33,270 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:10:33,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:10:33,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:33,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:10:33,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:33,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:33,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:33,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:33,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:33,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:33,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:33,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:33,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:33,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:10:33,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:33,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:33,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:33,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:33,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:33,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:33,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:33,688 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:33,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:33,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:10:33,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:33,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:33,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:33,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:33,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:33,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:33,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:33,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:33,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:33,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:10:33,926 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:33,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:33,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:33,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:34,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:34,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:34,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:34,063 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:34,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:34,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:10:34,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:34,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:34,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:34,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:34,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:34,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:34,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:34,244 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:34,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:34,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:10:34,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:34,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:34,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:34,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:34,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:34,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:34,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:34,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:34,433 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:34,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:10:34,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:34,533 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:34,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:34,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:34,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:34,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:34,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:34,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:34,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:34,673 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:10:34,693 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:34,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:34,732 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:34,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:34,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:34,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:34,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:34,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:34,836 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 09:10:34,839 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:10:35,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:10:35,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:10:36,273 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:10:36,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:10:36,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:10:36,670 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:10:36,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:10:37,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:10:37,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:10:37,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:10:37,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:10:37,552 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 09:10:37,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:10:37,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:10:42,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:10:43,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:10:43,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:10:43,895 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:10:43,923 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 09:10:43,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:10:43,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:10:44,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:10:44,078 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 09:10:44,078 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:10:44,081 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 09:10:44,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:10:44,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 09:10:44,463 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:10:44,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 09:10:44,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 09:10:44,492 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 09:10:44,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 09:10:44,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 09:10:44,540 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 09:10:44,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 09:10:44,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:10:44,585 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 09:10:44,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:10:44,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:10:44,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:10:44,673 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:10:44,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:10:44,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:10:44,714 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 09:10:44,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:10:44,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:10:44,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:10:44,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:44,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:10:44,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:44,829 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:44,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:44,877 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:44,900 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:44,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:44,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:44,969 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:10:44,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:45,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:10:45,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:45,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:45,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:45,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:45,094 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:45,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:45,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:45,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:10:45,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:45,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:10:45,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:45,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:45,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:45,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:45,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:45,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:45,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:45,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:10:45,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:45,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:10:45,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:45,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:45,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:45,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:45,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:45,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:45,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:45,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:10:45,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:45,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:10:45,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:45,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:45,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:45,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:45,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:45,670 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:45,686 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:45,701 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:10:45,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:45,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:10:45,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:45,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:45,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:45,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:45,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:45,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:45,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:45,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:10:45,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:45,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:10:45,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:45,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:45,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:45,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:45,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:45,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:46,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:46,030 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:10:46,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:46,063 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:10:46,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:46,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:46,117 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:46,135 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:46,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:46,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:46,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:46,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:10:46,226 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 09:10:46,227 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:10:46,308 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:10:46,538 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:10:46,599 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:10:46,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:10:46,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:10:46,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:10:46,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:10:46,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:10:46,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:10:46,691 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:10:46,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:10:46,726 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 09:10:46,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:10:46,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 11772 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:10:52,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:10:53,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:10:54,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:10:54,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:10:54,838 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 09:10:54,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:10:54,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:10:54,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:10:54,943 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 09:10:54,944 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:10:54,946 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:10:55,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:10:55,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:10:55,734 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:10:55,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:10:55,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:10:56,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:10:56,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:10:56,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:10:56,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:10:56,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:10:56,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:10:56,181 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 09:10:56,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:10:56,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:11:02,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:11:03,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:11:04,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:11:04,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:11:04,582 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 09:11:04,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:11:04,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:11:04,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:11:04,710 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 09:11:04,710 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:11:04,713 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 09:11:04,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:11:05,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 09:11:05,141 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:11:05,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 09:11:05,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 09:11:05,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 09:11:05,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 09:11:05,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 09:11:05,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 09:11:05,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 09:11:05,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:11:05,269 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 09:11:05,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:11:05,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:11:05,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:11:05,375 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:11:05,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:11:05,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:11:05,424 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 09:11:05,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:11:05,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:11:05,459 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:11:05,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:05,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:11:05,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:05,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:05,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:05,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:05,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:05,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:05,673 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:05,698 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:05,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:05,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:11:05,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:05,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:05,799 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:05,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:05,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:05,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:05,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:05,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:05,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:05,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:11:05,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:05,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:06,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:06,020 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:06,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:06,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:06,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:06,104 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:06,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:06,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:11:06,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:06,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:06,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:06,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:06,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:06,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:06,280 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:06,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:06,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:06,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:11:06,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:06,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:06,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:06,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:06,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:06,451 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:06,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:06,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:06,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:06,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:11:06,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:06,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:06,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:06,609 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:06,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:06,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:06,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:06,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:06,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:06,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:11:06,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:06,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:06,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:06,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:06,826 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:06,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:06,864 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:06,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:06,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:06,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:11:06,946 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:06,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:06,984 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:07,005 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:07,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:07,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:07,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:07,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:07,102 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 09:11:07,105 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 09:11:07,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:11:07,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 09:11:07,497 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:11:07,499 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 09:11:07,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 09:11:07,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 09:11:07,540 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 09:11:07,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 09:11:07,571 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 09:11:07,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 09:11:07,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:11:07,615 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 09:11:07,658 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:11:07,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:11:07,688 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:11:07,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:11:07,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:11:07,732 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:11:07,746 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 09:11:07,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:11:07,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:11:07,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:11:07,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:07,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:11:07,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:07,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:07,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:07,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:07,936 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:07,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:07,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:08,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:08,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:08,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:11:08,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:08,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:08,104 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:08,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:08,137 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:08,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:08,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:08,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:08,206 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:08,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:11:08,242 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:08,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:08,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:08,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:08,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:08,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:08,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:08,364 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:08,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:08,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:11:08,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:08,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:08,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:08,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:08,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:08,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:08,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:08,563 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:08,581 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:08,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:11:08,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:08,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:08,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:08,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:08,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:08,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:08,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:08,747 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:08,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:08,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:11:08,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:08,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:08,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:08,844 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:08,859 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:08,877 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:08,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:08,908 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:08,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:08,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:11:08,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:08,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:08,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:09,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:09,020 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:09,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:09,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:09,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:09,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:09,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:11:09,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:09,137 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:09,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:09,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:09,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:09,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:09,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:09,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:09,253 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 09:11:09,256 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:11:09,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:11:09,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:11:09,854 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:11:09,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:11:09,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:11:09,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:11:09,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:11:09,961 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:11:09,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:11:10,010 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:11:10,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:11:10,098 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 09:11:10,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:11:10,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:11:16,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:11:17,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:11:17,716 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:11:18,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:11:18,220 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 09:11:18,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:11:18,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:11:18,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:11:18,355 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 09:11:18,355 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:11:18,371 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:11:18,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:11:18,716 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 09:11:18,792 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:11:18,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:11:18,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:11:18,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:11:18,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:11:18,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:11:18,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 09:11:18,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:11:18,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:11:18,946 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 09:11:18,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:11:18,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:11:23,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:11:24,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:11:24,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:11:25,278 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:11:25,298 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 09:11:25,298 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:11:25,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:11:25,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:11:25,411 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 09:11:25,411 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:11:25,414 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:11:26,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:11:26,464 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:11:26,622 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:11:26,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:11:26,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:11:26,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:11:26,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:11:26,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:11:26,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:11:26,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:11:26,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:11:26,783 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 09:11:26,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:11:26,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:11:31,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:11:32,451 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:11:32,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:11:33,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:11:33,180 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 09:11:33,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:11:33,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:11:33,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:11:33,278 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 09:11:33,279 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:11:33,282 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 09:11:33,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:11:33,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:11:33,917 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:11:33,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:11:33,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:11:34,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:11:34,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:11:34,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:11:34,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:11:34,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:11:34,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:11:34,276 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 09:11:34,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:11:34,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:11:42,196 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:11:43,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:11:43,686 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:11:44,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:11:44,180 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 09:11:44,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:11:44,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:11:44,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:11:44,263 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 09:11:44,265 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:11:44,266 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 09:11:44,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:11:44,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 09:11:44,664 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:11:44,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 09:11:44,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 09:11:44,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 09:11:44,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 09:11:44,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 09:11:44,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 09:11:44,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 09:11:44,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:11:44,799 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 09:11:44,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:11:44,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:11:44,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:11:44,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:11:44,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:11:44,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:11:44,921 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 09:11:44,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:11:44,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:11:44,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:11:44,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:44,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:11:45,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:45,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:45,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:45,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:45,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:45,135 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:45,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:45,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:11:45,206 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:45,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:11:45,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:45,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:45,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:45,341 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:45,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:45,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:45,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:45,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:11:45,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:45,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:11:45,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:45,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:45,505 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:45,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:45,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:45,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:45,579 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:45,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:11:45,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:45,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:11:45,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:45,670 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:45,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:45,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:45,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:45,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:45,757 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:45,775 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:11:45,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:45,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:11:45,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:45,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:45,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:45,874 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:45,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:45,908 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:45,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:45,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:11:45,959 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:45,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:11:45,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:46,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:46,025 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:46,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:46,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:46,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:46,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:46,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:11:46,124 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:46,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:11:46,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:46,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:46,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:46,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:46,227 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:46,244 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:46,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:46,280 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:11:46,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:46,318 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:11:46,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:46,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:46,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:46,395 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:46,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:46,433 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:46,451 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:46,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:11:46,490 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 09:11:46,490 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:11:46,492 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:11:46,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:11:46,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:11:46,980 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:11:46,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:11:47,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:11:47,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:11:47,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:11:47,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:11:47,084 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:11:47,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:11:47,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:11:47,747 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:11:47,747 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.257999897. 2022-02-08 09:11:47,747 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:11:47,750 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:11:48,385 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:11:48,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:11:48,806 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:11:48,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:11:48,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:11:48,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:11:48,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:11:48,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:11:48,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:11:48,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:11:48,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:11:49,638 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:11:49,638 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 11797 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:11:51,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:11:51,839 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:11:51,841 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 09:11:53,841 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:11:54,017 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:11:54,019 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.27100014687. 2022-02-08 09:11:54,019 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:11:54,022 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:11:54,920 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:11:55,487 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:11:55,621 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:11:55,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:11:55,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:11:55,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:11:55,706 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:11:55,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:11:55,762 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:11:55,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:11:55,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:11:56,655 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:11:56,655 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 09:11:58,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:11:58,854 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:11:58,854 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 09:12:00,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:12:01,049 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:12:01,049 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.03099989891. 2022-02-08 09:12:01,049 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:12:01,052 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:12:02,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:12:02,693 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:12:03,193 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:12:03,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:12:03,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:12:03,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:12:03,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:12:03,385 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:12:03,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:12:03,438 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:12:03,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:12:04,135 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:12:04,135 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.08599996567. 2022-02-08 09:12:04,137 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:12:04,138 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:12:05,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:12:05,594 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:12:05,720 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:12:05,721 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:12:05,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:12:05,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:12:05,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:12:05,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:12:05,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:12:05,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:12:05,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:12:06,555 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:12:06,556 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.41999983788. 2022-02-08 09:12:06,556 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:12:06,559 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 09:12:07,507 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:12:07,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:12:08,036 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:12:08,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:12:08,114 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:12:08,211 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:12:08,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:12:08,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:12:08,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:12:08,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:12:08,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:12:09,184 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:12:09,184 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.62800002098. 2022-02-08 09:12:09,184 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:12:09,187 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:12:10,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:12:10,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:12:10,674 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:12:10,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:12:10,693 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:12:10,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:12:10,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:12:10,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:12:10,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:12:10,839 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:12:10,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:12:11,377 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:12:11,378 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.1930000782. 2022-02-08 09:12:11,378 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:12:11,381 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:12:12,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:12:12,665 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 09:12:13,038 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:12:13,038 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:12:13,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:12:13,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:12:13,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:12:13,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:12:13,339 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 09:12:13,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:12:13,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:12:14,776 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:12:14,776 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 09:12:16,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:12:17,020 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:12:17,020 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 09:12:19,022 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:12:19,213 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:12:19,214 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.83699989319. 2022-02-08 09:12:19,214 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:12:19,217 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:12:19,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:12:19,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:12:20,239 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:12:20,239 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:12:20,482 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:12:20,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:12:20,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:12:20,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:12:21,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:12:21,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:12:21,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:12:22,674 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:12:22,674 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:12:24,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:12:24,966 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:12:24,966 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:12:26,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:12:27,390 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:12:27,391 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:12:29,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:12:29,657 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:12:29,657 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:12:31,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:12:32,132 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:12:32,132 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:12:34,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:12:34,280 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:12:34,282 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 15.0670001507. ############################################################ # # RAM FootPrint: 4523.10546875 # ############################################################ 2022-02-08 09:12:34,319 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 09:12:34,319 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 71 seconds. Reason: Iteration:18 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 11820 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:13:45,325 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 09:13:45,326 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 5 queries submitted, 0 failed, 11845 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:13:50,329 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:13:51,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:13:51,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:13:52,158 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:13:52,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:13:52,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:13:52,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:13:52,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:13:52,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:13:52,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:13:52,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:13:52,566 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:13:52,566 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 09:13:52,625 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['40594 memcached\n'], []) 2022-02-08 09:13:52,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 09:13:52,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 09:13:52,697 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['40594 memcached\n'], []) 2022-02-08 09:13:52,697 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:13:52,700 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:13:53,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:13:53,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:13:53,980 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:13:53,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:13:53,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:13:54,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:13:54,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:13:54,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:13:54,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:13:54,127 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:13:54,147 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:13:54,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 09:13:54,250 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['93513 memcached\n'], []) 2022-02-08 09:13:54,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 09:13:54,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 09:13:54,312 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['93513 memcached\n'], []) 2022-02-08 09:13:54,313 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:13:54,315 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:13:55,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:13:56,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:13:56,200 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:13:56,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:13:56,221 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:13:56,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:13:56,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:13:56,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:13:56,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:13:56,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:13:56,375 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:13:56,375 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 09:13:56,490 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['76408 memcached\n'], []) 2022-02-08 09:13:56,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 09:13:56,525 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 09:13:56,553 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['76408 memcached\n'], []) 2022-02-08 09:13:56,553 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:13:56,556 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:13:57,451 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:13:57,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:13:58,101 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:13:58,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:13:58,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:13:58,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:13:58,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:13:58,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:13:58,216 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:13:58,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:13:58,269 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:13:58,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 09:13:58,335 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['104829 memcached\n'], []) 2022-02-08 09:13:58,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 09:13:58,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 09:13:58,395 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['104829 memcached\n'], []) 2022-02-08 09:13:58,397 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:13:58,400 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:13:59,487 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:13:59,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:14:00,029 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:14:00,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:14:00,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:14:00,084 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:14:00,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:14:00,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:14:00,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:14:00,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:14:00,209 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:14:00,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 09:14:00,267 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['98746 memcached\n'], []) 2022-02-08 09:14:00,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 09:14:00,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 09:14:00,369 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['98746 memcached\n'], []) 2022-02-08 09:14:00,371 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:14:00,394 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 09:14:01,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:14:01,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:14:01,915 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:14:01,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:14:01,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:14:01,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:14:01,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:14:02,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:14:02,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:14:02,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:14:02,092 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:14:02,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 09:14:02,161 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['3168 memcached\n'], []) 2022-02-08 09:14:02,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 09:14:02,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 09:14:02,214 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['3168 memcached\n'], []) 2022-02-08 09:14:02,214 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:14:02,217 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:14:03,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:14:03,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:14:04,157 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:14:04,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:14:04,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:14:04,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:14:04,224 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:14:04,244 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:14:04,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:14:04,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:14:04,325 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:14:04,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 09:14:04,398 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['67615 memcached\n'], []) 2022-02-08 09:14:04,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 09:14:04,428 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 09:14:04,450 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['67615 memcached\n'], []) 2022-02-08 09:14:04,450 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:14:04,454 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:14:05,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:14:05,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 09:14:05,938 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:14:05,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:14:05,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:14:06,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:14:06,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:14:06,061 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:14:06,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 09:14:06,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:14:06,118 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:14:06,119 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 09:14:06,173 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['82824 memcached\n'], []) 2022-02-08 09:14:06,173 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 09:14:06,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 09:14:06,223 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['82824 memcached\n'], []) 2022-02-08 09:14:06,224 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:14:06,226 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:14:06,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:14:07,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:14:07,486 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:14:07,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:14:07,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:14:07,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:14:07,538 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:14:07,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:14:07,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:14:07,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:14:07,618 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:14:07,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 09:14:07,687 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['18376 memcached\n'], []) 2022-02-08 09:14:07,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 09:14:07,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 09:14:07,742 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['18376 memcached\n'], []) 2022-02-08 09:14:07,743 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:14:07,744 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:14:07,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:14:08,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:14:08,121 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:14:08,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:14:08,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:14:08,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:14:08,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:14:08,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:14:08,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:14:08,216 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:14:08,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:14:08,247 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 09:14:08,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:14:08,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:14:13,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:14:14,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:14:15,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:14:15,681 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:14:15,696 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 09:14:15,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:14:15,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:14:15,775 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:14:15,789 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 09:14:15,789 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:14:15,792 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:14:15,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:14:16,096 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:14:16,160 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:14:16,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:14:16,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:14:16,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:14:16,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:14:16,216 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:14:16,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:14:16,242 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:14:16,255 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:14:16,269 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 09:14:16,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:14:16,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:14:22,612 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:14:23,372 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:14:23,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:14:24,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:14:24,217 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 09:14:24,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:14:24,232 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:14:24,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:14:24,328 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 09:14:24,328 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:14:24,329 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 09:14:24,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:14:24,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 09:14:24,729 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:14:24,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 09:14:24,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 09:14:24,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 09:14:24,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 09:14:24,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 09:14:24,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 09:14:24,829 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 09:14:24,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:14:24,864 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 09:14:24,897 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:14:24,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:14:24,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:14:24,946 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:14:24,961 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:14:24,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:14:25,002 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 09:14:25,002 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:14:25,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:14:25,035 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:14:25,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:25,073 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:14:25,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:25,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:25,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:25,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:25,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:25,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:25,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:25,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:25,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:25,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:14:25,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:25,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:25,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:25,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:25,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:25,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:25,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:25,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:25,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:25,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:14:25,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:25,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:25,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:25,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:25,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:25,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:25,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:25,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:25,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:25,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:14:25,724 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:25,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:25,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:25,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:25,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:25,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:25,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:25,864 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:25,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:25,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:14:25,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:25,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:25,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:25,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:25,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:26,010 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:26,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:26,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:26,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:26,089 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:14:26,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:26,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:26,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:26,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:26,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:26,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:26,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:26,234 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:26,252 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:26,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:14:26,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:26,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:26,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:26,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:26,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:26,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:26,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:26,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:26,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:26,438 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:14:26,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:26,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:26,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:26,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:26,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:26,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:26,566 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:26,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:26,605 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 09:14:26,608 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:14:26,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:14:27,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:14:27,993 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:14:27,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:14:28,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:14:28,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:14:28,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:14:28,732 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:14:28,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:14:29,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:14:29,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:14:29,430 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 09:14:29,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:14:29,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:14:34,971 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:14:35,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:14:35,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:14:36,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:14:36,207 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 09:14:36,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:14:36,306 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:14:36,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:14:36,413 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 09:14:36,413 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:14:36,415 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 09:14:36,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:14:36,732 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 09:14:36,796 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:14:36,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 09:14:36,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 09:14:36,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 09:14:36,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 09:14:36,859 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 09:14:36,874 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 09:14:36,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 09:14:36,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:14:36,918 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 09:14:36,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:14:36,969 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:14:36,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:14:37,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:14:37,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:14:37,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:14:37,055 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 09:14:37,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:14:37,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:14:37,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:14:37,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:37,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:14:37,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:37,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:37,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:37,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:37,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:37,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:37,339 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:37,367 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:37,390 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:37,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:14:37,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:37,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:37,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:37,484 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:37,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:37,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:37,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:37,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:37,571 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:37,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:14:37,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:37,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:37,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:37,658 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:37,676 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:37,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:37,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:37,727 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:37,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:37,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:14:37,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:37,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:37,813 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:37,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:37,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:37,864 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:37,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:37,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:14:37,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:37,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:14:37,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:37,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:37,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:38,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:38,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:38,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:38,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:38,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:14:38,089 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:38,107 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:14:38,127 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:38,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:38,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:38,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:38,196 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:38,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:38,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:38,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:14:38,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:38,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:14:38,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:38,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:38,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:38,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:38,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:38,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:38,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:38,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:14:38,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:38,451 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:14:38,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:38,486 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:38,505 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:38,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:38,542 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:38,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:38,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:38,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:14:38,621 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 09:14:38,624 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:14:38,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:14:38,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:14:39,134 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:14:39,135 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:14:39,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:14:39,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:14:39,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:14:39,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:14:39,216 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:14:39,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:14:39,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:14:39,282 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 09:14:39,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:14:39,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:14:45,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:14:46,630 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:14:47,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:14:47,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:14:47,608 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 09:14:47,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:14:47,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:14:47,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:14:47,710 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 09:14:47,711 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:14:47,714 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:14:48,020 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:14:48,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:14:49,688 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 5 queries submitted, 0 failed, 11870 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:14:49,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:14:49,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:14:49,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:14:49,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:14:49,874 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:14:49,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:14:49,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:14:49,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:14:49,957 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 09:14:49,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:14:50,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:14:56,397 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:14:57,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:14:57,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:14:58,214 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:14:58,305 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 09:14:58,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:14:58,334 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:14:58,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:14:58,578 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 09:14:58,578 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:14:58,581 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 09:14:58,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:14:58,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 09:14:59,065 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:14:59,066 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 09:14:59,086 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 09:14:59,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 09:14:59,127 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 09:14:59,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 09:14:59,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 09:14:59,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 09:14:59,211 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:14:59,230 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 09:14:59,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:14:59,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:14:59,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:14:59,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:14:59,346 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:14:59,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:14:59,391 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 09:14:59,392 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:14:59,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:14:59,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:14:59,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:59,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:14:59,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:59,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:59,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:59,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:59,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:59,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:59,686 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:59,716 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:14:59,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:59,767 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:14:59,793 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:59,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:59,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:59,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:59,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:59,900 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:59,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:59,946 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:14:59,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:14:59,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:15:00,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:00,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:00,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:00,073 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:00,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:00,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:00,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:00,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:00,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:00,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:15:00,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:00,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:00,278 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:00,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:00,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:00,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:00,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:00,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:00,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:00,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:15:00,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:00,471 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:00,492 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:00,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:00,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:00,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:00,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:00,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:00,619 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:00,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:15:00,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:00,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:00,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:00,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:00,743 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:00,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:00,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:00,806 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:00,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:00,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:15:00,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:00,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:00,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:00,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:00,956 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:00,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:00,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:01,022 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:01,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:01,066 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:15:01,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:01,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:01,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:01,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:01,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:01,211 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:01,234 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:01,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:01,279 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 09:15:01,282 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 09:15:01,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:15:01,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 09:15:01,665 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:15:01,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 09:15:01,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 09:15:01,696 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 09:15:01,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 09:15:01,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 09:15:01,752 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 09:15:01,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 09:15:01,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:15:01,798 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 09:15:01,829 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:15:01,845 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:15:01,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:15:01,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:15:01,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:15:01,903 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:15:01,917 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 09:15:01,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:15:01,933 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:15:01,948 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:15:01,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:01,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:15:02,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:02,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:02,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:02,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:02,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:02,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:02,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:02,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:02,244 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:02,263 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:15:02,280 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:02,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:02,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:02,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:02,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:02,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:02,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:02,410 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:02,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:02,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:15:02,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:02,484 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:02,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:02,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:02,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:02,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:02,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:02,594 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:02,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:02,630 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:15:02,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:02,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:02,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:02,701 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:02,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:02,734 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:02,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:02,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:02,788 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:02,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:15:02,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:02,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:02,854 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:02,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:02,885 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:02,903 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:02,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:02,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:02,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:02,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:15:02,984 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:03,002 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:03,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:03,038 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:03,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:03,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:03,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:03,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:03,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:03,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:15:03,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:03,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:03,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:03,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:03,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:03,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:03,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:03,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:03,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:03,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:15:03,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:03,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:03,372 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:03,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:03,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:03,426 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:03,443 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:03,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:03,476 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 09:15:03,480 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:15:03,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:15:03,982 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:15:04,173 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:15:04,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:15:04,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:15:04,272 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:15:04,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:15:04,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:15:04,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:15:04,426 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:15:04,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:15:04,463 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 09:15:04,487 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:15:04,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:15:10,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:15:11,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:15:11,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:15:12,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:15:12,358 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 09:15:12,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:15:12,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:15:12,438 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:15:12,466 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 09:15:12,466 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:15:12,470 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:15:12,609 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:15:12,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 09:15:13,105 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:15:13,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:15:13,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:15:13,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:15:13,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:15:13,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:15:13,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 09:15:13,255 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:15:13,278 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:15:13,299 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 09:15:13,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:15:13,364 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:15:18,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:15:19,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:15:20,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:15:20,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:15:20,650 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 09:15:20,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:15:20,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:15:20,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:15:20,980 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 09:15:20,982 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:15:20,984 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:15:21,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:15:21,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:15:22,055 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:15:22,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:15:22,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:15:22,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:15:22,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:15:22,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:15:22,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:15:22,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:15:22,196 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:15:22,214 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 09:15:22,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:15:22,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:15:27,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:15:27,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:15:28,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:15:28,561 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:15:28,576 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 09:15:28,576 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:15:28,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:15:28,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:15:28,648 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 09:15:28,650 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:15:28,651 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 09:15:28,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:15:29,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:15:29,157 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:15:29,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:15:29,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:15:29,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:15:29,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:15:29,252 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:15:29,299 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:15:29,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:15:29,342 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:15:29,365 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 09:15:29,392 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:15:29,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:15:37,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:15:38,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:15:38,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:15:39,454 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:15:39,601 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 09:15:39,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:15:39,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:15:39,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:15:39,701 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 09:15:39,701 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:15:39,703 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 09:15:39,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:15:40,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 09:15:40,088 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:15:40,089 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 09:15:40,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 09:15:40,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 09:15:40,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 09:15:40,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 09:15:40,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 09:15:40,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 09:15:40,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:15:40,226 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 09:15:40,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:15:40,279 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:15:40,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:15:40,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:15:40,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:15:40,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:15:40,374 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 09:15:40,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:15:40,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:15:40,411 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:15:40,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:40,453 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:15:40,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:40,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:40,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:40,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:40,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:40,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:40,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:40,670 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:15:40,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:40,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:15:40,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:40,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:40,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:40,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:40,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:40,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:40,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:40,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:15:40,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:40,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:15:40,933 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:40,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:40,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:40,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:41,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:41,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:41,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:41,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:15:41,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:41,094 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:15:41,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:41,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:41,147 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:41,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:41,183 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:41,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:41,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:41,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:15:41,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:41,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:15:41,298 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:41,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:41,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:41,348 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:41,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:41,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:41,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:41,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:15:41,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:41,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:15:41,480 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:41,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:41,517 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:41,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:41,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:41,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:41,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:41,612 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:15:41,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:41,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:15:41,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:41,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:41,729 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:41,752 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:41,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:41,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:41,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:41,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:15:41,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:41,864 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:15:41,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:41,903 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:41,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:41,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:41,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:41,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:41,987 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:42,005 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:15:42,019 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 09:15:42,019 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:15:42,022 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:15:42,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:15:42,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:15:42,490 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:15:42,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:15:42,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:15:42,543 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:15:42,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:15:42,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:15:42,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:15:42,619 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:15:42,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:15:43,141 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:15:43,142 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.12299990654. 2022-02-08 09:15:43,142 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:15:43,144 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:15:44,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:15:44,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:15:44,986 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:15:44,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:15:45,010 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:15:45,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:15:45,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:15:45,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:15:45,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:15:45,145 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:15:45,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:15:45,821 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:15:45,821 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 09:15:47,821 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:15:48,003 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:15:48,003 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.86000013351. 2022-02-08 09:15:48,003 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:15:48,006 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:15:49,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 11897 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:15:49,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:15:49,915 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:15:49,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:15:49,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:15:49,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:15:50,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:15:50,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:15:50,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:15:50,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:15:50,165 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:15:50,984 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:15:50,984 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 09:15:52,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:15:53,157 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:15:53,157 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 09:15:55,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:15:55,357 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:15:55,357 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.35300016403. 2022-02-08 09:15:55,358 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:15:55,359 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:15:56,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:15:56,839 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:15:57,003 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:15:57,005 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:15:57,028 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:15:57,061 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:15:57,081 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:15:57,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:15:57,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:15:57,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:15:57,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:15:58,061 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:15:58,062 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.70399999619. 2022-02-08 09:15:58,062 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:15:58,063 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:15:58,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:15:59,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:15:59,506 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:15:59,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:15:59,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:15:59,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:15:59,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:15:59,614 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:15:59,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:15:59,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:15:59,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:16:00,443 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:16:00,444 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.38199996948. 2022-02-08 09:16:00,444 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:16:00,447 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 09:16:01,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:16:01,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:16:01,456 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:16:01,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:16:01,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:16:01,507 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:16:01,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:16:01,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:16:01,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:16:01,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:16:01,630 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:16:02,188 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:16:02,190 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.7460000515. 2022-02-08 09:16:02,190 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:16:02,193 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:16:03,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:16:03,548 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:16:03,694 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:16:03,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:16:03,709 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:16:03,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:16:03,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:16:03,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:16:03,775 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:16:03,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:16:03,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:16:04,355 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:16:04,357 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.16700005531. 2022-02-08 09:16:04,357 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:16:04,358 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:16:05,035 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:16:05,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 09:16:05,493 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:16:05,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:16:05,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:16:05,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:16:05,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:16:05,619 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:16:05,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 09:16:05,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:16:05,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:16:06,230 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:16:06,232 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.875. 2022-02-08 09:16:06,232 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:16:06,233 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:16:06,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:16:06,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:16:07,407 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:16:07,408 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:16:07,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:16:07,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:16:07,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:16:07,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:16:07,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:16:07,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:16:08,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:16:08,361 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:16:08,362 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:16:10,362 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:16:10,964 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:16:10,966 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:16:12,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:16:13,473 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:16:13,473 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:16:15,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:16:16,312 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:16:16,312 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:16:18,312 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:16:18,747 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:16:18,749 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:16:20,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:16:21,158 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:16:21,158 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:16:23,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:16:23,594 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:16:23,594 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:16:25,595 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:16:25,805 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:16:25,805 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:16:27,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:16:28,757 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:16:28,759 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 22.5269999504. ############################################################ # # RAM FootPrint: 4669.55078125 # ############################################################ 2022-02-08 09:16:28,801 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 09:16:28,802 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 118 seconds. Reason: Iteration:19 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 11918 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 11941 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:18:26,806 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 09:18:26,808 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 09:18:31,811 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:18:32,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:18:33,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:18:33,510 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:18:33,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:18:33,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:18:33,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:18:33,660 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:18:33,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:18:33,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:18:33,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:18:33,871 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:18:33,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 09:18:33,933 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['41317 memcached\n'], []) 2022-02-08 09:18:33,933 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 09:18:33,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 09:18:34,019 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['41317 memcached\n'], []) 2022-02-08 09:18:34,019 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:18:34,023 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:18:35,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:18:35,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:18:36,010 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:18:36,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:18:36,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:18:36,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:18:36,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:18:36,094 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:18:36,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:18:36,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:18:36,174 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:18:36,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 09:18:36,250 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['94257 memcached\n'], []) 2022-02-08 09:18:36,252 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 09:18:36,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 09:18:36,315 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['94257 memcached\n'], []) 2022-02-08 09:18:36,315 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:18:36,319 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:18:37,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:18:38,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:18:38,536 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:18:38,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:18:38,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:18:38,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:18:38,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:18:38,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:18:38,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:18:38,724 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:18:38,749 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:18:38,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 09:18:38,822 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['77249 memcached\n'], []) 2022-02-08 09:18:38,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 09:18:38,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 09:18:38,888 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['77249 memcached\n'], []) 2022-02-08 09:18:38,888 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:18:38,890 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:18:39,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:18:40,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:18:41,066 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:18:41,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:18:41,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:18:41,135 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:18:41,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:18:41,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:18:41,191 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:18:41,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:18:41,239 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:18:41,239 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 09:18:41,303 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['105549 memcached\n'], []) 2022-02-08 09:18:41,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 09:18:41,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 09:18:41,365 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['105549 memcached\n'], []) 2022-02-08 09:18:41,365 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:18:41,368 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:18:42,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:18:43,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:18:43,365 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:18:43,367 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:18:43,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:18:43,410 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:18:43,426 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:18:43,446 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:18:43,459 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:18:43,489 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:18:43,503 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:18:43,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 09:18:43,605 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['99470 memcached\n'], []) 2022-02-08 09:18:43,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 09:18:43,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 09:18:43,654 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['99470 memcached\n'], []) 2022-02-08 09:18:43,654 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:18:43,657 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 09:18:44,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:18:45,724 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:18:45,934 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:18:45,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:18:45,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:18:45,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:18:45,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:18:46,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:18:46,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:18:46,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:18:46,098 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:18:46,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 09:18:46,161 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['3888 memcached\n'], []) 2022-02-08 09:18:46,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 09:18:46,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 09:18:46,217 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['3888 memcached\n'], []) 2022-02-08 09:18:46,217 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:18:46,220 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:18:47,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:18:47,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:18:47,796 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:18:47,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:18:47,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:18:47,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:18:47,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:18:47,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:18:47,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:18:47,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:18:47,937 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:18:47,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 09:18:47,993 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['68339 memcached\n'], []) 2022-02-08 09:18:47,994 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 09:18:48,022 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 09:18:48,045 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['68339 memcached\n'], []) 2022-02-08 09:18:48,045 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:18:48,049 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:18:48,933 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:18:49,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 09:18:49,601 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:18:49,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:18:49,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:18:49,637 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:18:49,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:18:49,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:18:49,691 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 5 queries submitted, 0 failed, 11966 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:18:49,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:18:49,732 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:18:49,732 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 09:18:49,789 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['83535 memcached\n'], []) 2022-02-08 09:18:49,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 09:18:49,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 09:18:49,845 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['83535 memcached\n'], []) 2022-02-08 09:18:49,845 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:18:49,848 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:18:50,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:18:51,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:18:51,585 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:18:51,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:18:51,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:18:51,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:18:51,632 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:18:51,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:18:51,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:18:51,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:18:51,710 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:18:51,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 09:18:51,786 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['19158 memcached\n'], []) 2022-02-08 09:18:51,788 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 09:18:51,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 09:18:51,835 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['19158 memcached\n'], []) 2022-02-08 09:18:51,835 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:18:51,838 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:18:51,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:18:52,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:18:52,213 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:18:52,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:18:52,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:18:52,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:18:52,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:18:52,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:18:52,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:18:52,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:18:52,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:18:52,336 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 09:18:52,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:18:52,380 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:18:59,132 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:18:59,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:00,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:00,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:19:00,858 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 09:19:00,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:19:00,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:00,948 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:19:00,969 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 09:19:00,969 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:19:00,971 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:19:01,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:19:01,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:19:01,349 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:19:01,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:19:01,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:19:01,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:19:01,391 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:19:01,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:19:01,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:19:01,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:19:01,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:19:01,463 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 09:19:01,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:19:01,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:08,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:19:09,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:09,459 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:09,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:19:09,888 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 09:19:09,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:19:09,903 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:09,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:19:09,999 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 09:19:09,999 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:19:10,002 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 09:19:10,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:19:10,323 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 09:19:10,387 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:19:10,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 09:19:10,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 09:19:10,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 09:19:10,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 09:19:10,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 09:19:10,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 09:19:10,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 09:19:10,492 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:19:10,507 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 09:19:10,540 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:19:10,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:10,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:19:10,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:10,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:10,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:19:10,631 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 09:19:10,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:19:10,647 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:10,661 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:19:10,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:10,701 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:19:10,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:10,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:10,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:10,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:10,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:10,861 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:10,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:10,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:10,936 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:10,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:19:10,971 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:10,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:11,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:11,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:11,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:11,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:11,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:11,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:11,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:11,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:19:11,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:11,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:11,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:11,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:11,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:11,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:11,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:11,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:11,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:11,344 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:19:11,364 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:11,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:11,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:11,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:11,434 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:11,451 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:11,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:11,492 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:11,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:11,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:19:11,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:11,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:11,578 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:11,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:11,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:11,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:11,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:11,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:11,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:11,688 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:19:11,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:11,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:11,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:11,752 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:11,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:11,782 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:11,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:11,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:11,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:11,839 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:19:11,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:11,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:11,884 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:11,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:11,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:11,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:11,946 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:11,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:11,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:11,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:19:12,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:12,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:12,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:12,061 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:12,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:12,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:12,114 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:12,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:12,148 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 09:19:12,151 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:19:12,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:19:12,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:19:12,756 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:19:12,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:19:12,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:19:12,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:19:12,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:19:12,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:19:12,936 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:19:12,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:19:12,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:19:13,003 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 09:19:13,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:19:13,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:18,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:19:18,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:19,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:19,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:19:19,911 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 09:19:19,913 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:19:19,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:20,030 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:19:20,055 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 09:19:20,055 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:19:20,059 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 09:19:20,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:19:20,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 09:19:20,444 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:19:20,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 09:19:20,459 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 09:19:20,471 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 09:19:20,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 09:19:20,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 09:19:20,519 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 09:19:20,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 09:19:20,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:19:20,559 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 09:19:20,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:19:20,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:20,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:19:20,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:20,653 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:20,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:19:20,683 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 09:19:20,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:19:20,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:20,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:19:20,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:20,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:19:20,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:20,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:20,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:20,864 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:20,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:20,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:20,950 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:20,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:21,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:21,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:19:21,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:21,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:21,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:21,096 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:21,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:21,132 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:21,150 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:21,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:21,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:21,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:19:21,239 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:21,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:21,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:21,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:21,306 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:21,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:21,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:21,362 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:21,380 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:21,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:19:21,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:21,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:21,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:21,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:21,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:21,500 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:21,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:21,533 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:21,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:21,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:19:21,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:21,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:21,612 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:21,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:21,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:21,665 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:21,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:21,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:21,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:21,737 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:19:21,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:21,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:21,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:21,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:21,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:21,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:21,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:21,864 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:21,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:21,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:19:21,910 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:21,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:21,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:21,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:21,971 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:21,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:22,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:22,017 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:22,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:22,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:19:22,071 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:22,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:22,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:22,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:22,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:22,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:22,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:22,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:22,213 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 09:19:22,216 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:19:22,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:19:22,517 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:19:22,585 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:19:22,586 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:19:22,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:19:22,617 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:19:22,630 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:19:22,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:19:22,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:19:22,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:19:22,691 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:19:22,707 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 09:19:22,732 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:19:22,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:28,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:19:29,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:30,346 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:30,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:19:30,921 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 09:19:30,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:19:30,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:30,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:19:31,009 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 09:19:31,009 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:19:31,010 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:19:31,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:19:31,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:19:32,026 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:19:32,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:19:32,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:19:32,063 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:19:32,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:19:32,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:19:32,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:19:32,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:19:32,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:19:33,144 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 09:19:33,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:19:33,209 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:39,769 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:19:40,568 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:41,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:41,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:19:41,460 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 09:19:41,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:19:41,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:41,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:19:41,746 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 09:19:41,746 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:19:41,750 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 09:19:41,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:19:42,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 09:19:42,170 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:19:42,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 09:19:42,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 09:19:42,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 09:19:42,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 09:19:42,244 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 09:19:42,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 09:19:42,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 09:19:42,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:19:42,315 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 09:19:42,349 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:19:42,368 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:42,385 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:19:42,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:42,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:42,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:19:42,457 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 09:19:42,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:19:42,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:42,494 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:19:42,512 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:42,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:19:42,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:42,607 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:42,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:42,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:42,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:42,724 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:42,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:42,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:42,811 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:42,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:19:42,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:42,885 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:42,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:42,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:42,971 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:42,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:43,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:43,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:43,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:43,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:19:43,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:43,117 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:43,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:43,158 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:43,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:43,206 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:43,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:43,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:43,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:43,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:19:43,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:43,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:43,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:43,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:43,392 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:43,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:43,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:43,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:43,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:43,497 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:19:43,520 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:43,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:43,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:43,584 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:43,602 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:43,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:43,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:43,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:43,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:43,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:19:43,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:43,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:43,775 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:43,793 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:43,815 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:43,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:43,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:43,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:43,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:43,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:19:43,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:43,954 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:43,973 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:43,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:44,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:44,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:44,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:44,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:44,094 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:44,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:19:44,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:44,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:44,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:44,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:44,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:44,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:44,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:44,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:44,319 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 09:19:44,322 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 09:19:44,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:19:44,670 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 09:19:44,734 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:19:44,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 09:19:44,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 09:19:44,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 09:19:44,779 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 09:19:44,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 09:19:44,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 09:19:44,825 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 09:19:44,839 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:19:44,857 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 09:19:44,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:19:44,903 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:44,917 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:19:44,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:44,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:44,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:19:44,976 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 09:19:44,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:19:44,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:45,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:19:45,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:45,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:19:45,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:45,102 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:45,132 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:45,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:45,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:45,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:45,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:45,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:19:45,306 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:45,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:19:45,346 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:45,364 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:45,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:45,401 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:45,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:45,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:45,466 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:45,483 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:19:45,499 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:45,516 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:19:45,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:45,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:45,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:45,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:45,604 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:45,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:45,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:45,660 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:19:45,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:45,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:19:45,719 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:45,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:45,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:45,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:45,788 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:45,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:45,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:45,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:19:45,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:45,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:19:45,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:45,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:45,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:45,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:45,956 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:45,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:45,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:46,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:19:46,025 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:46,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:19:46,058 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:46,073 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:46,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:46,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:46,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:46,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:46,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:46,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:19:46,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:46,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:19:46,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:46,234 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:46,257 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:46,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:46,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:46,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:46,332 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:46,349 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:19:46,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:46,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:19:46,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:46,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:46,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:46,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:46,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:46,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:46,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:46,526 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:19:46,545 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 09:19:46,546 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:19:46,651 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:19:46,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:19:47,039 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:19:47,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:19:47,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:19:47,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:19:47,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:19:47,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:19:47,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:19:47,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:19:47,224 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:19:47,250 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 09:19:47,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:19:47,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 11991 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:19:53,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:19:54,061 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:54,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:19:54,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:19:55,013 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 09:19:55,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:19:55,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:19:55,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:19:55,135 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 09:19:55,135 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:19:55,138 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:19:55,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:19:55,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 09:19:55,569 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:19:55,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:19:55,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:19:55,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:19:55,628 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:19:55,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:19:55,671 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 09:19:55,688 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:19:55,710 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:19:55,732 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 09:19:55,765 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:19:55,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:20:01,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:20:02,030 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:20:02,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:20:02,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:20:03,091 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 09:20:03,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:20:03,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:20:03,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:20:03,293 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 09:20:03,293 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:20:03,296 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:20:03,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:20:04,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:20:04,473 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:20:04,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:20:04,492 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:20:04,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:20:04,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:20:04,585 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:20:04,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:20:04,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:20:04,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:20:04,667 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 09:20:04,703 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:20:04,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:20:09,561 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:20:10,232 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:20:10,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:20:10,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:20:10,930 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 09:20:10,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:20:10,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:20:10,997 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:20:11,016 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 09:20:11,017 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:20:11,019 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 09:20:11,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:20:11,487 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:20:11,789 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:20:11,790 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:20:11,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:20:11,859 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:20:11,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:20:11,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:20:11,941 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:20:11,961 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:20:12,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:20:12,082 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 09:20:12,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:20:12,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:20:19,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:20:21,048 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:20:21,647 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:20:22,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:20:22,190 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 09:20:22,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:20:22,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:20:22,338 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:20:22,368 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 09:20:22,369 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:20:22,371 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 09:20:22,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:20:22,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 09:20:22,759 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:20:22,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 09:20:22,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 09:20:22,790 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 09:20:22,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 09:20:22,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 09:20:22,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 09:20:22,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 09:20:22,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:20:22,888 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 09:20:22,915 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:20:22,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:20:22,948 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:20:22,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:20:22,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:20:22,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:20:23,007 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 09:20:23,007 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:20:23,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:20:23,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:20:23,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:20:23,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:20:23,114 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:20:23,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:20:23,170 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:20:23,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:20:23,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:20:23,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:20:23,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:20:23,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:20:23,346 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:20:23,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:20:23,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:20:23,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:20:23,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:20:23,474 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:20:23,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:20:23,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:20:23,529 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:20:23,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:20:23,563 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:20:23,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:20:23,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:20:23,624 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:20:23,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:20:23,663 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:20:23,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:20:23,698 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:20:23,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:20:23,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:20:23,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:20:23,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:20:23,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:20:23,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:20:23,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:20:23,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:20:23,869 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:20:23,890 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:20:23,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:20:23,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:20:23,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:20:23,961 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:20:23,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:20:24,002 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:20:24,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:20:24,040 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:20:24,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:20:24,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:20:24,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:20:24,112 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:20:24,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:20:24,147 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:20:24,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:20:24,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:20:24,197 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:20:24,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:20:24,242 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:20:24,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:20:24,276 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:20:24,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:20:24,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:20:24,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:20:24,358 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:20:24,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:20:24,397 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:20:24,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:20:24,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:20:24,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:20:24,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:20:24,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:20:24,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:20:24,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:20:24,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:20:24,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:20:24,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:20:24,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:20:24,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:20:24,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:20:24,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:20:24,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:20:24,700 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 09:20:24,701 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:20:24,703 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:20:24,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:20:25,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:20:25,086 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:20:25,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:20:25,104 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:20:25,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:20:25,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:20:25,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:20:25,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:20:25,201 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:20:25,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:20:25,901 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:20:25,901 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.19900012016. 2022-02-08 09:20:25,901 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:20:25,904 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:20:26,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:20:27,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:20:27,426 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:20:27,427 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:20:27,457 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:20:27,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:20:27,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:20:27,591 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:20:27,619 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:20:27,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:20:27,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:20:28,437 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:20:28,437 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 09:20:30,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:20:30,700 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:20:30,700 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 09:20:32,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:20:32,905 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:20:32,905 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 09:20:34,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:20:35,096 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:20:35,098 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 09:20:37,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:20:37,278 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:20:37,279 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 11.3770000935. 2022-02-08 09:20:37,279 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:20:37,280 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:20:38,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:20:38,551 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:20:38,664 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:20:38,665 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:20:38,687 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:20:38,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:20:38,746 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:20:38,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:20:38,796 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:20:38,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:20:38,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:20:39,753 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:20:39,753 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.47399997711. 2022-02-08 09:20:39,753 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:20:39,756 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:20:40,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:20:41,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:20:41,421 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:20:41,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:20:41,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:20:41,513 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:20:41,533 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:20:41,559 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:20:41,579 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:20:41,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:20:41,640 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:20:42,309 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:20:42,311 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.55599999428. 2022-02-08 09:20:42,311 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:20:42,312 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:20:43,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:20:43,621 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:20:43,755 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:20:43,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:20:43,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:20:43,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:20:43,839 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:20:43,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:20:43,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:20:43,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:20:43,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:20:44,714 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:20:44,714 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.40399980545. 2022-02-08 09:20:44,716 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:20:44,717 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 09:20:45,558 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:20:45,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:20:46,002 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:20:46,002 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:20:46,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:20:46,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:20:46,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:20:46,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:20:46,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:20:46,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:20:46,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:20:46,865 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:20:46,865 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.14999985695. 2022-02-08 09:20:46,865 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:20:46,868 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:20:47,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:20:48,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:20:48,286 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:20:48,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:20:48,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:20:48,321 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:20:48,346 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:20:48,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:20:48,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:20:48,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:20:48,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:20:48,953 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:20:48,954 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.08899998665. 2022-02-08 09:20:48,954 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:20:48,957 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:20:49,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 12013 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:20:50,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 09:20:50,503 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:20:50,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:20:50,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:20:50,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:20:50,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:20:50,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:20:50,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 09:20:51,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:20:51,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:20:52,490 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:20:52,490 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 09:20:54,492 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:20:54,680 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:20:54,680 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.72500014305. 2022-02-08 09:20:54,680 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:20:54,683 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:20:55,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:20:55,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:20:56,467 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:20:56,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:20:56,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:20:56,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:20:56,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:20:56,855 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:20:56,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:20:57,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:20:57,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:20:58,369 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:20:58,371 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:21:00,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:21:00,828 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:21:00,828 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:21:02,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:21:03,135 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:21:03,135 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 09:21:05,137 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:21:05,427 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:21:05,428 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 10.746999979. ############################################################ # # RAM FootPrint: 4353.63671875 # ############################################################ 2022-02-08 09:21:05,470 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 09:21:05,470 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 68 seconds. Reason: Iteration:20 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 12037 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:22:13,476 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 09:22:13,476 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 09:22:18,479 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:22:19,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:22:19,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:22:19,963 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:22:19,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:22:19,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:22:20,017 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:22:20,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:22:20,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:22:20,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:22:20,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:22:20,173 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:22:20,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 09:22:20,286 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['42122 memcached\n'], []) 2022-02-08 09:22:20,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 09:22:20,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 09:22:20,351 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['42122 memcached\n'], []) 2022-02-08 09:22:20,352 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:22:20,355 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:22:21,221 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:22:21,660 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:22:21,904 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:22:21,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:22:21,924 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:22:21,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:22:21,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:22:22,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:22:22,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:22:22,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:22:22,111 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:22:22,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 09:22:22,226 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['95030 memcached\n'], []) 2022-02-08 09:22:22,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 09:22:22,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 09:22:22,286 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['95030 memcached\n'], []) 2022-02-08 09:22:22,286 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:22:22,289 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:22:23,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:22:23,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:22:24,105 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:22:24,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:22:24,127 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:22:24,161 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:22:24,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:22:24,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:22:24,227 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:22:24,265 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:22:24,283 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:22:24,283 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 09:22:24,361 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['78131 memcached\n'], []) 2022-02-08 09:22:24,362 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 09:22:24,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 09:22:24,476 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['78131 memcached\n'], []) 2022-02-08 09:22:24,476 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:22:24,480 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:22:25,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:22:25,729 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:22:26,033 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:22:26,033 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:22:26,055 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:22:26,081 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:22:26,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:22:26,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:22:26,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:22:26,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:22:26,197 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:22:26,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 09:22:26,260 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['106340 memcached\n'], []) 2022-02-08 09:22:26,260 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 09:22:26,293 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 09:22:26,319 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['106340 memcached\n'], []) 2022-02-08 09:22:26,321 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:22:26,322 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:22:27,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:22:27,803 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:22:27,982 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:22:27,982 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:22:28,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:22:28,032 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:22:28,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:22:28,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:22:28,078 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:22:28,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:22:28,145 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:22:28,147 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 09:22:28,210 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['100234 memcached\n'], []) 2022-02-08 09:22:28,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 09:22:28,237 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 09:22:28,269 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['100234 memcached\n'], []) 2022-02-08 09:22:28,269 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:22:28,272 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 09:22:29,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:22:29,795 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:22:29,986 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:22:29,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:22:30,000 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:22:30,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:22:30,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:22:30,048 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:22:30,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:22:30,091 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:22:30,118 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:22:30,118 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 09:22:30,190 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['4649 memcached\n'], []) 2022-02-08 09:22:30,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 09:22:30,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 09:22:30,242 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['4649 memcached\n'], []) 2022-02-08 09:22:30,242 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:22:30,244 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:22:31,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:22:31,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:22:31,811 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:22:31,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:22:31,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:22:31,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:22:31,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:22:31,888 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:22:31,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:22:31,938 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:22:31,963 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:22:31,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 09:22:32,066 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['69146 memcached\n'], []) 2022-02-08 09:22:32,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 09:22:32,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 09:22:32,121 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['69146 memcached\n'], []) 2022-02-08 09:22:32,122 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:22:32,125 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:22:32,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:22:33,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 09:22:33,388 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:22:33,388 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:22:33,410 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:22:33,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:22:33,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:22:33,461 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:22:33,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 09:22:33,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:22:33,520 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:22:33,520 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 09:22:33,573 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['84331 memcached\n'], []) 2022-02-08 09:22:33,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 09:22:33,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 09:22:33,627 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['84331 memcached\n'], []) 2022-02-08 09:22:33,628 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:22:33,630 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:22:34,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:22:34,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:22:34,684 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:22:34,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:22:34,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:22:34,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:22:34,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:22:34,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:22:34,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:22:34,812 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:22:34,828 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 09:22:34,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 09:22:34,892 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['20015 memcached\n'], []) 2022-02-08 09:22:34,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 09:22:34,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 09:22:34,967 | infra | DEBUG | Thread-13178 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['20015 memcached\n'], []) 2022-02-08 09:22:34,967 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:22:34,970 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:22:35,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:22:35,280 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:22:35,345 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:22:35,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:22:35,359 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:22:35,372 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:22:35,385 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:22:35,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:22:35,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:22:35,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:22:35,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:22:35,459 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 09:22:35,490 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:22:35,506 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:22:41,367 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:22:42,178 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:22:42,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:22:43,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:22:43,101 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 09:22:43,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:22:43,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:22:43,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:22:43,214 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 09:22:43,214 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:22:43,217 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:22:43,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:22:43,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:22:43,596 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:22:43,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:22:43,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:22:43,625 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:22:43,638 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:22:43,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:22:43,665 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:22:43,680 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:22:43,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:22:43,707 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 09:22:43,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:22:43,747 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 12061 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:22:50,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:22:51,418 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:22:51,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:22:52,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:22:52,305 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 09:22:52,305 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:22:52,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:22:52,410 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:22:52,426 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 09:22:52,427 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:22:52,428 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 09:22:52,520 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:22:52,756 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 09:22:52,831 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:22:52,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 09:22:52,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 09:22:52,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 09:22:52,877 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 09:22:52,895 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 09:22:52,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 09:22:52,926 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 09:22:52,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:22:52,957 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 09:22:52,982 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:22:52,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:22:53,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:22:53,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:22:53,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:22:53,058 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:22:53,075 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 09:22:53,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:22:53,092 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:22:53,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:22:53,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:22:53,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:22:53,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:22:53,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:22:53,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:22:53,272 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:22:53,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:22:53,328 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:22:53,357 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:22:53,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:22:53,413 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:22:53,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:22:53,450 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:22:53,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:22:53,484 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:22:53,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:22:53,520 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:22:53,538 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:22:53,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:22:53,573 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:22:53,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:22:53,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:22:53,627 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:22:53,647 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:22:53,665 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:22:53,683 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:22:53,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:22:53,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:22:53,740 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:22:53,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:22:53,789 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:22:53,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:22:53,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:22:53,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:22:53,867 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:22:53,885 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:22:53,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:22:53,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:22:53,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:22:53,959 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:22:53,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:22:53,994 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:22:54,016 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:22:54,035 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:22:54,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:22:54,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:22:54,088 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:22:54,105 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:22:54,124 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:22:54,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:22:54,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:22:54,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:22:54,196 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:22:54,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:22:54,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:22:54,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:22:54,267 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:22:54,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:22:54,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:22:54,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:22:54,339 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:22:54,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:22:54,387 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:22:54,404 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:22:54,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:22:54,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:22:54,456 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:22:54,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:22:54,493 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:22:54,510 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:22:54,528 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:22:54,546 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:22:54,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:22:54,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:22:54,599 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:22:54,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:22:54,635 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:22:54,655 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:22:54,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:22:54,693 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:22:54,710 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 09:22:54,711 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:22:54,967 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:22:55,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:22:55,848 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:22:55,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:22:55,931 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:22:56,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:22:56,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:22:56,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:22:56,451 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:22:56,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:22:56,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:22:56,773 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 09:22:56,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:22:56,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:23:01,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:23:02,673 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:03,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:03,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:23:03,828 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 09:23:03,829 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:23:03,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:23:04,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:23:04,184 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 09:23:04,184 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:23:04,187 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 09:23:04,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:23:04,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 09:23:04,561 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:23:04,561 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 09:23:04,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 09:23:04,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 09:23:04,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 09:23:04,618 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 09:23:04,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 09:23:04,645 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 09:23:04,658 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:23:04,674 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 09:23:04,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:23:04,714 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:23:04,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:23:04,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:04,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:04,773 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:23:04,786 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 09:23:04,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:23:04,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:23:04,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:23:04,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:04,858 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:23:04,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:04,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:04,953 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:04,979 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:05,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:05,039 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:05,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:05,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:05,127 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:05,144 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:23:05,163 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:05,183 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:05,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:05,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:05,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:05,250 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:05,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:05,285 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:05,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:05,318 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:23:05,336 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:05,355 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:05,371 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:05,390 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:05,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:05,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:05,443 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:05,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:05,476 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:05,503 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:23:05,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:05,551 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:05,571 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:05,588 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:05,605 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:05,622 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:05,641 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:05,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:05,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:05,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:23:05,707 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:05,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:05,739 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:05,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:05,770 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:05,786 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:05,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:05,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:05,834 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:05,849 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:23:05,867 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:05,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:05,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:05,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:05,933 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:05,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:05,963 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:05,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:05,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:06,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:23:06,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:06,038 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:06,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:06,068 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:06,082 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:06,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:06,117 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:06,132 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:06,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:06,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:23:06,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:06,203 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:06,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:06,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:06,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:06,270 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:06,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:06,303 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:06,321 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 09:23:06,322 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:23:06,410 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:23:06,611 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:23:06,680 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:23:06,681 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:23:06,697 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:23:06,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:23:06,726 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:23:06,744 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:23:06,759 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:23:06,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:23:06,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:23:06,799 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 09:23:06,828 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:23:06,842 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:23:13,025 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:23:14,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:14,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:15,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:23:15,173 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 09:23:15,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:23:15,193 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:23:15,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:23:15,279 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 09:23:15,279 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:23:15,280 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:23:15,372 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:23:15,805 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:23:16,312 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:23:16,313 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:23:16,400 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:23:16,536 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:23:16,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:23:16,717 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:23:16,895 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:23:17,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:23:17,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:23:17,596 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 09:23:17,631 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:23:17,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:23:24,417 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:23:25,262 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:25,732 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:26,233 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:23:26,322 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 09:23:26,322 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:23:26,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:23:26,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:23:26,532 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 09:23:26,532 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:23:26,535 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 09:23:26,644 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:23:26,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 09:23:27,053 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:23:27,053 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 09:23:27,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 09:23:27,094 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 09:23:27,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 09:23:27,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 09:23:27,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 09:23:27,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 09:23:27,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:23:27,214 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 09:23:27,249 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:23:27,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:23:27,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:23:27,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:27,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:27,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:23:27,374 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 09:23:27,374 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:23:27,394 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:23:27,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:23:27,431 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:27,463 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:23:27,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:27,535 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:27,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:27,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:27,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:27,668 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:27,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:27,733 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:27,760 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:27,780 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:23:27,801 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:27,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:27,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:27,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:27,897 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:27,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:27,944 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:27,966 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:27,986 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:28,013 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:23:28,036 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:28,059 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:28,081 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:28,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:28,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:28,147 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:28,167 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:28,187 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:28,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:28,230 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:23:28,255 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:28,273 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:28,296 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:28,316 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:28,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:28,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:28,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:28,398 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:28,423 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:28,448 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:23:28,470 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:28,489 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:28,509 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:28,532 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:28,552 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:28,572 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:28,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:28,614 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:28,632 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:28,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:23:28,674 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:28,690 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:28,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:28,732 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:28,753 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:28,776 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:28,798 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:28,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:28,839 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:28,862 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:23:28,887 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:28,914 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:28,934 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:28,951 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:28,971 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:28,990 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:29,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:29,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:29,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:29,069 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:23:29,094 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:29,117 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:29,140 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:29,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:29,184 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:29,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:29,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:29,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:29,273 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 09:23:29,276 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 09:23:29,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:23:29,594 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 09:23:29,664 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:23:29,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 09:23:29,678 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 09:23:29,691 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 09:23:29,704 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 09:23:29,723 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 09:23:29,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 09:23:29,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 09:23:29,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:23:29,779 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 09:23:29,806 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:23:29,822 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:23:29,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:23:29,852 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:29,867 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:29,881 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:23:29,894 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 09:23:29,895 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:23:29,911 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:23:29,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:23:29,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:29,969 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:23:30,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:30,029 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:30,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:30,094 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:30,125 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:30,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:30,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:30,220 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:23:30,247 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:30,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:23:30,289 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:30,309 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:30,329 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:30,346 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:30,365 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:30,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:30,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:30,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:23:30,441 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:30,460 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:23:30,479 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:30,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:30,515 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:30,530 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:30,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:30,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:30,589 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:30,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:23:30,634 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:30,657 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:23:30,677 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:30,694 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:30,713 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:30,730 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:30,750 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:30,766 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:30,785 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:30,802 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:23:30,819 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:30,835 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:23:30,852 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:30,868 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:30,885 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:30,904 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:30,926 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:30,943 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:30,959 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:30,976 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:23:30,992 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:31,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:23:31,026 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:31,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:31,062 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:31,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:31,095 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:31,111 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:31,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:31,142 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:23:31,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:31,176 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:23:31,194 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:31,210 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:31,224 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:31,239 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:31,253 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:31,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:31,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:31,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:23:31,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:31,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:23:31,354 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:31,369 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:31,385 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:31,403 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:31,420 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:31,437 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:31,454 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:31,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:23:31,490 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 09:23:31,492 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:23:31,592 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:23:31,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:23:32,051 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:23:32,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:23:32,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:23:32,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:23:32,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:23:32,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:23:32,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:23:32,190 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:23:32,206 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:23:32,229 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 09:23:32,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:23:32,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:23:38,243 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:23:39,121 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:39,608 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:40,045 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:23:40,095 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 09:23:40,096 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:23:40,114 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:23:40,180 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:23:40,207 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 09:23:40,207 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:23:40,211 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:23:40,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:23:40,711 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 09:23:40,832 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:23:40,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:23:40,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:23:40,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:23:40,927 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:23:40,974 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:23:41,012 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 09:23:41,038 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:23:41,056 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:23:41,075 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 09:23:41,099 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:23:41,115 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:23:46,269 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:23:47,009 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:47,467 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:48,006 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:23:48,046 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 09:23:48,046 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:23:48,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:23:48,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:23:48,203 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 09:23:48,203 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:23:48,207 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:23:48,838 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:23:49,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:23:49,246 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:23:49,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:23:49,262 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:23:49,286 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:23:49,301 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:23:49,319 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:23:49,335 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:23:49,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:23:49,377 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:23:49,392 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 09:23:49,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:23:49,440 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 12084 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:23:54,421 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:23:55,137 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:55,496 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:23:55,836 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:23:55,857 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 09:23:55,857 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:23:55,871 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:23:55,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:23:55,933 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 09:23:55,933 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:23:55,934 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 09:23:56,049 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:23:56,325 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:23:56,424 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:23:56,424 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:23:56,444 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:23:56,473 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:23:56,512 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:23:56,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:23:56,553 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:23:56,569 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:23:56,601 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:23:56,624 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 09:23:56,654 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:23:56,673 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:24:05,177 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:24:06,214 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:24:06,831 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:24:07,339 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:24:07,384 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 09:24:07,384 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:24:07,407 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:24:07,469 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:24:07,490 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 09:24:07,490 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:24:07,493 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 09:24:07,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:24:07,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 09:24:07,892 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:24:07,892 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 09:24:07,907 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 09:24:07,923 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 09:24:07,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 09:24:07,957 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 09:24:07,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 09:24:07,996 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 09:24:08,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 09:24:08,036 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 09:24:08,063 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 09:24:08,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:24:08,096 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 09:24:08,114 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:24:08,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 09:24:08,141 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 09:24:08,155 | test | DEBUG | Thread-13178 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 09:24:08,157 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 09:24:08,171 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 09:24:08,186 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 09:24:08,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:24:08,229 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 09:24:08,259 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:24:08,288 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:24:08,321 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:24:08,351 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:24:08,382 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:24:08,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:24:08,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:24:08,477 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 09:24:08,505 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:24:08,523 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 09:24:08,545 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:24:08,562 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:24:08,579 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:24:08,596 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:24:08,612 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:24:08,630 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:24:08,648 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:24:08,664 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 09:24:08,681 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:24:08,700 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 09:24:08,719 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:24:08,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:24:08,755 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:24:08,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:24:08,792 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:24:08,809 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:24:08,829 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:24:08,848 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 09:24:08,865 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:24:08,882 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 09:24:08,905 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:24:08,930 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:24:08,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:24:08,964 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:24:08,983 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:24:09,003 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:24:09,023 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:24:09,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 09:24:09,061 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:24:09,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 09:24:09,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:24:09,114 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:24:09,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:24:09,148 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:24:09,164 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:24:09,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:24:09,200 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:24:09,219 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 09:24:09,236 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:24:09,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 09:24:09,275 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:24:09,292 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:24:09,311 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:24:09,326 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:24:09,345 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:24:09,361 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:24:09,378 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:24:09,395 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 09:24:09,414 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:24:09,430 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 09:24:09,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:24:09,464 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:24:09,484 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:24:09,502 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:24:09,517 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:24:09,533 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:24:09,549 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:24:09,565 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 09:24:09,582 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:24:09,598 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 09:24:09,615 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:24:09,632 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:24:09,650 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:24:09,667 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:24:09,684 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:24:09,701 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:24:09,719 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:24:09,736 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 09:24:09,752 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 09:24:09,753 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:24:09,755 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 09:24:09,901 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:24:10,122 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 09:24:10,187 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:24:10,188 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 09:24:10,204 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 09:24:10,217 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 09:24:10,246 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 09:24:10,266 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 09:24:10,282 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 09:24:10,302 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 09:24:10,315 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:24:10,823 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 09:24:10,825 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.0720000267. 2022-02-08 09:24:10,825 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:24:10,826 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 09:24:11,555 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:24:11,894 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 09:24:12,022 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:24:12,022 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 09:24:12,038 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 09:24:12,065 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 09:24:12,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 09:24:12,130 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 09:24:12,155 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 09:24:12,198 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 09:24:12,226 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:24:12,938 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:24:12,940 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 09:24:14,940 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:24:15,150 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:24:15,151 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 09:24:17,151 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:24:17,351 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:24:17,351 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 09:24:19,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:24:19,539 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:24:19,539 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 09:24:21,539 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:24:21,720 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:24:21,720 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 09:24:23,720 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:24:23,898 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:24:23,898 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 09:24:25,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:24:26,089 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 09:24:26,091 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 15.2660000324. 2022-02-08 09:24:26,091 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:24:26,092 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 09:24:27,128 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:24:27,818 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 09:24:27,970 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:24:27,970 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 09:24:27,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 09:24:28,020 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 09:24:28,048 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 09:24:28,085 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 09:24:28,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 09:24:28,154 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 09:24:28,174 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:24:28,979 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 09:24:28,979 | test | INFO | Thread-13178 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 09:24:30,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:24:31,184 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 09:24:31,184 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.09399986267. 2022-02-08 09:24:31,184 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:24:31,187 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 09:24:32,207 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:24:32,872 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 09:24:33,051 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:24:33,052 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 09:24:33,079 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 09:24:33,101 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 09:24:33,131 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 09:24:33,181 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 09:24:33,240 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 09:24:33,381 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 09:24:33,415 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:24:34,246 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 09:24:34,259 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.07500004768. 2022-02-08 09:24:34,260 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:24:34,263 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 09:24:35,256 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:24:35,816 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 09:24:35,980 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:24:35,980 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 09:24:36,002 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 09:24:36,025 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 09:24:36,051 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 09:24:36,076 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 09:24:36,094 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 09:24:36,138 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 09:24:36,160 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:24:36,855 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 09:24:36,855 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.59500002861. 2022-02-08 09:24:36,855 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:24:36,858 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 09:24:37,447 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:24:37,808 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 09:24:37,917 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:24:37,918 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 09:24:37,937 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 09:24:37,960 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 09:24:37,989 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 09:24:38,019 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 09:24:38,043 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 09:24:38,075 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 09:24:38,098 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:24:39,282 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 09:24:39,283 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.4279999733. 2022-02-08 09:24:39,283 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:24:39,286 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 09:24:40,108 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:24:40,575 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 09:24:40,749 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:24:40,749 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 09:24:40,763 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 09:24:40,783 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 09:24:40,806 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 09:24:40,832 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 09:24:40,846 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 09:24:40,875 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 09:24:40,891 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:24:41,555 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 09:24:41,556 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.27200007439. 2022-02-08 09:24:41,556 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:24:41,559 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 09:24:42,352 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:24:42,716 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 09:24:42,851 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:24:42,851 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 09:24:42,878 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 09:24:42,898 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 09:24:42,921 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 09:24:42,947 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 09:24:42,977 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 09:24:43,010 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 09:24:43,063 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:24:44,000 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 09:24:44,000 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.44400000572. 2022-02-08 09:24:44,000 | test | DEBUG | Thread-13178 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 09:24:44,003 | infra | DEBUG | Thread-13178 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 09:24:45,223 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 09:24:45,772 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 09:24:45,993 | infra | DEBUG | Thread-13178 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 09:24:45,993 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 09:24:46,015 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 09:24:46,042 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 09:24:46,072 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 09:24:46,134 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 09:24:46,153 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 09:24:46,183 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 09:24:46,213 | infra | DEBUG | Thread-13178 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 09:24:47,239 | infra | DEBUG | Thread-13178 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 09:24:47,239 | test | INFO | Thread-13178 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.2380001545. 2022-02-08 09:24:47,242 | test | DEBUG | Thread-13178 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L836 2022-02-08 09:24:47,242 | test | DEBUG | Thread-13178 | [common_lib:sleep:24] Sleep 300 seconds. Reason: None 5 queries submitted, 0 failed, 12111 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 12136 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 12162 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 12181 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 12209 passed, 0 rejected, 0 cancelled, 0 timeout ############################################################ # # Step 12: Failover a node and FullRecovery that node # ############################################################ 2022-02-08 09:29:49,088 | test | DEBUG | MainThread | [rest_client:fail_over:1389] ns_1@172.23.121.115 - Failover successful 2022-02-08 09:29:49,096 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/Hospital.py -> SystemTestMagma():L640 2022-02-08 09:29:49,096 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 30 seconds. Reason: None 5 queries submitted, 0 failed, 12232 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:30:19,114 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 53.35 % 2022-02-08 09:30:19,121 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:30:19,122 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:30:22,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 55.14 % 2022-02-08 09:30:22,151 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:30:22,151 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:30:25,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.70 % 2022-02-08 09:30:25,178 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:30:25,178 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:30:28,201 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.00 % 2022-02-08 09:30:28,209 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:30:28,209 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:30:31,227 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 66.51 % 2022-02-08 09:30:31,236 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:30:31,236 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:30:34,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 67.40 % 2022-02-08 09:30:34,265 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:30:34,266 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:30:37,283 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 71.80 % 2022-02-08 09:30:37,292 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:30:37,292 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:30:40,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 72.69 % 2022-02-08 09:30:40,312 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:30:40,312 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:30:43,334 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 77.99 % 2022-02-08 09:30:43,342 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:30:43,342 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:30:46,364 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 84.07 % 2022-02-08 09:30:46,371 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:30:46,371 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:30:49,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 84.96 % 2022-02-08 09:30:49,394 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:30:49,395 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 12256 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:30:52,420 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 88.52 % 2022-02-08 09:30:52,431 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:30:52,431 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:30:55,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 91.14 % 2022-02-08 09:30:55,457 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:30:55,457 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:30:58,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 94.65 % 2022-02-08 09:30:58,483 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:30:58,483 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:31:01,510 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 98.22 % 2022-02-08 09:31:01,517 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:31:01,519 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:31:04,540 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 100.00 % 2022-02-08 09:31:04,551 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:31:04,551 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:31:07,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 100.00 % 2022-02-08 09:31:07,611 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:31:07,611 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:31:10,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 100.00 % 2022-02-08 09:31:10,644 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:31:10,645 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:31:13,714 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 92.59 % 2022-02-08 09:31:13,724 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:31:13,726 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:31:16,749 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-08 09:31:16,750 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 09:31:21,769 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 09:31:21,770 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 09:31:24,785 | test | INFO | MainThread | [rest_client:monitorRebalance:1599] Rebalance done. Taken 65.6870000362 seconds to complete 2022-02-08 09:31:24,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1602 2022-02-08 09:31:24,796 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-08 09:31:29,796 | test | DEBUG | MainThread | [rest_client:set_recovery_type:1397] ns_1@172.23.121.115 - Setting recoveryType=full 2022-02-08 09:31:29,835 | test | DEBUG | MainThread | [rest_client:set_recovery_type:1412] ns_1@172.23.121.115 - RecoveryType set successful 2022-02-08 09:31:29,891 | infra | DEBUG | MainThread | [task:__init__:249] Last known rebalance status_id: None 2022-02-08 09:31:29,892 | infra | INFO | pool-3-thread-18 | [task:start_task:91] Thread 'Rebalance_task_IN=[]_OUT=[]_1644341489.84' started 2022-02-08 09:31:29,892 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[]_1644341489.84 2022-02-08 09:31:29,894 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[]_1644341489.84 2022-02-08 09:31:29,948 | test | DEBUG | pool-3-thread-18 | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.115,ns_1@172.23.121.132,ns_1@172.23.121.135,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.134,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123,ns_1@172.23.121.131,ns_1@172.23.121.130,ns_1@172.23.121.133', 'ejectedNodes': '', 'user': 'Administrator'} 2022-02-08 09:31:30,009 | test | DEBUG | pool-3-thread-18 | [rest_client:rebalance:1459] Rebalance operation started 2022-02-08 09:31:30,025 | test | INFO | pool-3-thread-18 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.121.115 | kv | 7.1.0-2255-enterprise | 13.5666535588 | Cluster node | | 172.23.121.132 | index, n1ql | 7.1.0-2255-enterprise | 3.87261146497 | Cluster node | | 172.23.121.135 | kv | 7.1.0-2255-enterprise | 73.2671991755 | Cluster node | | 172.23.120.170 | kv | 7.1.0-2255-enterprise | 41.7239555791 | Cluster node | | 172.23.121.129 | kv | 7.1.0-2255-enterprise | 38.9937937409 | Cluster node | | 172.23.121.124 | kv | 7.1.0-2255-enterprise | 44.6900908014 | Cluster node | | 172.23.121.126 | kv | 7.1.0-2255-enterprise | 37.5681787947 | Cluster node | | 172.23.121.134 | index, n1ql | 7.1.0-2255-enterprise | 8.97144301238 | Cluster node | | 172.23.121.116 | kv | 7.1.0-2255-enterprise | 73.179533445 | Cluster node | | 172.23.121.127 | kv | 7.1.0-2255-enterprise | 38.9337213917 | Cluster node | | 172.23.121.123 | kv | 7.1.0-2255-enterprise | 37.2228088701 | Cluster node | | 172.23.121.131 | index, n1ql | 7.1.0-2255-enterprise | 18.2020705019 | Cluster node | | 172.23.121.130 | index, n1ql | 7.1.0-2255-enterprise | 7.76711452335 | Cluster node | | 172.23.121.133 | index, n1ql | 7.1.0-2255-enterprise | 3.70323237465 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2022-02-08 09:31:30,043 | infra | DEBUG | pool-3-thread-18 | [task:call:316] New rebalance status_id: 3a07f65e4a838db61c11d0032020a464 2022-02-08 09:31:30,059 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 09:31:30,059 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 09:31:30,096 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:31:30,101 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:31:30,101 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:31:40,115 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 09:31:40,117 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 09:31:40,148 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:31:40,151 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:31:40,151 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12282 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:31:50,181 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 09:31:50,181 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 09:31:50,221 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:31:50,226 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:31:50,226 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:32:00,243 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 09:32:00,244 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 09:32:00,273 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:32:00,276 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:32:00,276 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:32:10,301 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 09:32:10,301 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 09:32:10,345 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:32:10,349 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:32:10,349 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:32:20,365 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 09:32:20,365 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 09:32:20,398 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:32:20,401 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:32:20,401 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:32:30,434 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 09:32:30,434 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 09:32:30,470 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:32:30,473 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:32:30,473 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:32:40,493 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 09:32:40,493 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 09:32:40,526 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:32:40,529 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:32:40,529 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12302 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:32:50,549 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 09:32:50,549 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 09:32:50,578 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:32:50,582 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:32:50,582 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:33:00,641 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.23 % 2022-02-08 09:33:00,641 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 0.23 2022-02-08 09:33:00,677 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:33:00,680 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:33:00,680 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:33:10,698 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.47 % 2022-02-08 09:33:10,698 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 0.47 2022-02-08 09:33:10,736 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:33:10,740 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:33:10,740 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:33:20,785 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.70 % 2022-02-08 09:33:20,786 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 0.7 2022-02-08 09:33:20,825 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:33:20,828 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:33:20,828 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:33:30,861 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.93 % 2022-02-08 09:33:30,861 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 0.93 2022-02-08 09:33:30,897 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:33:30,898 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:33:30,900 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:33:40,917 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.40 % 2022-02-08 09:33:40,918 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 1.4 2022-02-08 09:33:40,950 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:33:40,951 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:33:40,953 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12328 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:33:50,970 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.63 % 2022-02-08 09:33:50,970 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 1.63 2022-02-08 09:33:51,002 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:33:51,006 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:33:51,006 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:34:01,023 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.86 % 2022-02-08 09:34:01,023 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 1.86 2022-02-08 09:34:01,062 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:34:01,065 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:34:01,065 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:34:11,091 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.09 % 2022-02-08 09:34:11,092 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 2.09 2022-02-08 09:34:11,145 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:34:11,150 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:34:11,150 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:34:21,167 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.09 % 2022-02-08 09:34:21,167 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 2.09 2022-02-08 09:34:21,203 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:34:21,207 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:34:21,207 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:34:31,227 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.79 % 2022-02-08 09:34:31,227 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 2.79 2022-02-08 09:34:31,282 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:34:31,285 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:34:31,286 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:34:41,309 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.79 % 2022-02-08 09:34:41,309 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 2.79 2022-02-08 09:34:41,361 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:34:41,374 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:34:41,374 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12353 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:34:51,392 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.03 % 2022-02-08 09:34:51,394 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 3.03 2022-02-08 09:34:51,434 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:34:51,437 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:34:51,438 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:35:01,453 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.26 % 2022-02-08 09:35:01,453 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 3.26 2022-02-08 09:35:01,483 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:35:01,486 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:35:01,486 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:35:11,506 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.49 % 2022-02-08 09:35:11,507 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 3.49 2022-02-08 09:35:11,545 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:35:11,548 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:35:11,548 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:35:21,572 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.49 % 2022-02-08 09:35:21,573 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 3.49 2022-02-08 09:35:21,615 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:35:21,618 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:35:21,618 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:35:31,644 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.96 % 2022-02-08 09:35:31,644 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 3.96 2022-02-08 09:35:31,703 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:35:31,707 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:35:31,709 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:35:41,726 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.42 % 2022-02-08 09:35:41,726 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 4.42 2022-02-08 09:35:41,759 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:35:41,762 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:35:41,762 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12380 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:35:51,799 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.65 % 2022-02-08 09:35:51,799 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 4.65 2022-02-08 09:35:51,832 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:35:51,834 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:35:51,834 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:36:01,851 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.65 % 2022-02-08 09:36:01,851 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 4.65 2022-02-08 09:36:01,903 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:36:01,911 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:36:01,913 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:36:11,934 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.89 % 2022-02-08 09:36:11,936 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 4.89 2022-02-08 09:36:11,980 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:36:11,984 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:36:11,984 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:36:22,003 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.12 % 2022-02-08 09:36:22,003 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 5.12 2022-02-08 09:36:22,040 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:36:22,043 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:36:22,043 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:36:32,082 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.12 % 2022-02-08 09:36:32,082 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 5.12 2022-02-08 09:36:32,128 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:36:32,131 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:36:32,131 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:36:42,151 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.35 % 2022-02-08 09:36:42,151 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 5.35 2022-02-08 09:36:42,190 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:36:42,193 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:36:42,193 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12406 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:36:52,210 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.59 % 2022-02-08 09:36:52,211 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 5.59 2022-02-08 09:36:52,246 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:36:52,250 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:36:52,250 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:37:02,263 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.59 % 2022-02-08 09:37:02,263 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 5.59 2022-02-08 09:37:02,305 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:37:02,311 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:37:02,311 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:37:12,344 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.59 % 2022-02-08 09:37:12,345 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 5.59 2022-02-08 09:37:12,382 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:37:12,385 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:37:12,387 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:37:22,417 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.82 % 2022-02-08 09:37:22,418 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 5.82 2022-02-08 09:37:22,470 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:37:22,474 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:37:22,474 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:37:32,493 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.05 % 2022-02-08 09:37:32,493 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 6.05 2022-02-08 09:37:32,539 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:37:32,542 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:37:32,543 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:37:42,562 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.52 % 2022-02-08 09:37:42,562 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 6.52 2022-02-08 09:37:42,599 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:37:42,601 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:37:42,602 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12429 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:37:52,619 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.52 % 2022-02-08 09:37:52,619 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 6.52 2022-02-08 09:37:52,661 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:37:52,664 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:37:52,664 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:38:02,681 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.52 % 2022-02-08 09:38:02,681 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 6.52 2022-02-08 09:38:02,716 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:38:02,719 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:38:02,719 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:38:12,743 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.52 % 2022-02-08 09:38:12,743 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 6.52 2022-02-08 09:38:12,789 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:38:12,792 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:38:12,792 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:38:22,811 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.75 % 2022-02-08 09:38:22,811 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 6.75 2022-02-08 09:38:22,851 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:38:22,854 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:38:22,854 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:38:32,871 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.45 % 2022-02-08 09:38:32,872 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 7.45 2022-02-08 09:38:32,914 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:38:32,917 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:38:32,917 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:38:42,937 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.45 % 2022-02-08 09:38:42,937 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 7.45 2022-02-08 09:38:42,987 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:38:42,993 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:38:42,993 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12453 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:38:53,009 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.45 % 2022-02-08 09:38:53,010 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 7.45 2022-02-08 09:38:53,059 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:38:53,062 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:38:53,063 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:39:03,085 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.45 % 2022-02-08 09:39:03,085 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 7.45 2022-02-08 09:39:03,125 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:39:03,128 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:39:03,128 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:39:13,148 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.45 % 2022-02-08 09:39:13,150 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 7.45 2022-02-08 09:39:13,190 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:39:13,194 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:39:13,196 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:39:23,213 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.68 % 2022-02-08 09:39:23,213 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 7.68 2022-02-08 09:39:23,250 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:39:23,252 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:39:23,253 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:39:33,282 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.15 % 2022-02-08 09:39:33,282 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 8.15 2022-02-08 09:39:33,323 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:39:33,326 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:39:33,326 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:39:43,346 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.38 % 2022-02-08 09:39:43,348 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 8.38 2022-02-08 09:39:43,378 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:39:43,381 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:39:43,381 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12479 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:39:53,403 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.38 % 2022-02-08 09:39:53,404 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 8.38 2022-02-08 09:39:53,440 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:39:53,443 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:39:53,443 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:40:03,467 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.38 % 2022-02-08 09:40:03,467 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 8.38 2022-02-08 09:40:03,509 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:40:03,515 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:40:03,515 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:40:13,538 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.38 % 2022-02-08 09:40:13,538 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 8.38 2022-02-08 09:40:13,589 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:40:13,595 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:40:13,595 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:40:23,608 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.38 % 2022-02-08 09:40:23,609 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 8.38 2022-02-08 09:40:23,640 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:40:23,641 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:40:23,641 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:40:33,657 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.61 % 2022-02-08 09:40:33,657 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 8.61 2022-02-08 09:40:33,687 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:40:33,690 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:40:33,690 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:40:43,704 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.08 % 2022-02-08 09:40:43,706 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 9.08 2022-02-08 09:40:43,740 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:40:43,744 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:40:43,744 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12499 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:40:53,766 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.08 % 2022-02-08 09:40:53,766 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 9.08 2022-02-08 09:40:53,803 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:40:53,806 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:40:53,808 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:41:03,828 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.31 % 2022-02-08 09:41:03,828 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 9.31 2022-02-08 09:41:03,901 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:41:03,904 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:41:03,904 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:41:13,918 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.31 % 2022-02-08 09:41:13,920 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 9.31 2022-02-08 09:41:13,973 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:41:13,976 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:41:13,976 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:41:23,992 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.31 % 2022-02-08 09:41:23,992 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 9.31 2022-02-08 09:41:24,029 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:41:24,032 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:41:24,032 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:41:34,048 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.31 % 2022-02-08 09:41:34,048 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 9.31 2022-02-08 09:41:34,115 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:41:34,119 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:41:34,119 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:41:44,164 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.01 % 2022-02-08 09:41:44,164 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 10.01 2022-02-08 09:41:44,201 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:41:44,204 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:41:44,204 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12526 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:41:54,223 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.24 % 2022-02-08 09:41:54,223 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 10.24 2022-02-08 09:41:54,269 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:41:54,273 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:41:54,273 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:42:04,295 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.24 % 2022-02-08 09:42:04,295 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 10.24 2022-02-08 09:42:04,331 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:42:04,334 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:42:04,335 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:42:14,348 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.24 % 2022-02-08 09:42:14,349 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 10.24 2022-02-08 09:42:14,388 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:42:14,391 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:42:14,391 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:42:24,407 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.24 % 2022-02-08 09:42:24,407 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 10.24 2022-02-08 09:42:24,440 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:42:24,443 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:42:24,444 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:42:34,461 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.24 % 2022-02-08 09:42:34,461 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 10.24 2022-02-08 09:42:34,505 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:42:34,507 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:42:34,507 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:42:44,532 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.47 % 2022-02-08 09:42:44,532 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 10.47 2022-02-08 09:42:44,569 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:42:44,572 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:42:44,572 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12552 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:42:54,585 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.17 % 2022-02-08 09:42:54,585 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 11.17 2022-02-08 09:42:54,618 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:42:54,621 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:42:54,621 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:43:04,641 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.17 % 2022-02-08 09:43:04,641 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 11.17 2022-02-08 09:43:04,678 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:43:04,681 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:43:04,681 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:43:14,697 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.17 % 2022-02-08 09:43:14,697 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 11.17 2022-02-08 09:43:14,740 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:43:14,743 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:43:14,743 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:43:24,785 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.17 % 2022-02-08 09:43:24,785 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 11.17 2022-02-08 09:43:24,818 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:43:24,821 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:43:24,821 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:43:34,838 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.17 % 2022-02-08 09:43:34,838 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 11.17 2022-02-08 09:43:34,872 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:43:34,875 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:43:34,875 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:43:44,898 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.17 % 2022-02-08 09:43:44,898 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 11.17 2022-02-08 09:43:44,950 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:43:44,953 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:43:44,953 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12580 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:43:54,993 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.40 % 2022-02-08 09:43:54,994 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 11.4 2022-02-08 09:43:55,023 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:43:55,026 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:43:55,026 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:44:05,042 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.87 % 2022-02-08 09:44:05,042 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 11.87 2022-02-08 09:44:05,085 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:44:05,088 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:44:05,088 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:44:15,105 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.10 % 2022-02-08 09:44:15,105 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 12.1 2022-02-08 09:44:15,141 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:44:15,144 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:44:15,144 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:44:25,160 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.10 % 2022-02-08 09:44:25,160 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 12.1 2022-02-08 09:44:25,203 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:44:25,207 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:44:25,207 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:44:35,236 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.10 % 2022-02-08 09:44:35,236 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 12.1 2022-02-08 09:44:35,269 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:44:35,273 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:44:35,273 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:44:45,286 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.10 % 2022-02-08 09:44:45,288 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 12.1 2022-02-08 09:44:45,322 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:44:45,325 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:44:45,325 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12603 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:44:55,351 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 12.33 % 2022-02-08 09:44:55,351 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 12.33 2022-02-08 09:44:55,391 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:44:55,394 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:44:55,395 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:45:05,410 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.03 % 2022-02-08 09:45:05,410 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 13.03 2022-02-08 09:45:05,463 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:45:05,467 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:45:05,467 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:45:15,479 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.03 % 2022-02-08 09:45:15,480 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 13.03 2022-02-08 09:45:15,510 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:45:15,513 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:45:15,513 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:45:25,542 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.03 % 2022-02-08 09:45:25,542 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 13.03 2022-02-08 09:45:25,579 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:45:25,582 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:45:25,584 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:45:35,598 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.03 % 2022-02-08 09:45:35,598 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 13.03 2022-02-08 09:45:35,638 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:45:35,641 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:45:35,642 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:45:45,655 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.03 % 2022-02-08 09:45:45,655 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 13.03 2022-02-08 09:45:45,690 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:45:45,691 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:45:45,693 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12625 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:45:55,707 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.27 % 2022-02-08 09:45:55,709 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 13.27 2022-02-08 09:45:55,753 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:45:55,756 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:45:55,756 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:46:05,769 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.73 % 2022-02-08 09:46:05,769 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 13.73 2022-02-08 09:46:05,809 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:46:05,812 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:46:05,812 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:46:15,832 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.96 % 2022-02-08 09:46:15,832 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 13.96 2022-02-08 09:46:15,884 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:46:15,887 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:46:15,888 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:46:25,901 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.96 % 2022-02-08 09:46:25,901 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 13.96 2022-02-08 09:46:25,943 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:46:25,946 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:46:25,946 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:46:35,980 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.96 % 2022-02-08 09:46:35,980 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 13.96 2022-02-08 09:46:36,023 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:46:36,026 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:46:36,026 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:46:46,038 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 13.96 % 2022-02-08 09:46:46,038 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 13.96 2022-02-08 09:46:46,065 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:46:46,068 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:46:46,069 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12647 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:46:56,084 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.20 % 2022-02-08 09:46:56,084 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 14.2 2022-02-08 09:46:56,122 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:46:56,127 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:46:56,127 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:47:06,141 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.20 % 2022-02-08 09:47:06,141 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 14.2 2022-02-08 09:47:06,177 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:47:06,180 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:47:06,180 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:47:16,200 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.20 % 2022-02-08 09:47:16,200 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 14.2 2022-02-08 09:47:16,240 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:47:16,243 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:47:16,243 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:47:26,257 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 14.20 % 2022-02-08 09:47:26,259 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 14.2 2022-02-08 09:47:26,299 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:47:26,302 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:47:26,302 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:47:36,315 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.23 % 2022-02-08 09:47:36,316 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 15.23 2022-02-08 09:47:36,348 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:47:36,351 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:47:36,351 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:47:46,365 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.23 % 2022-02-08 09:47:46,365 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 15.23 2022-02-08 09:47:46,400 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:47:46,403 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:47:46,403 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12669 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:47:56,414 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.23 % 2022-02-08 09:47:56,414 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 15.23 2022-02-08 09:47:56,456 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:47:56,459 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:47:56,459 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:48:06,473 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.23 % 2022-02-08 09:48:06,473 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 15.23 2022-02-08 09:48:06,516 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:48:06,519 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:48:06,519 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:48:16,532 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.57 % 2022-02-08 09:48:16,533 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 15.57 2022-02-08 09:48:16,572 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:48:16,573 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:48:16,575 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:48:26,588 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.57 % 2022-02-08 09:48:26,589 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 15.57 2022-02-08 09:48:26,631 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:48:26,634 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:48:26,634 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:48:36,651 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.57 % 2022-02-08 09:48:36,651 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 15.57 2022-02-08 09:48:36,690 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:48:36,693 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:48:36,694 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:48:46,706 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.91 % 2022-02-08 09:48:46,707 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 15.91 2022-02-08 09:48:46,733 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:48:46,736 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:48:46,736 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12693 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:48:56,749 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 15.91 % 2022-02-08 09:48:56,749 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 15.91 2022-02-08 09:48:56,779 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:48:56,782 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:48:56,782 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:49:06,803 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.60 % 2022-02-08 09:49:06,805 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 16.6 2022-02-08 09:49:06,842 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:49:06,846 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:49:06,846 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:49:16,858 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.60 % 2022-02-08 09:49:16,859 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 16.6 2022-02-08 09:49:16,911 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:49:16,914 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:49:16,914 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:49:26,937 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.60 % 2022-02-08 09:49:26,938 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 16.6 2022-02-08 09:49:26,976 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:49:26,979 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:49:26,979 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:49:37,013 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.83 % 2022-02-08 09:49:37,013 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 16.83 2022-02-08 09:49:37,065 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:49:37,068 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:49:37,068 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:49:47,086 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.83 % 2022-02-08 09:49:47,086 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 16.83 2022-02-08 09:49:47,125 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:49:47,128 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:49:47,128 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12713 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:49:57,141 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.83 % 2022-02-08 09:49:57,141 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 16.83 2022-02-08 09:49:57,178 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:49:57,181 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:49:57,181 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:50:07,197 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.53 % 2022-02-08 09:50:07,197 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 17.53 2022-02-08 09:50:07,227 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:50:07,230 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:50:07,230 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:50:17,253 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.53 % 2022-02-08 09:50:17,253 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 17.53 2022-02-08 09:50:17,296 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:50:17,299 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:50:17,299 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:50:27,312 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.53 % 2022-02-08 09:50:27,312 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 17.53 2022-02-08 09:50:27,346 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:50:27,349 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:50:27,349 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:50:37,365 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.53 % 2022-02-08 09:50:37,365 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 17.53 2022-02-08 09:50:37,400 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:50:37,404 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:50:37,404 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:50:47,420 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.75 % 2022-02-08 09:50:47,421 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 17.75 2022-02-08 09:50:47,448 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:50:47,450 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:50:47,451 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12738 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:50:57,466 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.75 % 2022-02-08 09:50:57,467 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 17.75 2022-02-08 09:50:57,503 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:50:57,506 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:50:57,506 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:51:07,529 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 17.75 % 2022-02-08 09:51:07,529 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 17.75 2022-02-08 09:51:07,569 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:51:07,572 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:51:07,572 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:51:17,585 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.20 % 2022-02-08 09:51:17,585 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 18.2 2022-02-08 09:51:17,624 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:51:17,627 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:51:17,627 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:51:27,645 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.20 % 2022-02-08 09:51:27,647 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 18.2 2022-02-08 09:51:27,688 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:51:27,693 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:51:27,693 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:51:37,706 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.42 % 2022-02-08 09:51:37,706 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 18.42 2022-02-08 09:51:37,736 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:51:37,739 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:51:37,740 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:51:47,780 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.64 % 2022-02-08 09:51:47,780 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 18.64 2022-02-08 09:51:47,834 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:51:47,836 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:51:47,836 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12762 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:51:57,857 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.88 % 2022-02-08 09:51:57,858 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 18.88 2022-02-08 09:51:57,898 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:51:57,901 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:51:57,901 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:52:07,943 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 18.88 % 2022-02-08 09:52:07,943 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 18.88 2022-02-08 09:52:07,993 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:52:07,997 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:52:07,997 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:52:18,012 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.10 % 2022-02-08 09:52:18,013 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 19.1 2022-02-08 09:52:18,043 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:52:18,046 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:52:18,046 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:52:28,061 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.10 % 2022-02-08 09:52:28,062 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 19.1 2022-02-08 09:52:28,105 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:52:28,109 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:52:28,109 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:52:38,121 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.10 % 2022-02-08 09:52:38,122 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 19.1 2022-02-08 09:52:38,153 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:52:38,155 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:52:38,155 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:52:48,171 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.54 % 2022-02-08 09:52:48,171 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 19.54 2022-02-08 09:52:48,203 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:52:48,206 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:52:48,207 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12784 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:52:58,220 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.54 % 2022-02-08 09:52:58,220 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 19.54 2022-02-08 09:52:58,253 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:52:58,255 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:52:58,256 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:53:08,299 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 19.77 % 2022-02-08 09:53:08,301 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 19.77 2022-02-08 09:53:08,345 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:53:08,346 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:53:08,346 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:53:18,359 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.00 % 2022-02-08 09:53:18,359 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 20.0 2022-02-08 09:53:18,401 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:53:18,404 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:53:18,404 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:53:28,421 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.00 % 2022-02-08 09:53:28,421 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 20.0 2022-02-08 09:53:28,469 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:53:28,473 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:53:28,473 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:53:38,490 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.34 % 2022-02-08 09:53:38,490 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 20.34 2022-02-08 09:53:38,526 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:53:38,529 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:53:38,529 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:53:48,549 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.57 % 2022-02-08 09:53:48,551 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 20.57 2022-02-08 09:53:48,586 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:53:48,589 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:53:48,591 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12808 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:53:58,611 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.57 % 2022-02-08 09:53:58,611 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 20.57 2022-02-08 09:53:58,655 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:53:58,658 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:53:58,658 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:54:08,674 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.57 % 2022-02-08 09:54:08,674 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 20.57 2022-02-08 09:54:08,717 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:54:08,720 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:54:08,720 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:54:18,736 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.79 % 2022-02-08 09:54:18,737 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 20.79 2022-02-08 09:54:18,786 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:54:18,790 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:54:18,790 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:54:28,808 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 20.79 % 2022-02-08 09:54:28,808 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 20.79 2022-02-08 09:54:28,852 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:54:28,855 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:54:28,855 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:54:38,868 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.01 % 2022-02-08 09:54:38,868 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 21.01 2022-02-08 09:54:38,905 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:54:38,908 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:54:38,908 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:54:48,944 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.23 % 2022-02-08 09:54:48,944 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 21.23 2022-02-08 09:54:48,993 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:54:48,997 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:54:48,997 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12833 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:54:59,009 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.45 % 2022-02-08 09:54:59,009 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 21.45 2022-02-08 09:54:59,042 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:54:59,046 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:54:59,046 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:55:09,065 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.45 % 2022-02-08 09:55:09,065 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 21.45 2022-02-08 09:55:09,122 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:55:09,125 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:55:09,125 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:55:19,145 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.68 % 2022-02-08 09:55:19,147 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 21.68 2022-02-08 09:55:19,193 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:55:19,196 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:55:19,197 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:55:29,210 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.68 % 2022-02-08 09:55:29,210 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 21.68 2022-02-08 09:55:29,247 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:55:29,252 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:55:29,252 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:55:39,275 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 21.90 % 2022-02-08 09:55:39,275 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 21.9 2022-02-08 09:55:39,316 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:55:39,319 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:55:39,321 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:55:49,332 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.24 % 2022-02-08 09:55:49,332 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 22.24 2022-02-08 09:55:49,364 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:55:49,367 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:55:49,368 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12857 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:55:59,378 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.24 % 2022-02-08 09:55:59,378 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 22.24 2022-02-08 09:55:59,413 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:55:59,415 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:55:59,415 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:56:09,430 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.47 % 2022-02-08 09:56:09,430 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 22.47 2022-02-08 09:56:09,473 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:56:09,476 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:56:09,476 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:56:19,490 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.69 % 2022-02-08 09:56:19,490 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 22.69 2022-02-08 09:56:19,520 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:56:19,523 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:56:19,523 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:56:29,536 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.69 % 2022-02-08 09:56:29,536 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 22.69 2022-02-08 09:56:29,571 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:56:29,573 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:56:29,575 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:56:39,654 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.69 % 2022-02-08 09:56:39,654 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 22.69 2022-02-08 09:56:39,690 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:56:39,694 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:56:39,694 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:56:49,721 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 22.91 % 2022-02-08 09:56:49,721 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 22.91 5 queries submitted, 0 failed, 12881 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:56:49,792 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:56:49,795 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:56:49,795 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:56:59,808 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.13 % 2022-02-08 09:56:59,808 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 23.13 2022-02-08 09:56:59,848 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:56:59,851 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:56:59,851 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:57:09,865 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.35 % 2022-02-08 09:57:09,865 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 23.35 2022-02-08 09:57:09,898 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:57:09,901 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:57:09,903 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:57:19,917 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.35 % 2022-02-08 09:57:19,917 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 23.35 2022-02-08 09:57:19,957 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:57:19,960 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:57:19,960 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:57:29,997 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.69 % 2022-02-08 09:57:29,997 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 23.69 2022-02-08 09:57:30,035 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:57:30,038 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:57:30,038 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:57:40,055 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.92 % 2022-02-08 09:57:40,056 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 23.92 2022-02-08 09:57:40,096 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:57:40,099 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:57:40,099 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12902 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:57:50,114 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 23.92 % 2022-02-08 09:57:50,115 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 23.92 2022-02-08 09:57:50,148 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:57:50,151 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:57:50,151 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:58:00,165 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 24.15 % 2022-02-08 09:58:00,165 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 24.15 2022-02-08 09:58:00,198 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:58:00,201 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:58:00,203 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:58:10,219 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 24.15 % 2022-02-08 09:58:10,220 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 24.15 2022-02-08 09:58:10,267 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:58:10,272 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:58:10,272 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:58:20,292 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 24.15 % 2022-02-08 09:58:20,292 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 24.15 2022-02-08 09:58:20,368 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:58:20,371 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:58:20,371 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:58:30,430 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 24.49 % 2022-02-08 09:58:30,430 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 24.49 2022-02-08 09:58:30,467 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:58:30,473 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:58:30,473 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:58:40,506 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.05 % 2022-02-08 09:58:40,507 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 25.05 2022-02-08 09:58:40,548 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:58:40,552 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:58:40,552 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12930 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:58:50,565 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.05 % 2022-02-08 09:58:50,565 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 25.05 2022-02-08 09:58:50,594 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:58:50,596 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:58:50,596 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:59:00,611 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.05 % 2022-02-08 09:59:00,612 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 25.05 2022-02-08 09:59:00,650 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:59:00,653 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:59:00,653 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:59:10,665 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.39 % 2022-02-08 09:59:10,667 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 25.39 2022-02-08 09:59:10,697 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:59:10,700 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:59:10,700 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:59:20,716 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.39 % 2022-02-08 09:59:20,717 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 25.39 2022-02-08 09:59:20,799 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:59:20,802 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:59:20,802 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:59:30,821 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.39 % 2022-02-08 09:59:30,822 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 25.39 2022-02-08 09:59:30,864 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:59:30,868 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:59:30,869 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 09:59:40,881 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.62 % 2022-02-08 09:59:40,881 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 25.62 2022-02-08 09:59:40,941 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:59:40,944 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:59:40,946 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12954 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 09:59:50,963 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.62 % 2022-02-08 09:59:50,963 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 25.62 2022-02-08 09:59:51,003 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 09:59:51,006 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 09:59:51,006 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:00:01,028 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.96 % 2022-02-08 10:00:01,029 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 25.96 2022-02-08 10:00:01,065 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:00:01,069 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:00:01,069 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:00:11,092 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 25.96 % 2022-02-08 10:00:11,092 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 25.96 2022-02-08 10:00:11,154 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:00:11,157 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:00:11,157 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:00:21,168 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.19 % 2022-02-08 10:00:21,168 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 26.19 2022-02-08 10:00:21,200 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:00:21,203 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:00:21,203 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:00:31,216 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.19 % 2022-02-08 10:00:31,216 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 26.19 2022-02-08 10:00:31,250 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:00:31,253 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:00:31,253 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:00:41,266 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.53 % 2022-02-08 10:00:41,266 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 26.53 2022-02-08 10:00:41,295 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:00:41,298 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:00:41,298 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12978 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:00:51,331 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.53 % 2022-02-08 10:00:51,332 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 26.53 2022-02-08 10:00:51,367 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:00:51,368 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:00:51,368 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:01:01,384 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.76 % 2022-02-08 10:01:01,384 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 26.76 2022-02-08 10:01:01,417 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:01:01,421 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:01:01,421 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:01:11,434 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.76 % 2022-02-08 10:01:11,434 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 26.76 2022-02-08 10:01:11,471 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:01:11,474 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:01:11,474 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:01:21,505 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.76 % 2022-02-08 10:01:21,506 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 26.76 2022-02-08 10:01:21,563 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:01:21,568 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:01:21,568 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:01:31,585 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.76 % 2022-02-08 10:01:31,585 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 26.76 2022-02-08 10:01:31,618 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:01:31,624 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:01:31,624 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:01:41,644 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.98 % 2022-02-08 10:01:41,644 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 26.98 2022-02-08 10:01:41,684 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:01:41,687 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:01:41,687 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 12999 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:01:51,710 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 26.98 % 2022-02-08 10:01:51,710 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 26.98 2022-02-08 10:01:51,753 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:01:51,755 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:01:51,756 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:02:01,773 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.32 % 2022-02-08 10:02:01,775 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 27.32 2022-02-08 10:02:01,812 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:02:01,815 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:02:01,816 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:02:11,845 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.32 % 2022-02-08 10:02:11,845 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 27.32 2022-02-08 10:02:11,884 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:02:11,888 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:02:11,888 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:02:21,910 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.32 % 2022-02-08 10:02:21,911 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 27.32 2022-02-08 10:02:21,946 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:02:21,948 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:02:21,948 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:02:31,960 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.32 % 2022-02-08 10:02:31,960 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 27.32 2022-02-08 10:02:31,990 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:02:31,992 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:02:31,992 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:02:42,006 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.32 % 2022-02-08 10:02:42,006 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 27.32 2022-02-08 10:02:42,045 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:02:42,046 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:02:42,046 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13023 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:02:52,061 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.32 % 2022-02-08 10:02:52,061 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 27.32 2022-02-08 10:02:52,098 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:02:52,101 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:02:52,101 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:03:02,131 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.55 % 2022-02-08 10:03:02,131 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 27.55 2022-02-08 10:03:02,174 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:03:02,177 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:03:02,177 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:03:12,194 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.89 % 2022-02-08 10:03:12,194 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 27.89 2022-02-08 10:03:12,226 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:03:12,233 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:03:12,233 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:03:22,256 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.11 % 2022-02-08 10:03:22,256 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 28.11 2022-02-08 10:03:22,302 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:03:22,305 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:03:22,305 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:03:32,322 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.11 % 2022-02-08 10:03:32,322 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 28.11 2022-02-08 10:03:32,365 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:03:32,368 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:03:32,368 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:03:42,385 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.11 % 2022-02-08 10:03:42,387 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 28.11 2022-02-08 10:03:42,433 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:03:42,436 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:03:42,436 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13045 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:03:52,479 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.11 % 2022-02-08 10:03:52,479 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 28.11 2022-02-08 10:03:52,538 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:03:52,540 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:03:52,540 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:04:02,556 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.45 % 2022-02-08 10:04:02,558 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 28.45 2022-02-08 10:04:02,595 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:04:02,598 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:04:02,598 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:04:12,632 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.45 % 2022-02-08 10:04:12,632 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 28.45 2022-02-08 10:04:12,678 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:04:12,683 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:04:12,683 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:04:22,698 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.45 % 2022-02-08 10:04:22,698 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 28.45 2022-02-08 10:04:22,746 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:04:22,750 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:04:22,750 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:04:32,770 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.79 % 2022-02-08 10:04:32,770 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 28.79 2022-02-08 10:04:32,802 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:04:32,805 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:04:32,805 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:04:42,816 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 28.79 % 2022-02-08 10:04:42,818 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 28.79 2022-02-08 10:04:42,845 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:04:42,848 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:04:42,848 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13068 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:04:52,878 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.13 % 2022-02-08 10:04:52,878 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 29.13 2022-02-08 10:04:52,930 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:04:52,934 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:04:52,936 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:05:02,950 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.47 % 2022-02-08 10:05:02,951 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 29.47 2022-02-08 10:05:02,982 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:05:02,984 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:05:02,984 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:05:13,000 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.47 % 2022-02-08 10:05:13,000 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 29.47 2022-02-08 10:05:13,046 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:05:13,049 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:05:13,049 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:05:23,081 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.71 % 2022-02-08 10:05:23,082 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 29.71 2022-02-08 10:05:23,130 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:05:23,131 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:05:23,131 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:05:33,148 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 29.71 % 2022-02-08 10:05:33,148 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 29.71 2022-02-08 10:05:33,197 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:05:33,200 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:05:33,200 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:05:43,226 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.27 % 2022-02-08 10:05:43,226 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 30.27 2022-02-08 10:05:43,273 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:05:43,276 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:05:43,276 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13092 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:05:53,289 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.61 % 2022-02-08 10:05:53,289 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 30.61 2022-02-08 10:05:53,329 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:05:53,332 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:05:53,332 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:06:03,355 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.61 % 2022-02-08 10:06:03,355 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 30.61 2022-02-08 10:06:03,388 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:06:03,392 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:06:03,392 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:06:13,404 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.61 % 2022-02-08 10:06:13,405 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 30.61 2022-02-08 10:06:13,444 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:06:13,447 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:06:13,447 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:06:23,474 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.61 % 2022-02-08 10:06:23,474 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 30.61 2022-02-08 10:06:23,528 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:06:23,532 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:06:23,532 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:06:33,551 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.61 % 2022-02-08 10:06:33,551 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 30.61 2022-02-08 10:06:33,592 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:06:33,595 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:06:33,595 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:06:43,611 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.61 % 2022-02-08 10:06:43,611 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 30.61 2022-02-08 10:06:43,647 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:06:43,650 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:06:43,650 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13117 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:06:53,661 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 30.95 % 2022-02-08 10:06:53,661 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 30.95 2022-02-08 10:06:53,704 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:06:53,707 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:06:53,707 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:07:03,720 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 31.29 % 2022-02-08 10:07:03,721 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 31.29 2022-02-08 10:07:03,757 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:07:03,760 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:07:03,760 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:07:13,775 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 31.29 % 2022-02-08 10:07:13,775 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 31.29 2022-02-08 10:07:13,819 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:07:13,822 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:07:13,822 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:07:23,836 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 31.52 % 2022-02-08 10:07:23,836 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 31.52 2022-02-08 10:07:23,868 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:07:23,871 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:07:23,871 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:07:33,884 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 31.52 % 2022-02-08 10:07:33,885 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 31.52 2022-02-08 10:07:33,917 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:07:33,920 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:07:33,920 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:07:43,931 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 31.52 % 2022-02-08 10:07:43,931 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 31.52 2022-02-08 10:07:43,960 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:07:43,963 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:07:43,963 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13140 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:07:53,976 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.20 % 2022-02-08 10:07:53,976 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 32.2 2022-02-08 10:07:54,017 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:07:54,020 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:07:54,020 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:08:04,038 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.20 % 2022-02-08 10:08:04,039 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 32.2 2022-02-08 10:08:04,073 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:08:04,076 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:08:04,076 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:08:14,095 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.20 % 2022-02-08 10:08:14,095 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 32.2 2022-02-08 10:08:14,127 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:08:14,131 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:08:14,131 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:08:24,148 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.42 % 2022-02-08 10:08:24,148 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 32.42 2022-02-08 10:08:24,176 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:08:24,178 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:08:24,178 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:08:34,193 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 32.42 % 2022-02-08 10:08:34,194 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 32.42 2022-02-08 10:08:34,233 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:08:34,236 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:08:34,236 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:08:44,253 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.34 % 2022-02-08 10:08:44,253 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 33.34 2022-02-08 10:08:44,283 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:08:44,286 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:08:44,286 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13168 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:08:54,299 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.34 % 2022-02-08 10:08:54,301 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 33.34 2022-02-08 10:08:54,332 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:08:54,335 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:08:54,335 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:09:04,346 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.34 % 2022-02-08 10:09:04,348 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 33.34 2022-02-08 10:09:04,375 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:09:04,378 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:09:04,378 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:09:14,392 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.34 % 2022-02-08 10:09:14,394 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 33.34 2022-02-08 10:09:14,424 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:09:14,427 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:09:14,427 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:09:24,440 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.68 % 2022-02-08 10:09:24,440 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 33.68 2022-02-08 10:09:24,471 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:09:24,473 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:09:24,474 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:09:34,493 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.68 % 2022-02-08 10:09:34,493 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 33.68 2022-02-08 10:09:34,539 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:09:34,543 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:09:34,543 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:09:44,565 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.68 % 2022-02-08 10:09:44,565 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 33.68 2022-02-08 10:09:44,601 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:09:44,604 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:09:44,604 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13195 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:09:54,615 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 33.91 % 2022-02-08 10:09:54,615 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 33.91 2022-02-08 10:09:54,647 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:09:54,650 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:09:54,650 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:10:04,661 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.25 % 2022-02-08 10:10:04,663 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 34.25 2022-02-08 10:10:04,696 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:10:04,698 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:10:04,700 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:10:14,713 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.59 % 2022-02-08 10:10:14,713 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 34.59 2022-02-08 10:10:14,744 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:10:14,747 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:10:14,747 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:10:24,763 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.59 % 2022-02-08 10:10:24,763 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 34.59 2022-02-08 10:10:24,799 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:10:24,802 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:10:24,803 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:10:34,815 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.59 % 2022-02-08 10:10:34,816 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 34.59 2022-02-08 10:10:34,852 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:10:34,855 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:10:34,855 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:10:44,869 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 34.59 % 2022-02-08 10:10:44,869 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 34.59 2022-02-08 10:10:44,898 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:10:44,901 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:10:44,901 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13216 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:10:54,917 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.15 % 2022-02-08 10:10:54,918 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 35.15 2022-02-08 10:10:54,960 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:10:54,963 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:10:54,963 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:11:04,980 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.15 % 2022-02-08 10:11:04,980 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 35.15 2022-02-08 10:11:05,016 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:11:05,019 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:11:05,019 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:11:15,035 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.72 % 2022-02-08 10:11:15,035 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 35.72 2022-02-08 10:11:15,094 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:11:15,096 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:11:15,096 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:11:25,109 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.72 % 2022-02-08 10:11:25,109 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 35.72 2022-02-08 10:11:25,154 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:11:25,155 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:11:25,157 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:11:35,170 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.72 % 2022-02-08 10:11:35,171 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 35.72 2022-02-08 10:11:35,213 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:11:35,216 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:11:35,217 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:11:45,233 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.72 % 2022-02-08 10:11:45,233 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 35.72 2022-02-08 10:11:45,263 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:11:45,266 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:11:45,266 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13240 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:11:55,276 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 35.72 % 2022-02-08 10:11:55,276 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 35.72 2022-02-08 10:11:55,319 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:11:55,322 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:11:55,322 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:12:05,359 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.06 % 2022-02-08 10:12:05,361 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 36.06 2022-02-08 10:12:05,397 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:12:05,400 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:12:05,400 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:12:15,417 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.63 % 2022-02-08 10:12:15,417 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 36.63 2022-02-08 10:12:15,448 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:12:15,451 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:12:15,451 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:12:25,466 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.86 % 2022-02-08 10:12:25,466 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 36.86 2022-02-08 10:12:25,506 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:12:25,510 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:12:25,510 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:12:35,522 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 36.86 % 2022-02-08 10:12:35,523 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 36.86 2022-02-08 10:12:35,553 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:12:35,555 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:12:35,556 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:12:45,586 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.20 % 2022-02-08 10:12:45,588 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 37.2 2022-02-08 10:12:45,621 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:12:45,625 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:12:45,625 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13267 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:12:55,638 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.54 % 2022-02-08 10:12:55,638 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 37.54 2022-02-08 10:12:55,694 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:12:55,697 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:12:55,697 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:13:05,713 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.54 % 2022-02-08 10:13:05,713 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 37.54 2022-02-08 10:13:05,760 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:13:05,763 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:13:05,763 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:13:15,778 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.54 % 2022-02-08 10:13:15,778 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 37.54 2022-02-08 10:13:15,815 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:13:15,816 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:13:15,818 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:13:25,831 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.77 % 2022-02-08 10:13:25,831 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 37.77 2022-02-08 10:13:25,861 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:13:25,864 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:13:25,864 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:13:35,875 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.99 % 2022-02-08 10:13:35,875 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 37.99 2022-02-08 10:13:35,905 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:13:35,907 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:13:35,907 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:13:45,921 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 37.99 % 2022-02-08 10:13:45,923 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 37.99 2022-02-08 10:13:45,951 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:13:45,953 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:13:45,953 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13293 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:13:55,989 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.33 % 2022-02-08 10:13:55,989 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 38.33 2022-02-08 10:13:56,043 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:13:56,048 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:13:56,048 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:14:06,062 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 38.67 % 2022-02-08 10:14:06,062 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 38.67 2022-02-08 10:14:06,092 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:14:06,095 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:14:06,095 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:14:16,109 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.01 % 2022-02-08 10:14:16,109 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 39.01 2022-02-08 10:14:16,148 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:14:16,151 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:14:16,151 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:14:26,163 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.23 % 2022-02-08 10:14:26,164 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 39.23 2022-02-08 10:14:26,207 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:14:26,211 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:14:26,211 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:14:36,223 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.23 % 2022-02-08 10:14:36,223 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 39.23 2022-02-08 10:14:36,252 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:14:36,255 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:14:36,255 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:14:46,282 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.23 % 2022-02-08 10:14:46,282 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 39.23 2022-02-08 10:14:46,311 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:14:46,312 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:14:46,312 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13317 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:14:56,328 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.57 % 2022-02-08 10:14:56,329 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 39.57 2022-02-08 10:14:56,377 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:14:56,381 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:14:56,381 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:15:06,395 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 39.57 % 2022-02-08 10:15:06,395 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 39.57 2022-02-08 10:15:06,446 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:15:06,448 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:15:06,448 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:15:16,471 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.14 % 2022-02-08 10:15:16,471 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 40.14 2022-02-08 10:15:16,503 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:15:16,506 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:15:16,506 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:15:26,522 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.14 % 2022-02-08 10:15:26,522 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 40.14 2022-02-08 10:15:26,559 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:15:26,562 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:15:26,562 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:15:36,579 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.14 % 2022-02-08 10:15:36,579 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 40.14 2022-02-08 10:15:36,617 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:15:36,619 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:15:36,619 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:15:46,631 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.48 % 2022-02-08 10:15:46,631 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 40.48 2022-02-08 10:15:46,661 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:15:46,664 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:15:46,664 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13336 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:15:56,684 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.48 % 2022-02-08 10:15:56,684 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 40.48 2022-02-08 10:15:56,732 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:15:56,734 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:15:56,736 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:16:06,753 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.48 % 2022-02-08 10:16:06,753 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 40.48 2022-02-08 10:16:06,801 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:16:06,803 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:16:06,803 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:16:16,815 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 40.82 % 2022-02-08 10:16:16,816 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 40.82 2022-02-08 10:16:16,854 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:16:16,857 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:16:16,857 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:16:26,875 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 41.16 % 2022-02-08 10:16:26,877 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 41.16 2022-02-08 10:16:26,926 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:16:26,930 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:16:26,930 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:16:36,947 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 41.16 % 2022-02-08 10:16:36,947 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 41.16 2022-02-08 10:16:36,983 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:16:36,986 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:16:36,986 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:16:47,003 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 41.50 % 2022-02-08 10:16:47,003 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 41.5 2022-02-08 10:16:47,046 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:16:47,049 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:16:47,049 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13360 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:16:57,062 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 41.50 % 2022-02-08 10:16:57,062 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 41.5 2022-02-08 10:16:57,092 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:16:57,094 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:16:57,095 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:17:07,111 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 41.84 % 2022-02-08 10:17:07,112 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 41.84 2022-02-08 10:17:07,157 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:17:07,160 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:17:07,160 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:17:17,173 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.18 % 2022-02-08 10:17:17,174 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 42.18 2022-02-08 10:17:17,204 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:17:17,207 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:17:17,207 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:17:27,221 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.18 % 2022-02-08 10:17:27,221 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 42.18 2022-02-08 10:17:27,252 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:17:27,253 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:17:27,253 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:17:37,267 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.18 % 2022-02-08 10:17:37,269 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 42.18 2022-02-08 10:17:37,296 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:17:37,299 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:17:37,299 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:17:47,311 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.18 % 2022-02-08 10:17:47,312 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 42.18 2022-02-08 10:17:47,342 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:17:47,345 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:17:47,345 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13383 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:17:57,361 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 42.86 % 2022-02-08 10:17:57,361 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 42.86 2022-02-08 10:17:57,405 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:17:57,408 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:17:57,410 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:18:07,421 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 43.20 % 2022-02-08 10:18:07,421 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 43.2 2022-02-08 10:18:07,447 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:18:07,450 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:18:07,450 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:18:17,463 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 43.20 % 2022-02-08 10:18:17,463 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 43.2 2022-02-08 10:18:17,496 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:18:17,500 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:18:17,500 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:18:27,519 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 43.54 % 2022-02-08 10:18:27,519 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 43.54 2022-02-08 10:18:27,549 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:18:27,552 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:18:27,552 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:18:37,569 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.22 % 2022-02-08 10:18:37,571 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 44.22 2022-02-08 10:18:37,615 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:18:37,618 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:18:37,618 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:18:47,632 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.22 % 2022-02-08 10:18:47,634 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 44.22 2022-02-08 10:18:47,690 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:18:47,694 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:18:47,694 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13403 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:18:57,746 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.22 % 2022-02-08 10:18:57,746 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 44.22 2022-02-08 10:18:57,782 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:18:57,785 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:18:57,785 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:19:07,819 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.22 % 2022-02-08 10:19:07,819 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 44.22 2022-02-08 10:19:07,885 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:19:07,890 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:19:07,890 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:19:17,908 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.56 % 2022-02-08 10:19:17,910 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 44.56 2022-02-08 10:19:17,956 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:19:17,959 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:19:17,959 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:19:27,986 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 44.88 % 2022-02-08 10:19:27,986 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 44.88 2022-02-08 10:19:28,016 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:19:28,019 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:19:28,019 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:19:38,033 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.22 % 2022-02-08 10:19:38,035 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 45.22 2022-02-08 10:19:38,092 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:19:38,095 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:19:38,095 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:19:48,107 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.22 % 2022-02-08 10:19:48,108 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 45.22 2022-02-08 10:19:48,142 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:19:48,147 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:19:48,147 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13428 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:19:58,160 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.53 % 2022-02-08 10:19:58,160 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 45.53 2022-02-08 10:19:58,190 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:19:58,193 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:19:58,194 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:20:08,206 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 45.53 % 2022-02-08 10:20:08,207 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 45.53 2022-02-08 10:20:08,240 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:20:08,243 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:20:08,243 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:20:18,256 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 46.53 % 2022-02-08 10:20:18,256 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 46.53 2022-02-08 10:20:18,286 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:20:18,289 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:20:18,289 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:20:28,303 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 46.53 % 2022-02-08 10:20:28,305 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 46.53 2022-02-08 10:20:28,344 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:20:28,346 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:20:28,346 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:20:38,359 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 46.85 % 2022-02-08 10:20:38,359 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 46.85 2022-02-08 10:20:38,391 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:20:38,394 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:20:38,394 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:20:48,430 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 46.85 % 2022-02-08 10:20:48,430 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 46.85 2022-02-08 10:20:48,470 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:20:48,473 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:20:48,473 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13450 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:20:58,486 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.17 % 2022-02-08 10:20:58,486 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 47.17 2022-02-08 10:20:58,516 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:20:58,519 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:20:58,519 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:21:08,532 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.17 % 2022-02-08 10:21:08,532 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 47.17 2022-02-08 10:21:08,562 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:21:08,565 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:21:08,565 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:21:18,578 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.51 % 2022-02-08 10:21:18,578 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 47.51 2022-02-08 10:21:18,608 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:21:18,611 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:21:18,611 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:21:28,654 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 47.82 % 2022-02-08 10:21:28,654 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 47.82 2022-02-08 10:21:28,683 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:21:28,686 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:21:28,686 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:21:38,700 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 48.16 % 2022-02-08 10:21:38,700 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 48.16 2022-02-08 10:21:38,742 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:21:38,744 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:21:38,744 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:21:48,757 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 48.16 % 2022-02-08 10:21:48,759 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 48.16 2022-02-08 10:21:48,805 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:21:48,811 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:21:48,811 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13477 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:21:58,825 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 48.48 % 2022-02-08 10:21:58,825 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 48.48 2022-02-08 10:21:58,862 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:21:58,865 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:21:58,865 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:22:08,881 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 48.48 % 2022-02-08 10:22:08,881 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 48.48 2022-02-08 10:22:08,927 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:22:08,930 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:22:08,930 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:22:18,963 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 48.48 % 2022-02-08 10:22:18,963 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 48.48 2022-02-08 10:22:18,990 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:22:18,993 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:22:18,993 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:22:29,006 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 48.82 % 2022-02-08 10:22:29,006 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 48.82 2022-02-08 10:22:29,040 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:22:29,043 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:22:29,043 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:22:39,056 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.16 % 2022-02-08 10:22:39,058 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 49.16 2022-02-08 10:22:39,086 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:22:39,088 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:22:39,088 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:22:49,101 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.50 % 2022-02-08 10:22:49,101 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 49.5 2022-02-08 10:22:49,131 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:22:49,134 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:22:49,134 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13500 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:22:59,148 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.50 % 2022-02-08 10:22:59,148 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 49.5 2022-02-08 10:22:59,188 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:22:59,193 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:22:59,193 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:23:09,207 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 49.82 % 2022-02-08 10:23:09,209 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 49.82 2022-02-08 10:23:09,246 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:23:09,250 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:23:09,250 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:23:19,262 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 50.16 % 2022-02-08 10:23:19,262 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 50.16 2022-02-08 10:23:19,293 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:23:19,296 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:23:19,296 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:23:29,309 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 50.16 % 2022-02-08 10:23:29,309 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 50.16 2022-02-08 10:23:29,341 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:23:29,346 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:23:29,348 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:23:39,362 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 50.16 % 2022-02-08 10:23:39,362 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 50.16 2022-02-08 10:23:39,395 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:23:39,398 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:23:39,398 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:23:49,417 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 50.84 % 2022-02-08 10:23:49,417 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 50.84 2022-02-08 10:23:49,460 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:23:49,463 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:23:49,463 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13523 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:23:59,474 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.16 % 2022-02-08 10:23:59,476 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 51.16 2022-02-08 10:23:59,523 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:23:59,526 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:23:59,526 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:24:09,538 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.16 % 2022-02-08 10:24:09,538 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 51.16 2022-02-08 10:24:09,572 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:24:09,575 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:24:09,575 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:24:19,592 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.16 % 2022-02-08 10:24:19,592 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 51.16 2022-02-08 10:24:19,628 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:24:19,630 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:24:19,631 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:24:29,648 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.16 % 2022-02-08 10:24:29,648 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 51.16 2022-02-08 10:24:29,687 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:24:29,691 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:24:29,691 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:24:39,707 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.16 % 2022-02-08 10:24:39,707 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 51.16 2022-02-08 10:24:39,753 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:24:39,756 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:24:39,756 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13549 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:24:49,782 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.50 % 2022-02-08 10:24:49,782 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 51.5 2022-02-08 10:24:49,816 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:24:49,821 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:24:49,821 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:24:59,836 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 51.81 % 2022-02-08 10:24:59,836 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 51.81 2022-02-08 10:24:59,871 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:24:59,874 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:24:59,874 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:25:09,898 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.15 % 2022-02-08 10:25:09,898 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 52.15 2022-02-08 10:25:09,950 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:25:09,953 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:25:09,953 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:25:19,970 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.49 % 2022-02-08 10:25:19,970 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 52.49 2022-02-08 10:25:20,009 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:25:20,010 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:25:20,012 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:25:30,023 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.49 % 2022-02-08 10:25:30,023 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 52.49 2022-02-08 10:25:30,052 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:25:30,055 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:25:30,055 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:25:40,066 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.49 % 2022-02-08 10:25:40,068 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 52.49 2022-02-08 10:25:40,117 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:25:40,119 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:25:40,119 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13570 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:25:50,131 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 52.49 % 2022-02-08 10:25:50,131 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 52.49 2022-02-08 10:25:50,161 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:25:50,164 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:25:50,164 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:26:00,180 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 53.15 % 2022-02-08 10:26:00,180 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 53.15 2022-02-08 10:26:00,223 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:26:00,227 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:26:00,227 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:26:10,244 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 53.49 % 2022-02-08 10:26:10,244 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 53.49 2022-02-08 10:26:10,292 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:26:10,296 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:26:10,296 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:26:20,315 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 53.49 % 2022-02-08 10:26:20,315 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 53.49 2022-02-08 10:26:20,351 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:26:20,355 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:26:20,355 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:26:30,369 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 53.49 % 2022-02-08 10:26:30,371 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 53.49 2022-02-08 10:26:30,401 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:26:30,404 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:26:30,404 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:26:40,423 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 54.15 % 2022-02-08 10:26:40,423 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 54.15 2022-02-08 10:26:40,464 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:26:40,467 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:26:40,467 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13595 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:26:50,480 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 54.15 % 2022-02-08 10:26:50,480 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 54.15 2022-02-08 10:26:50,513 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:26:50,516 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:26:50,516 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:27:00,535 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 54.15 % 2022-02-08 10:27:00,535 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 54.15 2022-02-08 10:27:00,568 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:27:00,571 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:27:00,571 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:27:10,614 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 54.47 % 2022-02-08 10:27:10,615 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 54.47 2022-02-08 10:27:10,661 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:27:10,667 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:27:10,667 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:27:20,707 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 55.15 % 2022-02-08 10:27:20,709 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 55.15 2022-02-08 10:27:20,743 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:27:20,746 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:27:20,747 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:27:30,765 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 55.15 % 2022-02-08 10:27:30,766 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 55.15 2022-02-08 10:27:30,795 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:27:30,796 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:27:30,798 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:27:40,815 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 55.15 % 2022-02-08 10:27:40,815 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 55.15 2022-02-08 10:27:40,845 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:27:40,848 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:27:40,848 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13615 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:27:50,858 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 55.15 % 2022-02-08 10:27:50,859 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 55.15 2022-02-08 10:27:50,890 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:27:50,892 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:27:50,892 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:28:00,905 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.15 % 2022-02-08 10:28:00,907 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 56.15 2022-02-08 10:28:00,946 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:28:00,948 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:28:00,948 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:28:10,960 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.15 % 2022-02-08 10:28:10,960 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 56.15 2022-02-08 10:28:10,994 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:28:10,996 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:28:10,996 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:28:21,040 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.15 % 2022-02-08 10:28:21,040 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 56.15 2022-02-08 10:28:21,085 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:28:21,088 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:28:21,088 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:28:31,101 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.15 % 2022-02-08 10:28:31,102 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 56.15 2022-02-08 10:28:31,140 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:28:31,142 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:28:31,142 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:28:41,157 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.46 % 2022-02-08 10:28:41,157 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 56.46 2022-02-08 10:28:41,197 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:28:41,200 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:28:41,200 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13635 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:28:51,217 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.46 % 2022-02-08 10:28:51,217 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 56.46 2022-02-08 10:28:51,263 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:28:51,267 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:28:51,269 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:29:01,283 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 56.80 % 2022-02-08 10:29:01,285 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 56.8 2022-02-08 10:29:01,312 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:29:01,315 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:29:01,316 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:29:11,352 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.14 % 2022-02-08 10:29:11,354 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 57.14 2022-02-08 10:29:11,395 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:29:11,398 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:29:11,398 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:29:21,411 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.46 % 2022-02-08 10:29:21,411 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 57.46 2022-02-08 10:29:21,447 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:29:21,450 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:29:21,450 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:29:31,473 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.46 % 2022-02-08 10:29:31,473 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 57.46 2022-02-08 10:29:31,512 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:29:31,515 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:29:31,516 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:29:41,532 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.46 % 2022-02-08 10:29:41,532 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 57.46 2022-02-08 10:29:41,576 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:29:41,579 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:29:41,581 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13661 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:29:51,598 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.46 % 2022-02-08 10:29:51,598 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 57.46 2022-02-08 10:29:51,628 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:29:51,631 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:29:51,631 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:30:01,644 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.46 % 2022-02-08 10:30:01,644 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 57.46 2022-02-08 10:30:01,684 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:30:01,687 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:30:01,687 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:30:11,698 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.80 % 2022-02-08 10:30:11,698 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 57.8 2022-02-08 10:30:11,730 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:30:11,733 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:30:11,733 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:30:21,773 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 58.14 % 2022-02-08 10:30:21,775 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 58.14 2022-02-08 10:30:21,815 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:30:21,818 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:30:21,819 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:30:31,832 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 58.14 % 2022-02-08 10:30:31,832 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 58.14 2022-02-08 10:30:31,864 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:30:31,865 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:30:31,865 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:30:41,882 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 58.80 % 2022-02-08 10:30:41,882 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 58.8 2022-02-08 10:30:41,921 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:30:41,924 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:30:41,924 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13684 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:30:51,937 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 58.80 % 2022-02-08 10:30:51,937 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 58.8 2022-02-08 10:30:51,976 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:30:51,980 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:30:51,980 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:31:01,993 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 58.80 % 2022-02-08 10:31:01,993 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 58.8 2022-02-08 10:31:02,026 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:31:02,029 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:31:02,029 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:31:12,040 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 58.80 % 2022-02-08 10:31:12,040 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 58.8 2022-02-08 10:31:12,072 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:31:12,075 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:31:12,075 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:31:22,091 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.14 % 2022-02-08 10:31:22,092 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 59.14 2022-02-08 10:31:22,134 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:31:22,138 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:31:22,138 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:31:32,150 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.14 % 2022-02-08 10:31:32,150 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 59.14 2022-02-08 10:31:32,178 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:31:32,180 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:31:32,181 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:31:42,197 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.14 % 2022-02-08 10:31:42,197 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 59.14 2022-02-08 10:31:42,240 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:31:42,243 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:31:42,243 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13709 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:31:52,302 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.14 % 2022-02-08 10:31:52,302 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 59.14 2022-02-08 10:31:52,334 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:31:52,335 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:31:52,335 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:32:02,352 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.14 % 2022-02-08 10:32:02,352 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 59.14 2022-02-08 10:32:02,394 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:32:02,398 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:32:02,398 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:32:12,414 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.48 % 2022-02-08 10:32:12,415 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 59.48 2022-02-08 10:32:12,447 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:32:12,450 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:32:12,450 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:32:22,463 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.48 % 2022-02-08 10:32:22,463 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 59.48 2022-02-08 10:32:22,496 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:32:22,500 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:32:22,500 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:32:32,526 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 59.80 % 2022-02-08 10:32:32,528 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 59.8 2022-02-08 10:32:32,571 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:32:32,575 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:32:32,575 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:32:42,585 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.14 % 2022-02-08 10:32:42,585 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 60.14 2022-02-08 10:32:42,630 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:32:42,632 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:32:42,632 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13734 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:32:52,651 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.45 % 2022-02-08 10:32:52,651 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 60.45 2022-02-08 10:32:52,694 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:32:52,697 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:32:52,698 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:33:02,717 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.45 % 2022-02-08 10:33:02,717 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 60.45 2022-02-08 10:33:02,779 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:33:02,782 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:33:02,782 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:33:12,805 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.45 % 2022-02-08 10:33:12,805 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 60.45 2022-02-08 10:33:12,844 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:33:12,846 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:33:12,846 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:33:22,861 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.45 % 2022-02-08 10:33:22,861 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 60.45 2022-02-08 10:33:22,907 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:33:22,908 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:33:22,910 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:33:32,923 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.79 % 2022-02-08 10:33:32,923 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 60.79 2022-02-08 10:33:32,959 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:33:32,963 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:33:32,963 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:33:42,982 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.79 % 2022-02-08 10:33:42,983 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 60.79 2022-02-08 10:33:43,025 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:33:43,028 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:33:43,029 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13758 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:33:53,059 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.79 % 2022-02-08 10:33:53,059 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 60.79 2022-02-08 10:33:53,099 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:33:53,102 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:33:53,102 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:34:03,118 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.79 % 2022-02-08 10:34:03,118 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 60.79 2022-02-08 10:34:03,177 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:34:03,180 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:34:03,180 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:34:13,204 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.79 % 2022-02-08 10:34:13,204 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 60.79 2022-02-08 10:34:13,247 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:34:13,252 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:34:13,252 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:34:23,265 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.79 % 2022-02-08 10:34:23,265 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 60.79 2022-02-08 10:34:23,309 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:34:23,312 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:34:23,312 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:34:33,325 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.79 % 2022-02-08 10:34:33,325 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 60.79 2022-02-08 10:34:33,368 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:34:33,371 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:34:33,371 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:34:43,385 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.79 % 2022-02-08 10:34:43,385 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 60.79 2022-02-08 10:34:43,423 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:34:43,426 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:34:43,427 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13776 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:34:53,438 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.79 % 2022-02-08 10:34:53,438 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 60.79 2022-02-08 10:34:53,469 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:34:53,471 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:34:53,471 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:35:03,483 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.79 % 2022-02-08 10:35:03,483 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 60.79 2022-02-08 10:35:03,522 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:35:03,526 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:35:03,526 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:35:13,539 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 61.75 % 2022-02-08 10:35:13,539 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 61.75 2022-02-08 10:35:13,566 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:35:13,569 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:35:13,569 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:35:23,588 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.06 % 2022-02-08 10:35:23,588 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 62.06 2022-02-08 10:35:23,630 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:35:23,631 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:35:23,632 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:35:33,644 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.06 % 2022-02-08 10:35:33,644 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 62.06 2022-02-08 10:35:33,674 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:35:33,677 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:35:33,677 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:35:43,693 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.06 % 2022-02-08 10:35:43,694 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 62.06 2022-02-08 10:35:43,753 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:35:43,756 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:35:43,756 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13801 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:35:53,772 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.06 % 2022-02-08 10:35:53,772 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 62.06 2022-02-08 10:35:53,812 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:35:53,815 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:35:53,815 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:36:03,826 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.06 % 2022-02-08 10:36:03,828 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 62.06 2022-02-08 10:36:03,858 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:36:03,861 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:36:03,861 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:36:13,877 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.06 % 2022-02-08 10:36:13,878 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 62.06 2022-02-08 10:36:13,915 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:36:13,920 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:36:13,920 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:36:23,931 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.06 % 2022-02-08 10:36:23,933 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 62.06 2022-02-08 10:36:23,966 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:36:23,969 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:36:23,969 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:36:33,990 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.06 % 2022-02-08 10:36:33,990 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 62.06 2022-02-08 10:36:34,032 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:36:34,035 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:36:34,036 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:36:44,051 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.06 % 2022-02-08 10:36:44,052 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 62.06 2022-02-08 10:36:44,111 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:36:44,114 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:36:44,114 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13824 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:36:54,132 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.06 % 2022-02-08 10:36:54,132 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 62.06 2022-02-08 10:36:54,178 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:36:54,181 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:36:54,181 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:37:04,194 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.06 % 2022-02-08 10:37:04,194 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 62.06 2022-02-08 10:37:04,223 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:37:04,226 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:37:04,226 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:37:14,242 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.06 % 2022-02-08 10:37:14,242 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 62.06 2022-02-08 10:37:14,285 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:37:14,288 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:37:14,289 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:37:24,305 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.06 % 2022-02-08 10:37:24,305 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 62.06 2022-02-08 10:37:24,346 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:37:24,351 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:37:24,351 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:37:34,365 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.38 % 2022-02-08 10:37:34,365 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 62.38 2022-02-08 10:37:34,417 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:37:34,421 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:37:34,421 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:37:44,436 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.38 % 2022-02-08 10:37:44,436 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 62.38 2022-02-08 10:37:44,486 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:37:44,490 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:37:44,490 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13847 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:37:54,507 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 62.70 % 2022-02-08 10:37:54,509 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 62.7 2022-02-08 10:37:54,552 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:37:54,553 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:37:54,555 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:38:04,569 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.02 % 2022-02-08 10:38:04,569 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 63.02 2022-02-08 10:38:04,601 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:38:04,602 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:38:04,604 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:38:14,615 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.02 % 2022-02-08 10:38:14,617 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 63.02 2022-02-08 10:38:14,651 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:38:14,654 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:38:14,654 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:38:24,673 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.02 % 2022-02-08 10:38:24,674 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 63.02 2022-02-08 10:38:24,717 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:38:24,720 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:38:24,721 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:38:34,734 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.02 % 2022-02-08 10:38:34,734 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 63.02 2022-02-08 10:38:34,779 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:38:34,782 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:38:34,782 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:38:44,802 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.02 % 2022-02-08 10:38:44,802 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 63.02 2022-02-08 10:38:44,862 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:38:44,865 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:38:44,865 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 13872 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:38:54,908 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.02 % 2022-02-08 10:38:54,908 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 63.02 2022-02-08 10:38:54,957 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:38:54,961 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:38:54,961 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:39:04,979 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.33 % 2022-02-08 10:39:04,979 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 63.33 2022-02-08 10:39:05,026 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:39:05,029 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:39:05,030 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:39:15,061 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.33 % 2022-02-08 10:39:15,061 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 63.33 2022-02-08 10:39:15,130 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:39:15,132 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:39:15,132 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:39:25,145 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 63.33 % 2022-02-08 10:39:25,145 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 63.33 2022-02-08 10:39:25,177 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:39:25,180 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:39:25,180 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:39:35,196 | test | DEBUG | pool-3-thread-18 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 64.29 % 2022-02-08 10:39:35,196 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: running, progress: 64.29 2022-02-08 10:39:35,236 | infra | DEBUG | pool-3-thread-18 | [task:check:441] Wait before next rebalance progress check 2022-02-08 10:39:35,239 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 10:39:35,240 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 10:39:45,253 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-08 10:39:45,253 | infra | DEBUG | pool-3-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 5 queries submitted, 0 failed, 13897 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:39:50,266 | test | INFO | pool-3-thread-18 | [task:check:414] Rebalance - status: none, progress: 100 2022-02-08 10:39:50,295 | test | INFO | pool-3-thread-18 | [task:check:473] Rebalance completed with progress: 100% in 4100.28299999 sec 2022-02-08 10:39:50,295 | infra | INFO | pool-3-thread-18 | [task:complete_task:106] Thread 'Rebalance_task_IN=[]_OUT=[]_1644341489.84' completed 2022-02-08 10:39:50,355 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3970] Stats: {u'opsPerSec': 18125.4, u'diskFetches': 6, u'dataUsed': 178508093453L, u'itemCount': 167999821, u'quotaPercentUsed': 79.05984521851282, u'storageTotals': {u'hdd': {u'usedByData': 309192556188L, u'used': 415743806785L, u'quotaTotal': 1168799956992L, u'total': 1168799956992L, u'free': 628230711222L}, u'ram': {u'usedByData': 61560901296L, u'used': 108870590464L, u'quotaTotal': 78809923584L, u'total': 110647066624L, u'quotaUsed': 77866205184L, u'quotaUsedPerNode': 8651800576L, u'quotaTotalPerNode': 8756658176L}}, u'diskUsed': 309192556188L, u'vbActiveNumNonResident': 126210774, u'memUsed': 61560901296L} 2022-02-08 10:39:51,086 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+-----------+-----------+-----------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+-----------+-----------+-----------+------------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 167999821 | 72.52 GiB | 57.33 GiB | 287.96 GiB | 24.8744592412 | +-----------------+-----------+-----------------+----------+------------+-----+-----------+-----------+-----------+------------+---------------+ 2022-02-08 10:39:51,088 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+------------------+------------------+-----------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+------------------+------------------+-----------------+---------------+ | 40000000 | 380000000 | 2000000-20000000 | 2000000-20000000 | 1000000-2000000 | 0-0 | +---------------+---------------+------------------+------------------+-----------------+---------------+ 2022-02-08 10:39:51,398 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 24.8744592412 2022-02-08 10:39:51,400 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:39:52,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:39:52,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:39:52,776 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:39:52,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:39:52,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:39:52,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:39:52,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:39:52,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:39:52,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:39:52,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:39:52,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 10:39:53,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 10:39:53,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 10:39:53,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 10:39:53,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 10:39:53,907 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:39:53,911 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:39:54,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:39:55,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:39:55,670 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:39:55,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:39:55,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:39:55,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:39:55,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:39:55,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:39:55,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:39:55,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:39:55,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 10:39:56,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 10:39:56,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 10:39:56,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 10:39:56,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 10:39:56,796 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:39:56,799 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 10:39:56,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:39:57,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 10:39:57,184 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:39:57,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 10:39:57,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 10:39:57,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 10:39:57,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 10:39:57,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 10:39:57,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 10:39:57,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 10:39:57,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 10:39:57,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 10:39:57,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 10:39:57,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 10:39:57,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 10:39:57,421 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 10:39:57,424 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:39:58,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:39:58,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:39:58,729 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:39:58,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:39:58,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:39:58,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:39:58,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:39:58,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:39:58,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:39:58,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:39:58,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 10:39:59,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 10:39:59,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 10:39:59,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 10:39:59,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 10:39:59,898 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:39:59,901 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 10:39:59,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:00,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 10:40:00,298 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:00,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 10:40:00,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 10:40:00,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 10:40:00,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 10:40:00,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 10:40:00,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 10:40:00,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 10:40:00,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 10:40:00,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 10:40:00,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 10:40:00,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 10:40:00,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 10:40:00,529 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 10:40:00,532 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:40:01,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:02,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:40:02,180 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:02,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:40:02,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:40:02,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:40:02,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:40:02,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:40:02,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:40:02,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:40:02,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 10:40:02,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 10:40:02,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 10:40:02,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 10:40:02,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 10:40:03,301 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:40:03,302 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:40:04,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:04,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:40:04,726 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:04,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:40:04,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:40:04,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:40:04,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:40:04,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:40:04,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:40:04,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:40:04,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 10:40:05,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 10:40:05,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 10:40:05,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 10:40:05,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 10:40:05,848 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:40:05,851 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 10:40:05,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:06,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 10:40:06,358 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:06,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 10:40:06,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 10:40:06,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 10:40:06,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 10:40:06,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 10:40:06,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 10:40:06,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 10:40:06,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 10:40:06,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 10:40:06,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 10:40:06,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 10:40:06,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 10:40:06,617 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 10:40:06,619 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 10:40:06,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:06,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 10:40:07,000 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:07,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 10:40:07,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 10:40:07,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 10:40:07,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 10:40:07,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 10:40:07,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 10:40:07,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 10:40:07,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 10:40:07,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 10:40:07,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 10:40:07,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 10:40:07,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 10:40:07,219 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 10:40:07,221 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:40:08,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:08,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:40:08,936 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:08,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:40:08,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:40:08,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:40:08,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:40:09,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:40:09,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:40:09,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:40:09,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 10:40:10,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 10:40:10,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 10:40:10,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 10:40:10,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 10:40:11,194 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:40:11,203 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:40:11,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:12,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:40:12,489 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:12,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:40:12,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:40:12,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:40:12,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:40:12,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:40:12,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:40:12,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:40:12,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 10:40:12,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 10:40:12,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 10:40:12,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 10:40:12,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 10:40:13,243 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:40:13,246 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:40:13,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:14,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:40:14,509 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:14,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:40:14,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:40:14,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:40:14,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:40:14,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:40:14,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:40:14,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:40:14,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 10:40:14,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 10:40:14,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 10:40:15,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 10:40:15,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 10:40:15,767 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:40:15,772 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:40:16,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:17,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:40:17,486 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:17,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:40:17,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:40:17,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:40:17,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:40:17,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:40:17,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:40:17,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:40:17,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 10:40:17,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 10:40:17,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 10:40:17,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 10:40:17,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 10:40:18,451 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:40:18,456 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 10:40:18,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:18,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 10:40:18,812 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:18,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 10:40:18,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 10:40:18,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 10:40:18,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 10:40:18,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 10:40:18,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 10:40:18,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 10:40:18,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 10:40:18,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 10:40:18,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 10:40:18,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 10:40:18,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 10:40:19,009 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 10:40:19,009 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for kvstore is 290759MB 2022-02-08 10:40:19,009 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for wal is 5405MB 2022-02-08 10:40:19,009 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:642] Total Disk usage for keyTree is 19621MB 2022-02-08 10:40:19,009 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:643] Total Disk usage for seqTree is 270865MB 2022-02-08 10:40:19,013 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:40:19,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:19,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:40:19,388 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:19,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:40:19,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:40:19,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:40:19,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:40:19,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:40:19,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:40:19,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:40:19,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 10:40:19,622 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.116 - core(s): 8 2022-02-08 10:40:19,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 10:40:20,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 10:40:20,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 10:40:20,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 10:40:20,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 10:40:20,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 10:40:21,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 10:40:21,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 10:40:21,421 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:40:21,424 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:40:21,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:21,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:40:21,812 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:21,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:40:21,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:40:21,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:40:21,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:40:21,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:40:21,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:40:21,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:40:21,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 10:40:21,971 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.123 - core(s): 8 2022-02-08 10:40:21,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 10:40:22,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 10:40:22,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 10:40:22,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 10:40:23,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 10:40:23,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 10:40:23,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 10:40:23,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 10:40:23,977 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:40:23,980 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:40:24,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:24,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:40:24,585 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:24,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:40:24,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:40:24,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:40:24,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:40:24,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:40:24,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:40:24,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:40:24,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 10:40:24,826 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.124 - core(s): 8 2022-02-08 10:40:24,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 10:40:25,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 10:40:25,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 10:40:25,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 10:40:26,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 10:40:26,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 10:40:26,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 10:40:26,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 10:40:26,658 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:40:26,661 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:40:26,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:27,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:40:27,102 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:27,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:40:27,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:40:27,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:40:27,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:40:27,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:40:27,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:40:27,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:40:27,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 10:40:27,339 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.126 - core(s): 8 2022-02-08 10:40:27,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 10:40:27,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 10:40:28,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 10:40:28,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 10:40:28,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 10:40:28,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 10:40:28,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 10:40:29,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 10:40:29,369 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:40:29,372 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:40:29,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:29,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:40:29,838 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:29,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:40:29,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:40:29,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:40:29,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:40:29,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:40:29,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:40:29,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:40:29,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 10:40:30,015 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.127 - core(s): 8 2022-02-08 10:40:30,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 10:40:30,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 10:40:30,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 10:40:31,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 10:40:31,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 10:40:31,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 10:40:31,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 10:40:31,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 10:40:31,987 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:40:31,990 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:40:32,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:32,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:40:32,552 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:32,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:40:32,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:40:32,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:40:32,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:40:32,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:40:32,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:40:32,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:40:32,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 10:40:32,855 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.129 - core(s): 8 2022-02-08 10:40:32,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 10:40:33,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 10:40:33,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 10:40:33,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 10:40:33,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 10:40:34,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 10:40:34,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 10:40:34,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 10:40:34,655 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:40:34,657 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:40:34,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:34,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:40:35,127 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:35,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:40:35,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:40:35,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:40:35,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:40:35,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:40:35,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:40:35,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:40:35,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 10:40:35,342 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.135 - core(s): 8 2022-02-08 10:40:35,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 10:40:35,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 10:40:36,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 10:40:36,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 10:40:36,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 10:40:36,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 10:40:36,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 10:40:37,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 10:40:37,240 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:40:37,246 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:40:37,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:37,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:40:37,730 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:37,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:40:37,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:40:37,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:40:37,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:40:37,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:40:37,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:40:37,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:40:37,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 10:40:37,938 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.115 - core(s): 8 2022-02-08 10:40:37,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 10:40:38,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 10:40:38,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 10:40:38,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 10:40:39,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 10:40:39,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 10:40:39,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 10:40:39,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 10:40:39,769 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:40:39,772 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:40:39,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:40:40,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:40:40,367 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:40:40,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:40:40,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:40:40,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:40:40,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:40:40,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:40:40,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:40:40,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:40:40,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 10:40:40,627 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.120.170 - core(s): 8 2022-02-08 10:40:40,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 10:40:41,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 10:40:41,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 10:40:41,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 10:40:41,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 10:40:42,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 10:40:42,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 10:40:42,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 10:40:42,812 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:40:42,812 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:723] {'172.23.121.116': {'rw_0:magma': {u'NReadBytesGetPerSec': 601077.3125, u'ReplayWALOffset': u'2177:3731456', u'NTablesCreated': 51311, u'RecentReadIOAmpSet': 0.023992352187633514, u'ReadIOAmp': 2.4744231700897217, u'BlockCacheMisses': 1004706, u'NFlushes': 12019, u'RecentWriteAmp': 2.0555288791656494, u'NumSyncFlushes': 165, u'TotalMemUsed': 93489497, u'NBloomFilterHits': 22998882, u'Fragmentation': 0.44380250573158264, u'NumNormalFlushes': 5878, u'RecentBlockCacheHitRatio': 65.40817260742188, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 25675, u'ReadIOAmp': 99.28571428571429, u'MemtablesMemUsed': 6120888, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 36702507, u'CheckpointOverhead': 3465216, u'NItemsGet': 56, u'NWriteIOs': 32625, u'NReadBytes': 747447044, u'LevelStats': [{u'TargetSize': 130023424, u'LogicalSize': 20184873, u'SizeRatio': 0.15524028193566106, u'NWriteBytes': 115425280, u'BloomFilterMemUsed': 0, u'NumTables': 219, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2162688}, {u'TargetSize': 130023424, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 130023424, u'LogicalSize': 97336181, u'SizeRatio': 0.7486049667481454, u'NWriteBytes': 54063104, u'BloomFilterMemUsed': 0, u'NumTables': 62, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 1646592}, {u'TargetSize': 2225564658L, u'LogicalSize': 1953999895, u'SizeRatio': 0.877979387377565, u'NWriteBytes': 30236672, u'BloomFilterMemUsed': 0, u'NumTables': 879, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 29626368}, {u'TargetSize': 8628652272L, u'LogicalSize': 4451129322L, u'SizeRatio': 0.5158545253288195, u'NWriteBytes': 1641529344, u'BloomFilterMemUsed': 0, u'NumTables': 1101, u'WastedSpace': 136803186, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 4451074048L}], u'NItemsGarbageCollected': 2232597, u'NInternalFragmentationCompacts': 40, u'NTTLCompacts': 4, u'ActiveIndexBlocksSize': 30990243, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 267648, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 404492, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 786216, u'CompactWriter': {u'NDataBlockWrites': 163931, u'NWriteBytes': 494821376, u'NIndexBlockWrites': 3393, u'NBytesIncoming': 749298796, u'NSyncs': 0, u'NInserts': 5165517, u'NWriteIOs': 3184}, u'NonResidentBloomFilterSize': 0, u'IteratorMemUsed': 0, u'NWriteBytes': 1844965978, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 5560, u'NReadBytes': 841128, u'NDataBlockReads': 56, u'NBytesOutgoing': 0, u'NIndexBlockReads': 77}, u'NBloomFilterMisses': 0, u'NItemsMerged': 552032, u'LogicalDiskSize': 4451129322L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.5030243961632944, u'BufferMemUsed': 1327104, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 2497129427L, u'NTableFiles': 2633, u'NFileCountCompacts': 680, u'TableObjectMemUsed': 960480, u'TreeSnapshotMemoryUsed': 366496, u'TotalDiskUsage': 5253902336L, u'BytesIncoming': 3667746519L, u'NItemsDel': 0, u'PendingCheckpointBytes': 5632000, u'NCompacts': 1280, u'NItemsPut': 5819482, u'NSyncs': 6149, u'CompactReader': {u'NReadIOs': 305435, u'NReadBytes': 713589645, u'NDataBlockReads': 289389, u'NBytesOutgoing': 0, u'NIndexBlockReads': 16046}, u'ActiveDiskUsage': 4484509696L, u'NTablesDeleted': 14115, u'IterateReader': {u'NReadIOs': 93497, u'NReadBytes': 33016271, u'NDataBlockReads': 47027, u'NBytesOutgoing': 0, u'NIndexBlockReads': 46470}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 463487, u'NWriteBytes': 1350144602, u'NIndexBlockWrites': 37818, u'NBytesIncoming': 2113696818, u'NSyncs': 6149, u'NInserts': 3453399, u'NWriteIOs': 29441}}, u'NReadBytes': 6122353300L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 182.63333129882812, u'NInternalFragmentationCompacts': 40, u'NTTLCompacts': 8, u'ActiveIndexBlocksSize': 34256400, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 25600, u'NTxns': 613424, u'NWriteBytes': 5769277440L, u'NSegments': 394, u'NSyncs': 613423, u'NLoggedBytes': 5769277440L, u'NActiveSegments': 7, u'NWrites': 613799, u'PreallocateSupported': True, u'NInserts': 7776284, u'DiskUsed': 105111552}, u'NGets': 437036, u'NReadBytesSet': 296042078, u'NReadBytesCompact': 2718370447L, u'IndexResidentRatio': 0.7128582000732422, u'CompactQueueSize': 0, u'NSyncsPerSec': 4.5, u'BlockCacheHitRatio': 57.659828186035156, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 3370966582L, u'BloomFilterMemUsedRatio': 0.059692319482564926, u'NonResidentBloomFilterSize': 1763344, u'NWriteBytes': 10174028738L, u'BytesOutgoing': 50252, u'RecentReadIOAmp': 1.777683138847351, u'WriteAmp': 2.2743160724639893, u'NBlocksCached': 22325, u'LogicalDataSize': 2563305959L, u'ActiveBloomFilterMemUsed': 6303896, u'NFileCountCompacts': 2061, u'NTableFiles': 3991, u'FlushQueueSize': 8, u'TableObjectMemUsed': 1286400, u'BlockCacheQuota': 79279243, u'TotalDiskUsage': 5578743808L, u'NPurgerRuns': 31, u'ReadIOAmpSet': 0.0, u'NSyncs': 18447, u'NReadIOsPerSec': 367.4666748046875, u'NReadIOsSetPerSec': 25.100000381469727, u'NTablesDeleted': 39677, u'BloomFilterFPR': 0.015937326475977898, u'NBloomFilterHitsPerSec': 5993.60009765625, u'WriteCacheMemUsed': 8188200, u'RecentBloomFilterFPR': 0.01637837663292885, u'BytesPerRead': 7035.86083984375, u'BytesIncomingPerSec': 1176971.375, u'NReadIOsGetPerSec': 208.6999969482422, u'NumKVStores': 31, u'NGetStats': 4178, u'TotalIndexBlocksSize': 40808170, u'NGetStatsPerSec': 0.8333333134651184, u'NWriteBytesCompact': 2725912576L, u'NReadIOsSet': 103744, u'ActiveDataSize': 2563305959L, u'NWriteBytesCompactPerSec': 474999.46875, u'NBloomFilterMissesPerSec': 99.80000305175781, u'NCommitBatches': 614883, u'NGetStatsComputed': 4178, u'NWriteIOs': 688149, u'NReadIOsGet': 1185156, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 19819812, u'LSMTreeObjectMemUsed': 534688, u'NSetsPerSec': 1046.1666259765625, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 3.2333333492279053, u'ManagedFileObjectMemUsed': 398016, u'ReadAmpGet': 61190.09375, u'BlockCacheMemUsed': 73103728, u'NWriterCompacts': 0, u'NReadIOs': 2323642, u'WALBufferMemUsed': 81920, u'ReadAmp': 115941.875, u'keyStats': {u'NItemsTransformed': 40797768, u'NTablesCreated': 13156, u'ReadIOAmp': 0.272285421477185, u'MemtablesMemUsed': 779704, u'NBloomFilterHits': 22998088, u'TotalIndexBlocksSize': 4075928, u'CheckpointOverhead': 62103552, u'NItemsGet': 4321693, u'NWriteIOs': 23096, u'NReadBytes': 5352185438L, u'LevelStats': [{u'TargetSize': 130023424, u'LogicalSize': 5032882, u'SizeRatio': 0.03870750242663968, u'NWriteBytes': 223731712, u'BloomFilterMemUsed': 58504, u'NumTables': 243, u'WastedSpace': 285, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 4571136}, {u'TargetSize': 130023424, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 130023424, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 0}, {u'TargetSize': 130023424, u'LogicalSize': 91319624, u'SizeRatio': 0.7023320967151273, u'NWriteBytes': 2120355840, u'BloomFilterMemUsed': 3183616, u'NumTables': 58, u'WastedSpace': 210161, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 91516928}, {u'TargetSize': 157167285, u'LogicalSize': 157167285, u'SizeRatio': 1.0, u'NWriteBytes': 101105664, u'BloomFilterMemUsed': 3056640, u'NumTables': 93, u'WastedSpace': 398332, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 157548544}], u'NItemsGarbageCollected': 3265117, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 2, u'ActiveIndexBlocksSize': 3251711, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 71808, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 1892026, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 1005305, u'NFailedCompacts': 14, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 228033, u'CompactWriter': {u'NDataBlockWrites': 764082, u'NWriteBytes': 2225397760L, u'NIndexBlockWrites': 10733, u'NBytesIncoming': 2872699809L, u'NSyncs': 0, u'NInserts': 58216897, u'NWriteIOs': 5176}, u'NonResidentBloomFilterSize': 1759068, u'IteratorMemUsed': 304, u'NWriteBytes': 2454459853L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 1176734, u'NReadBytes': 3369098130L, u'NDataBlockReads': 5226107, u'NBytesOutgoing': 0, u'NIndexBlockReads': 6642400}, u'NBloomFilterMisses': 372156, u'NItemsMerged': 0, u'LogicalDiskSize': 157167285, u'TotalBloomFilterMemUsed': 6450136, u'WriteAmp': 12.596290855386663, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 6298760, u'LogicalDataSize': 65847661, u'NTableFiles': 748, u'NFileCountCompacts': 688, u'TableObjectMemUsed': 179520, u'TreeSnapshotMemoryUsed': 180736, u'TotalDiskUsage': 319844352, u'BytesIncoming': 194855762, u'NItemsDel': 0, u'PendingCheckpointBytes': 8056832, u'NCompacts': 717, u'NItemsPut': 3897114, u'NSyncs': 6149, u'CompactReader': {u'NReadIOs': 715292, u'NReadBytes': 1983087308, u'NDataBlockReads': 792900, u'NBytesOutgoing': 0, u'NIndexBlockReads': 22278}, u'ActiveDiskUsage': 253636608, u'NTablesDeleted': 13074, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.015924352351648532, u'Writer': {u'NDataBlockWrites': 40771, u'NWriteBytes': 229062093, u'NIndexBlockWrites': 11771, u'NBytesIncoming': 130436302, u'NSyncs': 6149, u'NInserts': 2210790, u'NWriteIOs': 17920}}, u'NReadBytesCompactPerSec': 414927.6875, u'TableMetaMemUsed': 1189297, u'NDeletes': 0, u'BlockCacheHitsPerSec': 345.3333435058594, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 372477, u'LogicalDiskSize': 4608625478L, u'NSets': 3897114, u'TotalBloomFilterMemUsed': 6455576, u'BufferMemUsed': 1327104, u'RecentReadAmp': 0.0, u'NGetsPerSec': 117.4000015258789, u'NInserts': 0, u'NCommitBatchesPerSec': 147.89999389648438, u'FragmentationWithOverhead': 0.5405227541923523, u'NCompactsPerSec': 0.4333333373069763, u'WALMemUsed': 147536, u'MemoryQuota': 108147512, u'NReadBytesPerSec': 1016005.0625, u'TreeSnapshotMemoryUsed': 674944, u'BytesIncoming': 4473445600L, u'NCompacts': 2692, u'NumPersistentFlushes': 5976, u'NWriteIOsPerSec': 165.86666870117188, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 5119.9091796875, u'ActiveDiskUsage': 4740562944L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 0.8333333134651184, u'NReadBytesSetPerSec': 71745.296875, u'BlockCacheHits': 1368232, u'NWriteBytesPerSec': 2419298.75, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 12480, u'ReadIOAmp': 0.9814814814814815, u'MemtablesMemUsed': 1287608, u'NBloomFilterHits': 794, u'TotalIndexBlocksSize': 29735, u'CheckpointOverhead': 2433024, u'NItemsGet': 2916, u'NWriteIOs': 18629, u'NReadBytes': 22720818, u'LevelStats': [{u'TargetSize': 130023424, u'LogicalSize': 2693659, u'SizeRatio': 0.020716721011746313, u'NWriteBytes': 96198656, u'BloomFilterMemUsed': 4144, u'NumTables': 264, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2162688}, {u'TargetSize': 130023424, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 130023424, u'LogicalSize': 328871, u'SizeRatio': 0.0025293211783132246, u'NWriteBytes': 5668864, u'BloomFilterMemUsed': 992, u'NumTables': 31, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 253952}], u'NItemsGarbageCollected': 2072474, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 2, u'ActiveIndexBlocksSize': 14446, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 58560, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 27124, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 175048, u'CompactWriter': {u'NDataBlockWrites': 695, u'NWriteBytes': 5693440, u'NIndexBlockWrites': 695, u'NBytesIncoming': 2147605, u'NSyncs': 0, u'NInserts': 15985, u'NWriteIOs': 695}, u'NonResidentBloomFilterSize': 4276, u'IteratorMemUsed': 0, u'NWriteBytes': 105325467, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 2862, u'NReadBytes': 1027324, u'NDataBlockReads': 1437, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1437}, u'NBloomFilterMisses': 321, u'NItemsMerged': 0, u'LogicalDiskSize': 328871, u'TotalBloomFilterMemUsed': 5440, u'WriteAmp': 0.12003514599815487, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 5136, u'LogicalDataSize': 328871, u'NTableFiles': 610, u'NFileCountCompacts': 693, u'TableObjectMemUsed': 146400, u'TreeSnapshotMemoryUsed': 127712, u'TotalDiskUsage': 4997120, u'BytesIncoming': 877455233, u'NItemsDel': 0, u'PendingCheckpointBytes': 180224, u'NCompacts': 695, u'NItemsPut': 2075224, u'NSyncs': 6149, u'CompactReader': {u'NReadIOs': 24262, u'NReadBytes': 21693494, u'NDataBlockReads': 12488, u'NBytesOutgoing': 0, u'NIndexBlockReads': 12488}, u'ActiveDiskUsage': 2416640, u'NTablesDeleted': 12488, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.2878923766816143, u'Writer': {u'NDataBlockWrites': 11785, u'NWriteBytes': 99632027, u'NIndexBlockWrites': 11785, u'NBytesIncoming': 19205665, u'NSyncs': 6149, u'NInserts': 124452, u'NWriteIOs': 17934}}}}} 2022-02-08 10:40:42,813 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:728] magma stats fragmentation result {'172.23.121.123': [0.40410512685775757, 0.4005993604660034, 0.37638092041015625, 0.3403913676738739, 0.41379934549331665, 0.4286517798900604, 0.3860149085521698, 0.3887837529182434], '172.23.121.124': [0.3971068561077118, 0.40000438690185547, 0.3803377151489258, 0.3812621533870697, 0.37730100750923157, 0.34965944290161133, 0.4077310264110565, 0.4134356677532196], '172.23.121.135': [0.43432778120040894, 0.43177905678749084, 0.4458419680595398, 0.4604016840457916, 0.4423251152038574, 0.4391341209411621, 0.4354647099971771, 0.4446353316307068], '172.23.121.126': [0.3896237015724182, 0.4130128026008606, 0.39248669147491455, 0.41277340054512024, 0.4102669358253479, 0.3784589171409607, 0.37224677205085754, 0.353783518075943], '172.23.121.115': [0.09960731118917465, 0.09147171676158905, 0.08673277497291565, 0.09352719783782959, 0.089515320956707, 0.070395827293396, 0.08924219012260437, 0.09249184280633926], '172.23.121.116': [0.44380250573158264, 0.4421384036540985, 0.44467079639434814, 0.4473440647125244, 0.436629056930542, 0.45043307542800903, 0.4389609098434448, 0.438191682100296], '172.23.121.127': [0.33841168880462646, 0.3743930160999298, 0.37418806552886963, 0.403216153383255, 0.38861846923828125, 0.43235430121421814, 0.4326460361480713, 0.3635554611682892], '172.23.121.129': [0.3896990120410919, 0.4214743673801422, 0.40830308198928833, 0.3515966534614563, 0.3527943789958954, 0.3982015550136566, 0.38122740387916565, 0.400460809469223], '172.23.120.170': [0.39710289239883423, 0.34318190813064575, 0.3299987018108368, 0.3308737576007843, 0.3786831498146057, 0.42096927762031555, 0.43043166399002075, 0.3987157344818115]} 2022-02-08 10:40:43,234 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.116 is 39.0 2022-02-08 10:40:43,632 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.123 is 39.0 2022-02-08 10:40:43,987 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.124 is 39.0 2022-02-08 10:40:44,275 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.126 is 39.0 2022-02-08 10:40:44,559 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.127 is 39.0 2022-02-08 10:40:44,846 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.129 is 39.0 2022-02-08 10:40:45,213 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.135 is 39.0 2022-02-08 10:40:45,591 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.115 is 39.0 2022-02-08 10:40:45,946 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.120.170 is 39.0 2022-02-08 10:40:45,947 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:677] KV stats fragmentation values {'172.23.121.123': 39.0, '172.23.121.124': 39.0, '172.23.121.135': 39.0, '172.23.121.126': 39.0, '172.23.121.115': 39.0, '172.23.121.116': 39.0, '172.23.121.127': 39.0, '172.23.121.129': 39.0, '172.23.120.170': 39.0} ############################################################ # # RAM FootPrint: 4694.9921875 # ############################################################ 2022-02-08 10:40:45,993 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 10:40:45,993 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 97 seconds. Reason: Iteration:0 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 13921 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 13946 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:42:22,997 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 10:42:22,997 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 10:42:28,000 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:42:28,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:42:28,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:42:28,388 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:42:28,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:42:28,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:42:28,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:42:28,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:42:28,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:42:28,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:42:28,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:42:28,532 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:42:28,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 10:42:28,586 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['42839 memcached\n'], []) 2022-02-08 10:42:28,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 10:42:28,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 10:42:28,641 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['42839 memcached\n'], []) 2022-02-08 10:42:28,641 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:42:28,645 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:42:28,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:42:28,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:42:29,049 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:42:29,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:42:29,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:42:29,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:42:29,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:42:29,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:42:29,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:42:29,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:42:29,157 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:42:29,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 10:42:29,221 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['95794 memcached\n'], []) 2022-02-08 10:42:29,221 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 10:42:29,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 10:42:29,269 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['95794 memcached\n'], []) 2022-02-08 10:42:29,269 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:42:29,272 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:42:29,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:42:29,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:42:29,690 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:42:29,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:42:29,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:42:29,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:42:29,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:42:29,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:42:29,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:42:29,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:42:29,796 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:42:29,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 10:42:29,849 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['78945 memcached\n'], []) 2022-02-08 10:42:29,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 10:42:29,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 10:42:29,901 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['78945 memcached\n'], []) 2022-02-08 10:42:29,901 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:42:29,904 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:42:30,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:42:30,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:42:30,280 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:42:30,280 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:42:30,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:42:30,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:42:30,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:42:30,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:42:30,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:42:30,362 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:42:30,377 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:42:30,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 10:42:30,500 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['107057 memcached\n'], []) 2022-02-08 10:42:30,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 10:42:30,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 10:42:30,549 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['107057 memcached\n'], []) 2022-02-08 10:42:30,549 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:42:30,551 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:42:30,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:42:30,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:42:30,890 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:42:30,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:42:30,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:42:30,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:42:30,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:42:30,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:42:30,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:42:30,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:42:31,007 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:42:31,007 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 10:42:31,101 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['100947 memcached\n'], []) 2022-02-08 10:42:31,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 10:42:31,127 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 10:42:31,148 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['100947 memcached\n'], []) 2022-02-08 10:42:31,148 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:42:31,151 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:42:31,244 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:42:31,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:42:31,539 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:42:31,540 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:42:31,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:42:31,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:42:31,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:42:31,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:42:31,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:42:31,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:42:31,641 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:42:31,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 10:42:31,690 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['5362 memcached\n'], []) 2022-02-08 10:42:31,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 10:42:31,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 10:42:31,737 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['5362 memcached\n'], []) 2022-02-08 10:42:31,739 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:42:31,740 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:42:31,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:42:32,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:42:32,148 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:42:32,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:42:32,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:42:32,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:42:32,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:42:32,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:42:32,227 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:42:32,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:42:32,255 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:42:32,255 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 10:42:32,315 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['69865 memcached\n'], []) 2022-02-08 10:42:32,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 10:42:32,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 10:42:32,358 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['69865 memcached\n'], []) 2022-02-08 10:42:32,358 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:42:32,361 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:42:32,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:42:32,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:42:32,796 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:42:32,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:42:32,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:42:32,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:42:32,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:42:32,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:42:32,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:42:32,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:42:32,908 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:42:32,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 10:42:33,045 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['85057 memcached\n'], []) 2022-02-08 10:42:33,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 10:42:33,079 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 10:42:33,105 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['85057 memcached\n'], []) 2022-02-08 10:42:33,105 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:42:33,107 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:42:33,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:42:33,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:42:33,470 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:42:33,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:42:33,487 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:42:33,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:42:33,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:42:33,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:42:33,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:42:33,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:42:33,575 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:42:33,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 10:42:33,627 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['20770 memcached\n'], []) 2022-02-08 10:42:33,627 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 10:42:33,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 10:42:33,681 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['20770 memcached\n'], []) 2022-02-08 10:42:33,681 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:42:33,684 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:42:33,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:42:33,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:42:34,081 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:42:34,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:42:34,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:42:34,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:42:34,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:42:34,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:42:34,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:42:34,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:42:34,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:42:34,197 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 10:42:34,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:42:34,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:42:40,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:42:41,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:42:42,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:42:42,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:42:43,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:42:43,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:42:43,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:42:43,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:42:43,615 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 10:42:43,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:42:43,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:42:43,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:42:43,717 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 10:42:43,717 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:42:43,720 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:42:43,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:42:44,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:42:44,088 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:42:44,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:42:44,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:42:44,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:42:44,132 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:42:44,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:42:44,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:42:44,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:42:44,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:42:44,210 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 10:42:44,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:42:44,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 13965 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:42:51,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:42:52,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:42:52,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:42:53,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:42:53,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:42:53,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:42:53,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:42:53,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:42:53,191 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 10:42:53,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:42:53,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:42:53,285 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:42:53,303 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 10:42:53,305 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:42:53,306 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 10:42:53,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:42:53,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 10:42:53,671 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:42:53,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 10:42:53,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 10:42:53,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 10:42:53,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 10:42:53,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 10:42:53,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 10:42:53,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 10:42:53,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:42:53,780 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 10:42:53,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:42:53,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:42:53,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:42:53,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:42:53,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:42:53,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:42:53,898 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 10:42:53,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:42:53,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:42:53,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:42:53,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:42:53,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:42:54,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:42:54,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:42:54,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:42:54,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:42:54,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:42:54,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:42:54,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:42:54,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:42:54,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:42:54,257 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:42:54,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:42:54,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:42:54,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:42:54,326 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:42:54,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:42:54,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:42:54,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:42:54,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:42:54,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:42:54,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:42:54,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:42:54,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:42:54,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:42:54,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:42:54,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:42:54,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:42:54,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:42:54,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:42:54,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:42:54,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:42:54,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:42:54,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:42:54,640 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:42:54,655 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:42:54,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:42:54,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:42:54,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:42:54,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:42:54,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:42:54,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:42:54,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:42:54,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:42:54,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:42:54,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:42:54,874 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:42:54,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:42:54,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:42:54,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:42:54,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:42:54,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:42:54,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:42:54,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:42:55,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:42:55,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:42:55,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:42:55,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:42:55,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:42:55,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:42:55,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:42:55,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:42:55,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:42:55,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:42:55,191 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:42:55,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:42:55,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:42:55,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:42:55,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:42:55,272 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:42:55,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:42:55,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:42:55,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:42:55,336 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:42:55,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:42:55,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:42:55,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:42:55,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:42:55,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:42:55,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:42:55,457 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 10:42:55,460 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:42:55,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:42:55,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:42:55,821 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:42:55,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:42:55,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:42:55,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:42:55,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:42:55,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:42:55,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:42:55,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:42:55,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:42:55,936 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 10:42:55,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:42:55,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:01,163 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:43:01,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:02,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:02,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:43:02,397 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 10:43:02,397 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:43:02,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:02,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:43:02,482 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 10:43:02,482 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:43:02,484 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 10:43:02,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:43:02,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 10:43:02,841 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:43:02,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 10:43:02,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 10:43:02,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 10:43:02,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 10:43:02,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 10:43:02,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 10:43:02,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 10:43:02,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:43:02,959 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 10:43:02,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:43:03,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:03,022 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:43:03,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:03,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:03,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:43:03,076 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 10:43:03,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:43:03,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:03,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:43:03,119 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:03,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:43:03,186 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:03,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:03,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:03,285 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:03,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:03,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:03,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:03,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:03,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:03,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:43:03,482 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:03,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:03,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:03,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:03,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:03,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:03,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:03,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:03,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:03,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:43:03,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:03,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:03,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:03,696 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:03,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:03,729 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:03,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:03,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:03,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:03,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:43:03,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:03,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:03,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:03,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:03,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:03,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:03,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:03,918 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:03,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:03,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:43:03,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:03,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:03,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:04,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:04,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:04,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:04,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:04,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:04,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:04,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:43:04,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:04,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:04,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:04,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:04,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:04,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:04,224 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:04,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:04,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:04,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:43:04,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:04,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:04,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:04,346 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:04,362 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:04,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:04,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:04,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:04,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:04,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:43:04,454 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:04,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:04,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:04,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:04,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:04,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:04,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:04,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:04,586 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 10:43:04,588 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:43:04,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:43:04,982 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:43:05,049 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:43:05,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:43:05,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:43:05,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:43:05,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:43:05,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:43:05,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:43:05,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:43:05,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:43:05,190 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 10:43:05,214 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:43:05,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:11,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:43:12,397 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:12,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:13,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:13,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:43:13,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:43:13,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:43:13,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:43:13,980 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 10:43:13,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:43:13,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:14,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:43:14,062 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 10:43:14,062 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:43:14,065 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:43:14,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:43:14,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:43:14,467 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:43:14,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:43:14,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:43:14,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:43:14,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:43:14,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:43:14,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:43:14,581 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:43:14,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:43:14,617 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 10:43:14,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:43:14,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:22,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:43:23,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:23,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:24,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:24,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:43:24,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:43:24,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:43:24,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:43:24,470 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 10:43:24,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:43:24,487 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:24,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:43:24,571 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 10:43:24,572 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:43:24,573 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 10:43:24,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:43:24,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 10:43:24,957 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:43:24,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 10:43:24,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 10:43:24,987 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 10:43:25,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 10:43:25,017 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 10:43:25,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 10:43:25,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 10:43:25,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:43:25,084 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 10:43:25,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:43:25,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:25,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:43:25,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:25,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:25,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:43:25,234 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 10:43:25,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:43:25,260 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:25,278 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:43:25,298 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:25,326 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:43:25,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:25,395 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:25,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:25,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:25,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:25,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:25,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:25,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:25,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:25,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:43:25,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:25,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:25,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:25,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:25,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:25,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:25,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:25,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:25,793 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:25,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:43:25,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:25,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:25,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:25,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:25,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:25,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:25,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:25,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:25,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:26,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:43:26,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:26,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:26,056 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:26,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:26,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:26,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:26,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:26,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:26,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:26,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:43:26,201 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:26,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:26,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:26,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:26,270 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:26,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:26,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:26,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:26,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:26,357 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:43:26,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:26,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:26,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:26,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:26,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:26,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:26,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:26,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:26,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:26,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:43:26,543 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:26,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:26,573 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:26,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:26,607 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:26,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:26,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:26,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:26,668 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:26,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:43:26,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:26,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:26,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:26,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:26,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:26,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:26,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:26,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:26,845 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 10:43:26,848 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 10:43:26,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:43:27,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 10:43:27,217 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:43:27,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 10:43:27,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 10:43:27,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 10:43:27,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 10:43:27,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 10:43:27,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 10:43:27,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 10:43:27,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:43:27,331 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 10:43:27,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:43:27,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:27,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:43:27,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:27,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:27,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:43:27,450 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 10:43:27,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:43:27,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:27,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:43:27,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:27,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:43:27,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:27,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:27,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:27,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:27,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:27,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:27,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:27,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:43:27,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:27,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:43:27,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:27,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:27,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:27,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:27,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:27,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:27,936 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:27,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:43:27,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:27,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:43:28,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:28,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:28,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:28,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:28,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:28,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:28,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:28,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:43:28,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:28,170 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:43:28,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:28,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:28,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:28,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:28,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:28,270 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:28,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:28,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:43:28,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:28,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:43:28,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:28,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:28,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:28,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:28,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:28,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:28,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:28,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:43:28,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:28,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:43:28,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:28,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:28,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:28,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:28,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:28,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:28,596 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:28,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:43:28,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:28,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:43:28,658 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:28,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:28,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:28,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:28,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:28,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:28,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:28,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:43:28,778 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:28,793 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:43:28,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:28,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:28,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:28,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:28,887 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:28,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:28,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:28,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:43:28,953 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 10:43:28,956 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:43:29,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:43:29,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:43:29,378 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:43:29,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:43:29,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:43:29,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:43:29,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:43:29,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:43:29,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:43:29,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:43:29,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:43:29,505 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 10:43:29,535 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:43:29,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:36,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:43:37,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:37,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:38,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:38,119 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:43:38,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:43:38,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:43:38,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:43:38,203 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 10:43:38,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:43:38,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:38,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:43:38,326 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 10:43:38,328 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:43:38,331 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:43:38,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:43:39,237 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:43:39,341 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:43:39,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:43:39,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:43:39,374 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:43:39,397 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:43:39,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:43:39,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:43:39,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:43:39,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:43:39,493 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 10:43:39,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:43:39,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:44,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:43:45,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:45,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:46,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:43:46,220 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 10:43:46,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:43:46,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:46,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:43:46,316 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 10:43:46,318 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:43:46,319 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:43:46,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:43:47,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:43:47,256 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:43:47,257 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:43:47,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:43:47,326 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:43:47,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:43:47,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:43:47,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:43:47,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:43:47,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:43:47,446 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 10:43:47,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:43:47,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 13966 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:43:52,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:43:53,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:53,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:43:53,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:43:53,970 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 10:43:53,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:43:53,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:43:54,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:43:54,079 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 10:43:54,081 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:43:54,082 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:43:54,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:43:55,272 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:43:55,486 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:43:55,487 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:43:55,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:43:55,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:43:55,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:43:55,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:43:55,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:43:55,721 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:43:55,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:43:55,812 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 10:43:55,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:43:55,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:44:03,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:44:12,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:44:13,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:44:13,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:44:15,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:44:15,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:44:15,581 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:44:15,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:44:15,694 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 10:44:15,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:44:15,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:44:15,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:44:15,785 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 10:44:15,786 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:44:15,789 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 10:44:15,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:44:16,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 10:44:16,151 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:44:16,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 10:44:16,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 10:44:16,178 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 10:44:16,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 10:44:16,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 10:44:16,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 10:44:16,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 10:44:16,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:44:16,260 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 10:44:16,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:44:16,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:44:16,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:44:16,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:44:16,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:44:16,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:44:16,372 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 10:44:16,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:44:16,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:44:16,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:44:16,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:44:16,474 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:44:16,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:44:16,543 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:44:16,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:44:16,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:44:16,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:44:16,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:44:16,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:44:16,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:44:16,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:44:16,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:44:16,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:44:16,816 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:44:16,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:44:16,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:44:16,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:44:16,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:44:16,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:44:16,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:44:16,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:44:16,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:44:16,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:44:16,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:44:17,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:44:17,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:44:17,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:44:17,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:44:17,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:44:17,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:44:17,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:44:17,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:44:17,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:44:17,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:44:17,173 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:44:17,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:44:17,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:44:17,221 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:44:17,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:44:17,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:44:17,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:44:17,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:44:17,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:44:17,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:44:17,336 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:44:17,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:44:17,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:44:17,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:44:17,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:44:17,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:44:17,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:44:17,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:44:17,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:44:17,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:44:17,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:44:17,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:44:17,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:44:17,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:44:17,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:44:17,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:44:17,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:44:17,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:44:17,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:44:17,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:44:17,655 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:44:17,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:44:17,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:44:17,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:44:17,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:44:17,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:44:17,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:44:17,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:44:17,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:44:17,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:44:17,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:44:17,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:44:17,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:44:17,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:44:17,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:44:17,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:44:17,921 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 10:44:17,921 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:44:17,924 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:44:18,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:44:18,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:44:18,338 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:44:18,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:44:18,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:44:18,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:44:18,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:44:18,397 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:44:18,410 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:44:18,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:44:18,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:44:18,688 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:44:18,688 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.766999959946. 2022-02-08 10:44:18,688 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:44:18,690 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:44:19,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:44:19,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:44:19,854 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:44:19,854 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:44:19,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:44:19,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:44:19,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:44:19,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:44:19,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:44:19,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:44:20,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:44:20,262 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:44:20,262 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.57300019264. 2022-02-08 10:44:20,262 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:44:20,263 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:44:20,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:44:21,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:44:21,108 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:44:21,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:44:21,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:44:21,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:44:21,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:44:21,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:44:21,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:44:21,237 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:44:21,257 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:44:21,500 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:44:21,500 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.2389998436. 2022-02-08 10:44:21,500 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:44:21,503 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:44:22,191 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:44:22,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:44:22,543 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:44:22,543 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:44:22,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:44:22,573 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:44:22,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:44:22,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:44:22,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:44:22,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:44:22,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:44:22,921 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:44:22,923 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.42200016975. 2022-02-08 10:44:22,923 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:44:22,924 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:44:23,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:44:23,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:44:23,895 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:44:23,897 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:44:23,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:44:23,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:44:23,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:44:23,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:44:23,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:44:23,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:44:24,012 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:44:24,338 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:44:24,338 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.41499996185. 2022-02-08 10:44:24,338 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:44:24,339 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:44:24,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:44:24,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:44:24,700 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:44:24,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:44:24,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:44:24,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:44:24,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:44:24,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:44:24,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:44:24,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:44:24,839 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:44:25,121 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:44:25,122 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.784999847412. 2022-02-08 10:44:25,122 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:44:25,125 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:44:25,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:44:25,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:44:25,937 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:44:25,938 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:44:25,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:44:25,969 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:44:25,982 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:44:25,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:44:26,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:44:26,038 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:44:26,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:44:26,338 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:44:26,338 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.21500015259. 2022-02-08 10:44:26,338 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:44:26,339 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:44:27,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:44:27,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:44:27,418 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:44:27,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:44:27,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:44:27,454 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:44:27,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:44:27,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:44:27,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:44:27,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:44:27,540 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:44:27,816 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:44:27,816 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.47799992561. 2022-02-08 10:44:27,818 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:44:27,819 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:44:28,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:44:29,178 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:44:29,298 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:44:29,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:44:29,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:44:29,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:44:29,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:44:29,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:44:29,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:44:29,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:44:29,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:44:30,206 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:44:30,207 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.38899993896. ############################################################ # # RAM FootPrint: 4638.4921875 # ############################################################ 2022-02-08 10:44:30,242 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 10:44:30,243 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 106 seconds. Reason: Iteration:1 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 13977 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 13983 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:46:16,247 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 10:46:16,247 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 10:46:21,252 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:46:21,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:46:21,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:46:21,650 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:46:21,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:46:21,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:46:21,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:46:21,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:46:21,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:46:21,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:46:21,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:46:21,755 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:46:21,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 10:46:21,811 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['50648 memcached\n'], []) 2022-02-08 10:46:21,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 10:46:21,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 10:46:21,861 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['50648 memcached\n'], []) 2022-02-08 10:46:21,861 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:46:21,865 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:46:22,443 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:46:22,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:46:22,792 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:46:22,793 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:46:22,808 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:46:22,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:46:22,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:46:22,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:46:22,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:46:22,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:46:22,920 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:46:22,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 10:46:22,980 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['103927 memcached\n'], []) 2022-02-08 10:46:22,982 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 10:46:23,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 10:46:23,030 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['103927 memcached\n'], []) 2022-02-08 10:46:23,032 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:46:23,033 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:46:23,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:46:23,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:46:23,994 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:46:23,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:46:24,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:46:24,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:46:24,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:46:24,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:46:24,096 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:46:24,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:46:24,181 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:46:24,183 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 10:46:24,325 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['87225 memcached\n'], []) 2022-02-08 10:46:24,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 10:46:24,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 10:46:24,378 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['87225 memcached\n'], []) 2022-02-08 10:46:24,378 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:46:24,381 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:46:24,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:46:24,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:46:24,746 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:46:24,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:46:24,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:46:24,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:46:24,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:46:24,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:46:24,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:46:24,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:46:24,839 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:46:24,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 10:46:24,882 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['114887 memcached\n'], []) 2022-02-08 10:46:24,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 10:46:24,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 10:46:24,931 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['114887 memcached\n'], []) 2022-02-08 10:46:24,931 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:46:24,934 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:46:25,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:46:25,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:46:25,273 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:46:25,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:46:25,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:46:25,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:46:25,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:46:25,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:46:25,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:46:25,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:46:25,371 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:46:25,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 10:46:25,418 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['108758 memcached\n'], []) 2022-02-08 10:46:25,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 10:46:25,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 10:46:25,466 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['108758 memcached\n'], []) 2022-02-08 10:46:25,467 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:46:25,469 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:46:25,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:46:25,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:46:25,861 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:46:25,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:46:25,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:46:25,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:46:25,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:46:25,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:46:25,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:46:25,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:46:25,979 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:46:25,979 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 10:46:26,061 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['13162 memcached\n'], []) 2022-02-08 10:46:26,061 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 10:46:26,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 10:46:26,118 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['13162 memcached\n'], []) 2022-02-08 10:46:26,118 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:46:26,121 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:46:26,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:46:26,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:46:26,552 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:46:26,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:46:26,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:46:26,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:46:26,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:46:26,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:46:26,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:46:26,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:46:26,642 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:46:26,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 10:46:26,690 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['77668 memcached\n'], []) 2022-02-08 10:46:26,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 10:46:26,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 10:46:26,733 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['77668 memcached\n'], []) 2022-02-08 10:46:26,733 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:46:26,736 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:46:26,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:46:27,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:46:27,107 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:46:27,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:46:27,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:46:27,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:46:27,147 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:46:27,163 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:46:27,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:46:27,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:46:27,203 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:46:27,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 10:46:27,269 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['92893 memcached\n'], []) 2022-02-08 10:46:27,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 10:46:27,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 10:46:27,318 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['92893 memcached\n'], []) 2022-02-08 10:46:27,319 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:46:27,322 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:46:28,285 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:46:28,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:46:28,742 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:46:28,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:46:28,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:46:28,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:46:28,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:46:28,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:46:28,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:46:28,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:46:28,861 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:46:28,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 10:46:28,915 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['28628 memcached\n'], []) 2022-02-08 10:46:28,915 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 10:46:28,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 10:46:28,966 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['28628 memcached\n'], []) 2022-02-08 10:46:28,966 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:46:28,973 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:46:29,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:46:29,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:46:29,421 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:46:29,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:46:29,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:46:29,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:46:29,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:46:29,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:46:29,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:46:29,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:46:29,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:46:29,539 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 10:46:29,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:46:29,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:46:36,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:46:37,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:46:37,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:46:38,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:46:38,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:46:38,676 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:46:38,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:46:38,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:46:38,796 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 10:46:38,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:46:38,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:46:38,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:46:38,894 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 10:46:38,894 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:46:38,898 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:46:38,982 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:46:39,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:46:39,244 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:46:39,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:46:39,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:46:39,272 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:46:39,285 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:46:39,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:46:39,313 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:46:39,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:46:39,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:46:39,357 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 10:46:39,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:46:39,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:46:41,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:46:42,729 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:46:43,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:46:43,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:46:43,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:46:44,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:46:44,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:46:44,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:46:44,052 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 10:46:44,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:46:44,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:46:44,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:46:44,160 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 10:46:44,161 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:46:44,163 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 10:46:44,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:46:44,494 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 10:46:44,555 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:46:44,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 10:46:44,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 10:46:44,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 10:46:44,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 10:46:44,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 10:46:44,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 10:46:44,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 10:46:44,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:46:44,667 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 10:46:44,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:46:44,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:46:44,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:46:44,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:46:44,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:46:44,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:46:44,772 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 10:46:44,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:46:44,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:46:44,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:46:44,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:44,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:46:44,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:44,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:44,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:44,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:44,989 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:45,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:45,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:45,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:45,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:45,119 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:46:45,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:45,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:45,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:45,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:45,198 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:45,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:45,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:45,244 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:45,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:45,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:46:45,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:45,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:45,323 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:45,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:45,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:45,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:45,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:45,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:45,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:46:45,428 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:46:45,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:46:45,459 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:46:45,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:46:45,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:46:45,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:46:45,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:46:45,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:46:45,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:46:45,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:45,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:46:45,591 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:45,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:45,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:45,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:45,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:45,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:45,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:45,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:45,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:45,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:46:45,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:45,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:45,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:45,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:45,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:45,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:45,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:45,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:45,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:45,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:46:45,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:45,923 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:45,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:45,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:45,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:45,979 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:45,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:46,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:46,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:46,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:46:46,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:46,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:46,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:46,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:46,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:46,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:46,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:46,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:46,177 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 10:46:46,180 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:46:46,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:46:46,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:46:46,539 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:46:46,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:46:46,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:46:46,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:46:46,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:46:46,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:46:46,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:46:46,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:46:46,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:46:46,644 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 10:46:46,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:46:46,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:46:47,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:46:47,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:46:48,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:46:48,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:46:48,378 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 10:46:48,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:46:48,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:46:48,446 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:46:48,460 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 10:46:48,460 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:46:48,463 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 10:46:48,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:46:48,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 10:46:48,809 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:46:48,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 10:46:48,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 10:46:48,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 10:46:48,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 10:46:48,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 10:46:48,874 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 10:46:48,885 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 10:46:48,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:46:48,911 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 10:46:48,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:46:48,956 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:46:48,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:46:48,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:46:48,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:46:49,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:46:49,019 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 10:46:49,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:46:49,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:46:49,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:46:49,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:49,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:46:49,119 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:49,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:49,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:49,214 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:49,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:49,280 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:49,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:49,346 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:46:49,374 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:49,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:46:49,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:49,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:49,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:49,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:49,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:49,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:49,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:49,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:46:49,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:49,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:46:49,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:49,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:49,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:49,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:49,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:49,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:49,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:49,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:46:49,721 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:46:49,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:46:49,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:46:49,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:46:49,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:46:49,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 5 queries submitted, 0 failed, 14004 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:46:49,816 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:46:49,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:46:49,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:46:49,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:46:49,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:49,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:46:49,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:49,918 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:49,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:49,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:49,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:49,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:49,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:50,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:46:50,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:50,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:46:50,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:50,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:50,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:50,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:50,096 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:50,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:50,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:50,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:46:50,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:50,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:46:50,183 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:50,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:50,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:50,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:50,237 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:50,255 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:50,270 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:50,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:46:50,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:50,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:46:50,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:50,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:50,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:50,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:50,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:50,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:50,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:50,446 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:46:50,471 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 10:46:50,476 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:46:50,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:46:50,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:46:50,894 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:46:50,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:46:50,910 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:46:50,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:46:50,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:46:50,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:46:50,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:46:50,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:46:51,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:46:51,022 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 10:46:51,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:46:51,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:46:53,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:46:54,043 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:46:54,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:46:55,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:46:56,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:46:56,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:46:56,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:46:56,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:46:56,684 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 10:46:56,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:46:56,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:46:56,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:46:56,928 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 10:46:56,928 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:46:56,930 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:46:57,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:46:57,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:46:57,408 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:46:57,410 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:46:57,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:46:57,446 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:46:57,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:46:57,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:46:57,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:46:57,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:46:57,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:46:57,555 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 10:46:57,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:46:57,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:46:59,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:47:00,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:47:00,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:47:01,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:01,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:47:01,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:47:01,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:47:01,658 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:47:01,677 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 10:47:01,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:47:01,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:01,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:47:01,785 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 10:47:01,785 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:47:01,788 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 10:47:01,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:47:02,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 10:47:02,165 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:47:02,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 10:47:02,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 10:47:02,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 10:47:02,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 10:47:02,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 10:47:02,237 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 10:47:02,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 10:47:02,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:47:02,283 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 10:47:02,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:47:02,326 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:02,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:47:02,357 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:47:02,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:47:02,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:47:02,398 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 10:47:02,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:47:02,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:02,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:47:02,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:02,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:47:02,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:02,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:02,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:02,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:02,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:02,676 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:02,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:02,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:02,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:02,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:47:02,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:02,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:02,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:02,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:02,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:02,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:02,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:02,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:02,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:03,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:47:03,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:03,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:03,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:03,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:03,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:03,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:03,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:03,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:03,191 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:03,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:47:03,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:03,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:03,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:03,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:03,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:03,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:03,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:03,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:03,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:03,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:47:03,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:03,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:03,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:03,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:03,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:03,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:03,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:03,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:03,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:03,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:47:03,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:03,624 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:03,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:03,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:03,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:03,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:03,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:03,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:03,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:03,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:47:03,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:03,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:03,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:03,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:03,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:03,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:03,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:03,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:03,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:03,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:47:03,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:03,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:03,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:04,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:04,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:04,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:04,071 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:04,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:04,111 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 10:47:04,115 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 10:47:04,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:47:04,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 10:47:04,506 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:47:04,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 10:47:04,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 10:47:04,535 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 10:47:04,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 10:47:04,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 10:47:04,579 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 10:47:04,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 10:47:04,607 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:47:04,621 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 10:47:04,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:47:04,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:04,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:47:04,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:47:04,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:47:04,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:47:04,730 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 10:47:04,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:47:04,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:04,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:47:04,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:04,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:47:04,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:04,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:04,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:04,923 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:04,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:04,989 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:05,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:05,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:05,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:05,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:47:05,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:05,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:05,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:05,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:05,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:05,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:05,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:05,227 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:05,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:05,257 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:47:05,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:05,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:05,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:05,323 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:05,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:05,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:05,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:05,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:05,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:05,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:47:05,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:05,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:05,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:05,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:05,494 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:05,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:05,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:05,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:05,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:05,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:47:05,584 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:05,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:05,614 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:05,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:05,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:05,658 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:05,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:05,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:05,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:05,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:47:05,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:05,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:05,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:05,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:05,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:05,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:05,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:05,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:05,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:05,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:47:05,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:05,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:05,915 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:05,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:05,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:05,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:05,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:05,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:06,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:06,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:47:06,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:06,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:06,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:06,079 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:06,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:06,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:06,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:06,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:06,153 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 10:47:06,154 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:47:06,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:47:06,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:47:06,538 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:47:06,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:47:06,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:47:06,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:47:06,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:47:06,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:47:06,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:47:06,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:47:06,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:47:06,651 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 10:47:06,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:47:06,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:08,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:47:09,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:47:09,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:47:10,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:10,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:47:10,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:47:10,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:47:10,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:47:10,303 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 10:47:10,303 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:47:10,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:10,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:47:10,392 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 10:47:10,394 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:47:10,395 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:47:10,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:47:10,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:47:10,776 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:47:10,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:47:10,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:47:10,808 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:47:10,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:47:10,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:47:10,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:47:10,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:47:10,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:47:10,904 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 10:47:10,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:47:10,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:11,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:47:12,183 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:47:12,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:47:12,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:47:12,944 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 10:47:12,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:47:12,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:13,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:47:13,019 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 10:47:13,019 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:47:13,020 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:47:13,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:47:13,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:47:13,387 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:47:13,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:47:13,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:47:13,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:47:13,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:47:13,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:47:13,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:47:13,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:47:13,487 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:47:13,503 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 10:47:13,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:47:13,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:14,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:47:14,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:47:15,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:47:15,326 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:47:15,342 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 10:47:15,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:47:15,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:15,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:47:15,384 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 10:47:15,384 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:47:15,388 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:47:15,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:47:15,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:47:15,809 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:47:15,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:47:15,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:47:15,839 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:47:15,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:47:15,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:47:15,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:47:15,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:47:15,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:47:15,934 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 10:47:15,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:47:15,979 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:16,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:47:17,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:47:18,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:47:18,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:18,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:47:18,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:47:18,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:47:19,022 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:47:19,036 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 10:47:19,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:47:19,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:19,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:47:19,082 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 10:47:19,082 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:47:19,085 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 10:47:19,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:47:19,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 10:47:19,438 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:47:19,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 10:47:19,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 10:47:19,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 10:47:19,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 10:47:19,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 10:47:19,505 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 10:47:19,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 10:47:19,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:47:19,543 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 10:47:19,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:47:19,584 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:19,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:47:19,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:47:19,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:47:19,645 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:47:19,658 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 10:47:19,660 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:47:19,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:47:19,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:47:19,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:19,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:47:19,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:19,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:19,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:19,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:19,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:19,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:19,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:19,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:47:20,002 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:20,017 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:47:20,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:20,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:20,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:20,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:20,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:20,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:20,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:20,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:47:20,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:20,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:47:20,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:20,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:20,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:20,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:20,257 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:20,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:20,290 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:20,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:47:20,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:20,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:47:20,357 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:20,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:20,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:20,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:20,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:20,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:20,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:20,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:47:20,482 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:20,497 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:47:20,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:20,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:20,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:20,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:20,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:20,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:20,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:20,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:47:20,640 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:20,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:47:20,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:20,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:20,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:20,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:20,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:20,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:20,795 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:20,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:47:20,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:20,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:47:20,854 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:20,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:20,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:20,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:20,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:20,923 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:20,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:20,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:47:20,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:20,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:47:20,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:21,012 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:21,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:21,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:21,056 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:21,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:21,086 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:21,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:47:21,117 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 10:47:21,118 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:47:21,119 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:47:21,191 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:47:21,395 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:47:21,451 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:47:21,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:47:21,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:47:21,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:47:21,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:47:21,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:47:21,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:47:21,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:47:21,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:47:21,676 | test | WARNING | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 10:47:21,683 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:47:21,683 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:47:23,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:47:23,799 | test | WARNING | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 10:47:23,803 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:47:23,803 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:47:25,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:47:25,917 | test | WARNING | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 10:47:25,923 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:47:25,923 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:47:27,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:47:28,039 | test | WARNING | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 10:47:28,043 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:47:28,045 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:47:30,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:47:30,163 | test | WARNING | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 10:47:30,168 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:47:30,168 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:47:32,170 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:47:32,290 | test | WARNING | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 10:47:32,296 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:47:32,296 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:47:34,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:47:34,407 | test | WARNING | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4205] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1017, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2022-02-08 10:47:34,413 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:47:34,414 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:47:36,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:47:36,555 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:47:36,555 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:47:38,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:47:38,697 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:47:38,697 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:47:40,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:47:40,858 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:47:40,859 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:47:42,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:47:43,134 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:47:43,135 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:47:45,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:47:45,937 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:47:45,937 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:47:47,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:47:48,323 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:47:48,323 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 5 queries submitted, 0 failed, 14004 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:47:50,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:47:50,602 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:47:50,602 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:47:52,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:47:52,818 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:47:52,818 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:47:54,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:47:55,141 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:47:55,141 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:47:57,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:47:57,868 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:47:57,868 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:47:59,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:48:00,211 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:48:00,211 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:48:02,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:48:02,628 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:48:02,628 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:48:04,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:48:04,844 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:48:04,844 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:48:06,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:48:07,013 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:48:07,013 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 45.8959999084. 2022-02-08 10:48:07,013 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:48:07,016 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:48:07,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:48:07,349 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:48:07,421 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:48:07,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:48:07,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:48:07,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:48:07,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:48:07,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:48:07,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:48:07,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:48:07,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:48:07,769 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:48:07,770 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.756999969482. 2022-02-08 10:48:07,770 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:48:07,773 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:48:07,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:48:08,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:48:08,150 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:48:08,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:48:08,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:48:08,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:48:08,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:48:08,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:48:08,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:48:08,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:48:08,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:48:08,487 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:48:08,489 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.717000007629. 2022-02-08 10:48:08,489 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:48:08,490 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:48:09,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:48:09,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:48:09,503 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:48:09,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:48:09,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:48:09,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:48:09,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:48:09,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:48:09,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:48:09,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:48:09,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:48:09,884 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:48:09,884 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.39599990845. 2022-02-08 10:48:09,885 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:48:09,888 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:48:10,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:48:10,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:48:11,059 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:48:11,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:48:11,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:48:11,096 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:48:11,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:48:11,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:48:11,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:48:11,170 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:48:11,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:48:11,482 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:48:11,483 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.59700012207. 2022-02-08 10:48:11,483 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:48:11,484 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:48:12,237 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:48:12,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:48:12,573 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:48:12,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:48:12,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:48:12,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:48:12,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:48:12,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:48:12,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:48:12,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:48:12,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:48:12,928 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:48:12,930 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.4470000267. 2022-02-08 10:48:12,930 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:48:12,931 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:48:13,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:48:13,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:48:13,989 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:48:13,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:48:14,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:48:14,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:48:14,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:48:14,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:48:14,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:48:14,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:48:14,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:48:14,516 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:48:14,516 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.58599996567. 2022-02-08 10:48:14,516 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:48:14,519 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:48:14,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:48:14,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:48:14,907 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:48:14,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:48:14,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:48:14,936 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:48:14,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:48:14,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:48:14,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:48:14,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:48:15,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:48:15,326 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:48:15,326 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.80999994278. 2022-02-08 10:48:15,326 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:48:15,328 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:48:16,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:48:16,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:48:17,157 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:48:17,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:48:17,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:48:17,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:48:17,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:48:17,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:48:17,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:48:17,290 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:48:17,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:48:18,342 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:48:18,344 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.01700019836. ############################################################ # # RAM FootPrint: 4424.66015625 # ############################################################ 2022-02-08 10:48:18,378 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 10:48:18,378 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 78 seconds. Reason: Iteration:2 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14011 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:49:36,384 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 10:49:36,384 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 10:49:41,388 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:49:41,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:49:42,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:49:42,312 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:49:42,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:49:42,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:49:42,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:49:42,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:49:42,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:49:42,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:49:42,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:49:42,437 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:49:42,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 10:49:42,503 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['51324 memcached\n'], []) 2022-02-08 10:49:42,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 10:49:42,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 10:49:42,559 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['51324 memcached\n'], []) 2022-02-08 10:49:42,561 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:49:42,563 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:49:42,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:49:42,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:49:42,934 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:49:42,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:49:42,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:49:42,961 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:49:42,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:49:42,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:49:43,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:49:43,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:49:43,030 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:49:43,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 10:49:43,101 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['104678 memcached\n'], []) 2022-02-08 10:49:43,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 10:49:43,127 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 10:49:43,148 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['104678 memcached\n'], []) 2022-02-08 10:49:43,148 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:49:43,151 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:49:43,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:49:43,443 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:49:43,510 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:49:43,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:49:43,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:49:43,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:49:43,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:49:43,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:49:43,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:49:43,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:49:43,651 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:49:43,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 10:49:43,690 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['88080 memcached\n'], []) 2022-02-08 10:49:43,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 10:49:43,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 10:49:43,743 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['88080 memcached\n'], []) 2022-02-08 10:49:43,743 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:49:43,746 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:49:44,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:49:44,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:49:44,865 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:49:44,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:49:44,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:49:44,900 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:49:44,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:49:44,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:49:44,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:49:44,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:49:44,987 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:49:44,987 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 10:49:45,063 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['115608 memcached\n'], []) 2022-02-08 10:49:45,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 10:49:45,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 10:49:45,135 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['115608 memcached\n'], []) 2022-02-08 10:49:45,137 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:49:45,138 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:49:45,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:49:45,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:49:45,507 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:49:45,507 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:49:45,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:49:45,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:49:45,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:49:45,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:49:45,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:49:45,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:49:45,611 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:49:45,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 10:49:45,657 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['109509 memcached\n'], []) 2022-02-08 10:49:45,658 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 10:49:45,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 10:49:45,704 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['109509 memcached\n'], []) 2022-02-08 10:49:45,706 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:49:45,707 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:49:45,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:49:45,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:49:46,062 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:49:46,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:49:46,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:49:46,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:49:46,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:49:46,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:49:46,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:49:46,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:49:46,217 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:49:46,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 10:49:46,269 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['13912 memcached\n'], []) 2022-02-08 10:49:46,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 10:49:46,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 10:49:46,318 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['13912 memcached\n'], []) 2022-02-08 10:49:46,318 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:49:46,321 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:49:46,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:49:46,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:49:46,680 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:49:46,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:49:46,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:49:46,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:49:46,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:49:46,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:49:46,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:49:46,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:49:46,799 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:49:46,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 10:49:46,868 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['78388 memcached\n'], []) 2022-02-08 10:49:46,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 10:49:46,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 10:49:46,913 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['78388 memcached\n'], []) 2022-02-08 10:49:46,913 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:49:46,914 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:49:46,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:49:47,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:49:47,273 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:49:47,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:49:47,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:49:47,298 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:49:47,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:49:47,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:49:47,336 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:49:47,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:49:47,361 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:49:47,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 10:49:47,424 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['93617 memcached\n'], []) 2022-02-08 10:49:47,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 10:49:47,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 10:49:47,470 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['93617 memcached\n'], []) 2022-02-08 10:49:47,470 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:49:47,473 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:49:47,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:49:47,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:49:47,825 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:49:47,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:49:47,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:49:47,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:49:47,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:49:47,885 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:49:47,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:49:47,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:49:47,924 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:49:47,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 10:49:47,986 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['29378 memcached\n'], []) 2022-02-08 10:49:47,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 10:49:48,012 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 10:49:48,032 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['29378 memcached\n'], []) 2022-02-08 10:49:48,032 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:49:48,035 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:49:48,119 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:49:48,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:49:48,418 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:49:48,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:49:48,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:49:48,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:49:48,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:49:48,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:49:48,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:49:48,505 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:49:48,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:49:48,532 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 10:49:48,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:49:48,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 14015 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:49:54,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:49:55,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:49:56,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:49:56,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:49:57,428 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:49:57,505 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:49:57,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:49:57,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:49:57,621 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 10:49:57,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:49:57,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:49:57,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:49:57,720 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 10:49:57,721 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:49:57,724 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:49:57,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:49:58,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:49:58,109 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:49:58,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:49:58,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:49:58,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:49:58,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:49:58,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:49:58,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:49:58,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:49:58,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:49:58,214 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 10:49:58,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:49:58,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:01,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:02,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:03,237 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:03,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:03,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:03,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:50:03,971 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:50:03,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:50:04,007 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 10:50:04,007 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:50:04,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:04,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:50:04,128 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 10:50:04,130 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:50:04,132 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 10:50:04,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:50:04,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 10:50:04,474 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:50:04,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 10:50:04,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 10:50:04,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 10:50:04,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 10:50:04,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 10:50:04,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 10:50:04,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 10:50:04,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:50:04,575 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 10:50:04,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:50:04,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:04,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:04,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:04,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:04,676 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:50:04,687 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 10:50:04,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:50:04,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:04,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:50:04,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:04,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:50:04,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:04,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:04,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:04,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:04,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:04,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:04,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:05,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:05,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:05,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:50:05,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:05,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:05,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:05,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:05,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:05,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:05,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:05,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:05,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:05,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:50:05,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:05,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:05,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:05,278 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:05,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:05,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:05,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:05,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:05,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:05,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:50:05,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:05,413 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:05,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:05,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:05,459 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:05,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:05,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:05,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:05,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:05,535 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:50:05,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:05,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:05,579 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:05,594 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:05,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:05,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:05,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:05,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:05,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:05,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:50:05,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:05,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:05,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:05,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:05,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:05,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:05,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:05,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:05,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:05,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:50:05,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:05,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:05,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:05,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:05,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:05,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:05,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:05,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:05,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:05,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:50:05,989 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:06,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:06,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:06,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:06,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:06,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:06,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:06,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:06,144 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 10:50:06,147 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:50:06,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:50:06,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:50:06,634 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:50:06,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:50:06,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:50:06,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:50:06,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:50:06,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:50:06,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:50:06,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:50:06,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:50:06,747 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 10:50:06,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:50:06,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:08,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:08,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:09,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:09,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:50:09,438 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 10:50:09,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:50:09,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:09,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:50:09,517 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 10:50:09,517 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:50:09,519 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 10:50:09,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:50:09,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 10:50:09,861 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:50:09,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 10:50:09,874 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 10:50:09,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 10:50:09,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 10:50:09,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 10:50:09,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 10:50:09,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 10:50:09,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:50:09,960 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 10:50:09,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:50:09,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:10,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:10,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:10,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:10,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:50:10,058 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 10:50:10,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:50:10,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:10,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:50:10,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:10,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:50:10,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:10,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:10,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:10,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:10,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:10,308 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:10,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:10,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:10,395 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:10,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:50:10,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:10,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:10,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:10,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:10,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:10,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:10,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:10,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:10,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:10,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:50:10,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:10,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:10,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:10,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:10,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:10,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:10,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:10,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:10,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:10,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:50:10,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:10,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:10,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:10,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:10,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:10,816 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:10,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:10,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:10,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:10,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:50:10,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:10,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:10,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:10,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:10,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:10,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:10,979 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:10,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:11,007 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:11,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:50:11,038 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:11,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:11,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:11,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:11,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:11,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:11,119 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:11,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:11,147 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:11,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:50:11,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:11,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:11,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:11,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:11,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:11,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:11,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:11,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:11,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:11,303 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:50:11,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:11,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:11,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:11,362 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:11,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:11,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:11,410 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:11,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:11,438 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 10:50:11,440 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:50:11,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:50:11,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:50:12,144 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:50:12,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:50:12,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:50:12,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:50:12,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:50:12,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:50:12,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:50:12,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:50:12,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:50:12,516 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 10:50:12,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:50:12,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:15,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:16,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:17,170 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:17,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:18,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:18,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:50:18,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:50:18,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:50:18,760 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 10:50:18,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:50:18,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:18,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:50:18,852 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 10:50:18,854 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:50:18,855 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:50:18,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:50:19,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:50:19,289 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:50:19,290 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:50:19,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:50:19,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:50:19,346 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:50:19,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:50:19,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:50:19,410 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:50:19,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:50:19,453 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 10:50:19,482 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:50:19,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:27,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:28,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:28,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:29,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:29,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:29,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:50:29,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:50:29,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:50:29,582 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 10:50:29,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:50:29,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:29,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:50:29,684 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 10:50:29,686 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:50:29,688 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 10:50:29,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:50:29,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 10:50:30,062 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:50:30,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 10:50:30,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 10:50:30,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 10:50:30,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 10:50:30,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 10:50:30,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 10:50:30,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 10:50:30,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:50:30,187 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 10:50:30,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:50:30,227 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:30,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:30,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:30,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:30,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:50:30,306 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 10:50:30,308 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:50:30,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:30,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:50:30,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:30,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:50:30,426 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:30,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:30,494 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:30,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:30,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:30,596 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:30,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:30,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:30,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:30,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:50:30,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:30,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:30,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:30,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:30,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:30,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:30,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:30,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:30,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:30,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:50:30,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:30,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:30,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:30,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:30,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:30,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:31,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:31,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:31,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:31,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:50:31,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:31,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:31,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:31,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:31,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:31,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:31,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:31,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:31,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:31,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:50:31,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:31,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:31,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:31,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:31,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:31,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:31,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:31,403 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:31,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:31,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:50:31,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:31,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:31,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:31,497 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:31,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:31,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:31,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:31,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:31,573 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:31,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:50:31,604 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:31,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:31,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:31,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:31,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:31,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:31,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:31,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:31,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:31,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:50:31,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:31,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:31,839 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:31,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:31,874 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:31,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:31,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:31,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:31,947 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 10:50:31,950 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 10:50:32,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:50:32,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 10:50:32,326 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:50:32,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 10:50:32,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 10:50:32,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 10:50:32,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 10:50:32,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 10:50:32,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 10:50:32,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 10:50:32,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:50:32,447 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 10:50:32,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:50:32,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:32,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:32,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:32,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:32,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:50:32,571 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 10:50:32,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:50:32,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:32,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:50:32,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:32,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:50:32,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:32,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:32,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:32,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:32,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:32,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:32,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:32,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:32,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:32,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:50:32,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:33,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:33,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:33,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:33,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:33,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:33,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:33,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:50:33,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:33,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:50:33,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:33,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:33,196 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:33,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:33,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:33,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:33,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:33,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:50:33,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:33,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:50:33,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:33,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:33,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:33,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:33,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:33,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:33,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:33,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:50:33,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:33,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:50:33,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:33,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:33,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:33,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:33,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:33,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:33,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:33,591 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:50:33,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:33,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:50:33,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:33,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:33,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:33,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:33,696 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:33,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:33,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:33,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:50:33,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:33,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:50:33,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:33,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:33,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:33,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:33,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:33,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:33,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:33,885 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:50:33,900 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:33,915 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:50:33,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:33,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:33,961 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:33,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:33,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:34,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:34,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:34,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:50:34,062 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 10:50:34,065 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:50:34,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:50:34,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:50:34,493 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:50:34,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:50:34,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:50:34,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:50:34,535 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:50:34,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:50:34,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:50:34,584 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:50:34,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:50:34,611 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 10:50:34,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:50:34,658 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:39,655 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:40,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:41,446 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:41,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:42,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:42,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:50:42,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:50:42,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:50:42,348 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 10:50:42,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:50:42,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:42,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:50:42,447 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 10:50:42,447 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:50:42,450 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:50:42,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:50:42,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:50:43,108 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:50:43,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:50:43,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:50:43,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:50:43,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:50:43,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:50:43,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:50:43,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:50:43,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:50:43,282 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 10:50:43,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:50:43,331 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:44,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:44,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:45,306 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:45,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:50:45,726 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 10:50:45,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:50:45,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:45,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:50:45,790 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 10:50:45,790 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:50:45,793 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:50:45,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:50:46,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:50:46,201 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:50:46,201 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:50:46,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:50:46,237 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:50:46,255 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:50:46,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:50:46,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:50:46,308 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:50:46,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:50:46,344 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 10:50:46,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:50:46,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:46,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:47,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:47,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:48,255 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:50:48,275 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 10:50:48,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:50:48,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:48,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:50:48,323 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 10:50:48,325 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:50:48,326 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:50:48,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:50:48,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:50:48,855 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:50:48,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:50:48,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:50:48,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:50:48,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:50:48,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:50:48,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:50:48,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:50:48,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:50:49,020 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 10:50:49,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:50:49,071 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 14017 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:50:55,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:56,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:57,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:58,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:58,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:58,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:50:58,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:50:58,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:50:59,025 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 10:50:59,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:50:59,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:59,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:50:59,073 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 10:50:59,075 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:50:59,076 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 10:50:59,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:50:59,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 10:50:59,499 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:50:59,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 10:50:59,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 10:50:59,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 10:50:59,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 10:50:59,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 10:50:59,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 10:50:59,594 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 10:50:59,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:50:59,627 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 10:50:59,660 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:50:59,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:59,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:50:59,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:59,729 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:50:59,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:50:59,762 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 10:50:59,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:50:59,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:50:59,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:50:59,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:59,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:50:59,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:59,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:50:59,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:51:00,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:51:00,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:51:00,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:51:00,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:51:00,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:51:00,198 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:51:00,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:51:00,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:51:00,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:51:00,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:51:00,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:51:00,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:51:00,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:51:00,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:51:00,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:51:00,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:51:00,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:51:00,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:51:00,454 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:51:00,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:51:00,494 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:51:00,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:51:00,540 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:51:00,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:51:00,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:51:00,604 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:51:00,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:51:00,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:51:00,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:51:00,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:51:00,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:51:00,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:51:00,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:51:00,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:51:00,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:51:00,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:51:00,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:51:00,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:51:00,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:51:00,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:51:00,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:51:00,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:51:00,938 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:51:00,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:51:00,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:51:01,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:51:01,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:51:01,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:51:01,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:51:01,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:51:01,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:51:01,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:51:01,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:51:01,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:51:01,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:51:01,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:51:01,201 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:51:01,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:51:01,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:51:01,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:51:01,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:51:01,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:51:01,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:51:01,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:51:01,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:51:01,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:51:01,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:51:01,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:51:01,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:51:01,428 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:51:01,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:51:01,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:51:01,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:51:01,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:51:01,520 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:51:01,540 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 10:51:01,540 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:51:01,543 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:51:01,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:51:02,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:51:02,838 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:51:02,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:51:02,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:51:02,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:51:03,147 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:51:03,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:51:03,428 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:51:03,494 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:51:03,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:03,719 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:51:03,720 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:51:05,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:05,983 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:51:05,984 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:51:07,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:08,398 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:51:08,398 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:51:10,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:10,852 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:51:10,852 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:51:12,854 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:13,365 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:51:13,367 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:51:15,367 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:15,548 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:51:15,548 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.116 2022-02-08 10:51:17,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:17,696 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:51:17,697 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 16.1560001373. 2022-02-08 10:51:17,697 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:51:17,700 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:51:17,811 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:51:18,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:51:18,137 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:51:18,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:51:18,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:51:18,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:51:18,178 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:51:18,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:51:18,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:51:18,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:51:18,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:18,500 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:51:18,500 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.803999900818. 2022-02-08 10:51:18,500 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:51:18,503 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:51:18,854 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:51:19,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:51:19,227 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:51:19,227 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:51:19,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:51:19,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:51:19,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:51:19,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:51:19,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:51:19,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:51:19,357 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:19,628 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:51:19,628 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.12700009346. 2022-02-08 10:51:19,628 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:51:19,631 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:51:19,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:51:19,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:51:20,039 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:51:20,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:51:20,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:51:20,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:51:20,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:51:20,102 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:51:20,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:51:20,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:51:20,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:20,667 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:51:20,667 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.03900003433. 2022-02-08 10:51:20,668 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:51:20,671 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:51:21,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:51:21,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:51:21,674 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:51:21,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:51:21,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:51:21,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:51:21,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:51:21,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:51:21,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:51:21,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:51:21,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:22,062 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:51:22,063 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.39599990845. 2022-02-08 10:51:22,063 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:51:22,065 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:51:22,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:51:23,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:51:23,318 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:51:23,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:51:23,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:51:23,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:51:23,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:51:23,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:51:23,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:51:23,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:51:23,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:23,782 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:51:23,782 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.71900010109. 2022-02-08 10:51:23,783 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:51:23,785 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:51:24,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:51:24,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:51:24,802 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:51:24,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:51:24,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:51:24,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:51:24,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:51:24,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:51:24,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:51:24,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:51:24,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:25,428 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:51:25,428 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.64599990845. 2022-02-08 10:51:25,428 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:51:25,430 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:51:25,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:51:26,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:51:26,296 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:51:26,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:51:26,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:51:26,329 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:51:26,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:51:26,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:51:26,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:51:26,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:51:26,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:26,892 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:51:26,894 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.46500015259. 2022-02-08 10:51:26,894 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:51:26,895 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:51:27,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:51:27,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:51:27,894 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:51:27,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:51:27,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:51:28,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:51:28,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:51:28,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:51:28,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:51:28,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:51:28,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:29,605 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:51:29,605 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 10:51:31,607 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:32,006 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:51:32,006 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 10:51:34,007 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:34,326 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:51:34,326 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 10:51:36,326 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:36,951 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:51:36,953 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 10:51:38,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:51:39,094 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:51:39,094 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 12.2000000477. ############################################################ # # RAM FootPrint: 4301.8203125 # ############################################################ 2022-02-08 10:51:39,128 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 10:51:39,128 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 67 seconds. Reason: Iteration:3 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14024 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:52:46,132 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 10:52:46,132 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 5 queries submitted, 0 failed, 14025 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:52:51,134 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:52:51,640 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:52:51,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:52:52,052 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:52:52,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:52:52,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:52:52,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:52:52,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:52:52,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:52:52,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:52:52,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:52:52,191 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:52:52,191 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 10:52:52,244 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['52259 memcached\n'], []) 2022-02-08 10:52:52,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 10:52:52,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 10:52:52,302 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['52259 memcached\n'], []) 2022-02-08 10:52:52,302 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:52:52,309 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:52:52,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:52:52,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:52:52,752 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:52:52,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:52:52,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:52:52,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:52:52,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:52:52,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:52:52,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:52:52,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:52:52,874 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:52:52,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 10:52:52,940 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['105355 memcached\n'], []) 2022-02-08 10:52:52,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 10:52:52,966 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 10:52:52,986 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['105355 memcached\n'], []) 2022-02-08 10:52:52,987 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:52:52,990 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:52:53,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:52:54,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:52:54,167 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:52:54,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:52:54,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:52:54,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:52:54,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:52:54,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:52:54,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:52:54,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:52:54,302 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:52:54,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 10:52:54,390 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['88859 memcached\n'], []) 2022-02-08 10:52:54,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 10:52:54,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 10:52:54,444 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['88859 memcached\n'], []) 2022-02-08 10:52:54,444 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:52:54,447 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:52:54,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:52:54,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:52:54,857 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:52:54,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:52:54,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:52:54,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:52:54,897 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:52:54,913 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:52:54,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:52:54,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:52:54,957 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:52:54,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 10:52:55,016 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['116331 memcached\n'], []) 2022-02-08 10:52:55,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 10:52:55,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 10:52:55,065 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['116331 memcached\n'], []) 2022-02-08 10:52:55,065 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:52:55,069 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:52:55,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:52:55,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:52:55,418 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:52:55,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:52:55,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:52:55,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:52:55,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:52:55,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:52:55,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:52:55,510 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:52:55,532 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:52:55,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 10:52:55,591 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['110190 memcached\n'], []) 2022-02-08 10:52:55,591 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 10:52:55,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 10:52:55,638 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['110190 memcached\n'], []) 2022-02-08 10:52:55,638 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:52:55,640 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:52:55,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:52:55,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:52:56,009 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:52:56,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:52:56,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:52:56,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:52:56,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:52:56,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:52:56,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:52:56,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:52:56,104 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:52:56,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 10:52:56,157 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['14592 memcached\n'], []) 2022-02-08 10:52:56,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 10:52:56,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 10:52:56,201 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['14592 memcached\n'], []) 2022-02-08 10:52:56,201 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:52:56,204 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:52:56,278 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:52:56,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:52:56,561 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:52:56,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:52:56,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:52:56,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:52:56,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:52:56,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:52:56,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:52:56,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:52:56,667 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:52:56,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 10:52:56,717 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['79081 memcached\n'], []) 2022-02-08 10:52:56,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 10:52:56,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 10:52:56,763 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['79081 memcached\n'], []) 2022-02-08 10:52:56,763 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:52:56,766 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:52:56,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:52:57,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:52:57,150 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:52:57,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:52:57,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:52:57,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:52:57,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:52:57,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:52:57,221 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:52:57,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:52:57,249 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:52:57,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 10:52:57,332 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['94279 memcached\n'], []) 2022-02-08 10:52:57,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 10:52:57,367 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 10:52:57,398 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['94279 memcached\n'], []) 2022-02-08 10:52:57,398 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:52:57,400 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:52:57,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:52:58,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:52:58,628 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:52:58,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:52:58,647 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:52:58,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:52:58,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:52:58,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:52:58,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:52:58,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:52:58,766 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:52:58,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 10:52:58,835 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['30070 memcached\n'], []) 2022-02-08 10:52:58,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 10:52:58,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 10:52:58,881 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['30070 memcached\n'], []) 2022-02-08 10:52:58,881 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:52:58,884 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:52:58,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:52:59,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:52:59,236 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:52:59,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:52:59,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:52:59,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:52:59,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:52:59,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:52:59,306 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:52:59,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:52:59,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:52:59,348 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 10:52:59,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:52:59,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:05,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:06,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:07,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:07,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:07,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:07,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:53:07,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:53:07,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:53:07,910 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 10:53:07,910 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:53:07,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:07,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:53:08,006 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 10:53:08,006 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:53:08,009 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:53:08,086 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:53:08,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:53:08,364 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:53:08,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:53:08,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:53:08,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:53:08,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:53:08,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:53:08,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:53:08,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:53:08,459 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:53:08,473 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 10:53:08,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:53:08,520 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:11,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:12,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:12,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:13,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:13,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:13,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:53:13,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:53:13,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:53:13,736 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 10:53:13,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:53:13,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:13,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:53:13,838 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 10:53:13,838 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:53:13,841 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 10:53:13,913 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:53:14,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 10:53:14,207 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:53:14,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 10:53:14,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 10:53:14,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 10:53:14,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 10:53:14,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 10:53:14,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 10:53:14,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 10:53:14,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:53:14,319 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 10:53:14,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:53:14,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:14,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:14,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:14,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:14,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:53:14,433 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 10:53:14,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:53:14,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:14,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:53:14,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:14,507 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:53:14,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:14,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:14,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:14,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:14,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:14,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:14,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:14,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:14,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:14,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:53:14,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:14,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:14,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:14,913 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:14,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:14,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:14,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:14,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:14,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:15,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:53:15,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:15,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:15,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:15,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:15,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:15,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:15,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:15,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:15,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:15,196 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:53:15,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:15,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:15,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:15,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:15,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:15,308 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:15,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:15,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:15,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:15,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:53:15,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:15,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:15,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:15,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:15,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:15,487 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:15,505 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:15,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:15,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:15,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:53:15,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:15,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:15,607 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:15,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:15,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:15,655 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:15,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:15,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:15,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:15,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:53:15,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:15,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:15,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:15,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:15,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:15,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:15,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:15,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:15,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:15,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:53:15,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:15,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:15,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:15,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:15,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:16,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:16,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:16,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:16,059 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 10:53:16,062 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:53:16,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:53:16,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:53:17,346 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:53:17,346 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:53:17,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:53:17,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:53:17,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:53:18,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:53:18,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:53:18,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:53:18,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:53:18,559 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 10:53:18,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:53:18,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:19,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:20,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:21,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:21,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:53:21,710 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 10:53:21,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:53:21,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:21,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:53:21,926 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 10:53:21,926 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:53:21,928 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 10:53:22,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:53:22,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 10:53:22,319 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:53:22,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 10:53:22,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 10:53:22,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 10:53:22,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 10:53:22,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 10:53:22,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 10:53:22,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 10:53:22,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:53:22,434 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 10:53:22,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:53:22,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:22,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:22,510 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:22,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:22,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:53:22,553 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 10:53:22,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:53:22,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:22,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:53:22,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:22,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:53:22,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:22,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:22,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:22,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:22,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:22,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:22,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:22,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:22,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:22,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:53:23,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:23,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:23,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:23,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:23,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:23,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:23,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:23,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:23,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:23,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:53:23,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:23,198 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:23,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:23,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:23,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:23,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:23,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:23,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:23,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:23,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:53:23,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:23,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:23,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:23,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:23,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:23,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:23,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:23,482 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:23,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:23,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:53:23,535 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:23,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:23,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:23,584 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:23,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:23,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:23,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:23,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:23,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:23,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:53:23,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:23,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:23,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:23,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:23,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:23,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:23,795 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:23,811 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:23,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:23,839 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:53:23,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:23,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:23,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:23,897 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:23,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:23,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:23,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:23,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:23,969 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:23,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:53:24,002 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:24,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:24,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:24,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:24,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:24,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:24,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:24,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:24,141 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 10:53:24,142 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:53:24,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:53:24,469 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:53:24,539 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:53:24,540 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:53:24,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:53:24,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:53:24,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:53:24,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:53:24,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:53:24,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:53:24,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:53:24,720 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 10:53:24,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:53:24,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:26,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:27,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:28,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:28,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:29,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:29,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:53:29,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:53:29,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:53:29,282 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 10:53:29,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:53:29,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:29,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:53:29,368 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 10:53:29,368 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:53:29,371 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:53:29,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:53:29,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:53:29,776 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:53:29,778 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:53:29,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:53:29,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:53:29,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:53:29,854 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:53:29,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:53:29,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:53:29,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:53:29,928 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 10:53:29,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:53:29,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:31,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:32,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:32,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:33,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:33,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:33,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:53:33,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:53:33,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:53:33,270 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 10:53:33,270 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:53:33,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:33,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:53:33,372 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 10:53:33,372 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:53:33,375 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 10:53:33,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:53:33,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 10:53:33,746 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:53:33,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 10:53:33,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 10:53:33,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 10:53:33,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 10:53:33,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 10:53:33,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 10:53:33,834 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 10:53:33,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:53:33,865 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 10:53:33,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:53:33,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:33,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:33,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:33,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:33,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:53:33,973 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 10:53:33,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:53:34,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:34,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:53:34,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:34,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:53:34,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:34,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:34,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:34,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:34,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:34,280 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:34,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:34,349 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:34,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:34,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:53:34,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:34,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:34,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:34,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:34,482 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:34,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:34,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:34,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:34,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:34,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:53:34,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:34,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:34,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:34,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:34,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:34,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:34,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:34,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:34,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:34,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:53:34,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:34,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:34,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:34,808 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:34,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:34,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:34,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:34,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:34,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:34,910 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:53:34,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:34,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:34,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:34,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:34,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:35,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:35,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:35,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:35,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:35,079 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:53:35,096 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:35,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:35,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:35,147 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:35,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:35,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:35,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:35,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:35,224 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:35,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:53:35,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:35,278 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:35,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:35,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:35,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:35,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:35,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:35,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:35,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:35,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:53:35,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:35,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:35,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:35,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:35,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:35,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:35,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:35,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:35,559 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 10:53:35,562 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 10:53:35,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:53:35,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 10:53:35,907 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:53:35,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 10:53:35,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 10:53:35,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 10:53:35,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 10:53:35,966 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 10:53:35,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 10:53:35,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 10:53:36,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:53:36,026 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 10:53:36,061 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:53:36,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:36,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:36,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:36,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:36,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:53:36,142 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 10:53:36,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:53:36,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:36,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:53:36,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:36,214 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:53:36,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:36,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:36,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:36,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:36,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:36,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:36,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:36,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:36,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:36,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:53:36,573 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:36,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:36,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:36,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:36,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:36,660 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:36,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:36,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:36,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:36,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:53:36,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:36,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:36,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:36,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:36,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:36,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:36,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:36,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:36,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:36,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:53:36,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:36,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:36,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:36,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:36,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:37,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:37,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:37,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:37,071 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:37,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:53:37,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:37,127 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:37,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:37,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:37,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:37,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:37,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:37,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:37,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:37,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:53:37,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:37,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:37,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:37,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:37,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:37,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:37,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:37,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:37,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:37,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:53:37,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:37,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:37,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:37,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:37,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:37,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:37,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:37,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:37,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:37,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:53:37,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:37,614 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:37,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:37,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:37,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:37,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:37,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:37,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:37,743 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 10:53:37,744 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:53:37,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:53:38,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:53:38,125 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:53:38,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:53:38,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:53:38,214 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:53:38,229 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:53:38,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:53:38,260 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:53:38,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:53:38,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:53:38,308 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 10:53:38,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:53:38,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:39,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:40,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:40,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:41,079 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:41,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:41,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:53:41,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:53:41,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:53:41,247 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 10:53:41,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:53:41,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:41,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:53:41,332 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 10:53:41,332 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:53:41,335 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:53:41,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:53:41,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:53:41,713 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:53:41,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:53:41,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:53:41,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:53:41,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:53:41,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:53:41,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:53:41,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:53:41,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:53:41,838 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 10:53:41,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:53:41,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:42,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:43,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:43,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:43,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:53:43,867 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 10:53:43,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:53:43,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:43,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:53:43,931 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 10:53:43,931 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:53:43,934 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:53:44,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:53:44,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:53:44,953 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:53:44,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:53:44,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:53:45,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:53:45,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:53:45,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:53:45,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:53:45,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:53:45,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:53:45,168 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 10:53:45,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:53:45,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:46,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:46,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:47,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:47,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:53:47,471 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 10:53:47,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:53:47,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:47,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:53:47,552 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 10:53:47,553 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:53:47,555 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:53:47,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:53:47,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:53:47,927 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:53:47,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:53:47,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:53:47,956 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:53:47,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:53:47,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:53:48,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:53:48,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:53:48,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:53:48,046 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 10:53:48,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:53:48,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:49,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 5 queries submitted, 0 failed, 14033 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:53:50,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:51,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:51,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:52,573 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:52,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:53:52,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:53:52,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:53:52,924 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 10:53:52,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:53:52,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:52,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:53:53,003 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 10:53:53,003 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:53:53,005 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 10:53:53,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:53:53,349 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 10:53:53,414 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:53:53,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 10:53:53,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 10:53:53,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 10:53:53,454 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 10:53:53,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 10:53:53,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 10:53:53,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 10:53:53,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:53:53,532 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 10:53:53,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:53:53,573 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:53,591 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:53:53,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:53,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:53:53,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:53:53,648 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 10:53:53,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:53:53,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:53:53,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:53:53,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:53,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:53:53,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:53,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:53,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:53,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:53,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:53,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:53,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:54,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:53:54,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:54,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:53:54,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:54,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:54,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:54,160 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:54,178 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:54,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:54,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:54,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:53:54,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:54,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:53:54,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:54,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:54,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:54,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:54,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:54,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:54,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:54,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:53:54,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:54,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:53:54,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:54,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:54,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:54,556 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:54,573 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:54,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:54,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:54,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:53:54,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:54,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:53:54,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:54,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:54,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:54,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:54,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:54,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:54,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:54,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:53:54,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:54,836 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:53:54,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:54,874 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:54,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:54,910 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:54,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:54,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:54,961 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:54,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:53:55,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:55,017 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:53:55,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:55,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:55,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:55,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:55,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:55,119 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:55,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:55,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:53:55,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:55,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:53:55,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:55,237 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:55,255 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:55,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:55,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:55,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:55,329 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:55,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:53:55,365 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 10:53:55,365 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:53:55,368 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:53:55,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:53:55,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:53:55,782 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:53:55,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:53:55,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:53:55,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:53:55,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:53:55,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:53:55,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:53:55,918 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:53:55,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:53:56,221 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:53:56,223 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.857000112534. 2022-02-08 10:53:56,223 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:53:56,224 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:53:57,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:53:57,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:53:58,010 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:53:58,012 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:53:58,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:53:58,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:53:58,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:53:58,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:53:58,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:53:58,303 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:53:58,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:53:58,888 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:53:58,890 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 10:54:00,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:54:01,269 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:54:01,270 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 10:54:03,272 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:54:03,548 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:54:03,548 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 10:54:05,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:54:05,736 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:54:05,736 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 10:54:07,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:54:07,914 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:54:07,914 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 10:54:09,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:54:10,141 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:54:10,141 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 13.9179999828. 2022-02-08 10:54:10,141 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:54:10,142 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:54:10,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:54:10,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:54:10,736 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:54:10,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:54:10,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:54:10,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:54:10,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:54:10,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:54:10,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:54:10,936 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:54:10,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:54:11,358 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:54:11,358 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 10:54:13,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:54:13,719 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:54:13,720 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 10:54:15,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:54:15,901 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:54:15,901 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 10:54:17,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:54:18,144 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:54:18,144 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 10:54:20,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:54:20,375 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:54:20,375 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 10.2339999676. 2022-02-08 10:54:20,375 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:54:20,378 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:54:20,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:54:21,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:54:21,332 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:54:21,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:54:21,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:54:21,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:54:21,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:54:21,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:54:21,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:54:21,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:54:21,520 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:54:22,153 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:54:22,153 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.77800011635. 2022-02-08 10:54:22,154 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:54:22,154 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:54:22,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:54:23,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:54:23,407 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:54:23,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:54:23,428 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:54:23,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:54:23,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:54:23,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:54:23,510 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:54:23,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:54:23,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:54:23,868 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:54:23,869 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.71599984169. 2022-02-08 10:54:23,869 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:54:23,871 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:54:24,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:54:24,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:54:24,907 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:54:24,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:54:24,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:54:24,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:54:24,966 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:54:24,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:54:25,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:54:25,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:54:25,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:54:25,494 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:54:25,496 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.62600016594. 2022-02-08 10:54:25,496 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:54:25,497 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:54:26,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:54:26,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:54:26,877 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:54:26,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:54:26,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:54:26,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:54:26,918 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:54:26,936 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:54:26,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:54:26,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:54:26,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:54:27,433 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:54:27,434 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.93799996376. 2022-02-08 10:54:27,434 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:54:27,436 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:54:28,426 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:54:28,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:54:28,986 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:54:28,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:54:29,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:54:29,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:54:29,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:54:29,102 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:54:29,119 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:54:29,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:54:29,183 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:54:29,704 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:54:29,704 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 10:54:31,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:54:31,910 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:54:31,910 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.47600007057. 2022-02-08 10:54:31,910 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:54:31,911 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:54:32,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:54:32,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:54:32,944 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:54:32,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:54:32,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:54:32,979 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:54:33,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:54:33,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:54:33,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:54:33,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:54:33,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:54:33,624 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:54:33,624 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.71399998665. ############################################################ # # RAM FootPrint: 4288.08984375 # ############################################################ 2022-02-08 10:54:33,655 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 10:54:33,655 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 98 seconds. Reason: Iteration:4 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14045 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 14068 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:56:11,661 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 10:56:11,661 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 10:56:16,664 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:56:16,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:16,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:56:17,032 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:17,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:56:17,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:56:17,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:56:17,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:56:17,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:56:17,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:56:17,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:56:17,128 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:56:17,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 10:56:17,190 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['52984 memcached\n'], []) 2022-02-08 10:56:17,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 10:56:17,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 10:56:17,234 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['52984 memcached\n'], []) 2022-02-08 10:56:17,234 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:56:17,236 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:56:17,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:17,614 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:56:17,671 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:17,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:56:17,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:56:17,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:56:17,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:56:17,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:56:17,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:56:17,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:56:17,769 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:56:17,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 10:56:17,834 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['106038 memcached\n'], []) 2022-02-08 10:56:17,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 10:56:17,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 10:56:17,882 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['106038 memcached\n'], []) 2022-02-08 10:56:17,882 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:56:17,884 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:56:18,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:18,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:56:18,299 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:18,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:56:18,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:56:18,331 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:56:18,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:56:18,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:56:18,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:56:18,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:56:18,408 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:56:18,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 10:56:18,446 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['89650 memcached\n'], []) 2022-02-08 10:56:18,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 10:56:18,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 10:56:18,497 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['89650 memcached\n'], []) 2022-02-08 10:56:18,499 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:56:18,500 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:56:18,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:18,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:56:18,891 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:18,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:56:18,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:56:18,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:56:18,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:56:18,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:56:18,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:56:18,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:56:18,990 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:56:18,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 10:56:19,042 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['117000 memcached\n'], []) 2022-02-08 10:56:19,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 10:56:19,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 10:56:19,092 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['117000 memcached\n'], []) 2022-02-08 10:56:19,092 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:56:19,094 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:56:19,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:19,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:56:19,448 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:19,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:56:19,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:56:19,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:56:19,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:56:19,510 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:56:19,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:56:19,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:56:19,552 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:56:19,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 10:56:19,599 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['110871 memcached\n'], []) 2022-02-08 10:56:19,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 10:56:19,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 10:56:19,647 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['110871 memcached\n'], []) 2022-02-08 10:56:19,647 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:56:19,648 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:56:19,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:20,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:56:20,253 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:20,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:56:20,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:56:20,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:56:20,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:56:20,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:56:20,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:56:20,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:56:20,506 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:56:20,507 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 10:56:20,579 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['15272 memcached\n'], []) 2022-02-08 10:56:20,581 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 10:56:20,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 10:56:20,635 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['15272 memcached\n'], []) 2022-02-08 10:56:20,635 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:56:20,638 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:56:20,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:20,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:56:20,986 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:20,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:56:21,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:56:21,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:56:21,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:56:21,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:56:21,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:56:21,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:56:21,081 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:56:21,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 10:56:21,130 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['79736 memcached\n'], []) 2022-02-08 10:56:21,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 10:56:21,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 10:56:21,178 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['79736 memcached\n'], []) 2022-02-08 10:56:21,178 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:56:21,181 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:56:21,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:21,497 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:56:21,559 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:21,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:56:21,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:56:21,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:56:21,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:56:21,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:56:21,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:56:21,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:56:21,654 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:56:21,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 10:56:21,704 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['94919 memcached\n'], []) 2022-02-08 10:56:21,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 10:56:21,729 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 10:56:21,753 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['94919 memcached\n'], []) 2022-02-08 10:56:21,753 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:56:21,756 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:56:21,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:22,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:56:22,142 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:22,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:56:22,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:56:22,173 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:56:22,186 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:56:22,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:56:22,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:56:22,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:56:22,243 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:56:22,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 10:56:22,286 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['30794 memcached\n'], []) 2022-02-08 10:56:22,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 10:56:22,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 10:56:22,334 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['30794 memcached\n'], []) 2022-02-08 10:56:22,335 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:56:22,336 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:56:22,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:22,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:56:22,710 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:22,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:56:22,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:56:22,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:56:22,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:56:22,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:56:22,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:56:22,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:56:22,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:56:22,834 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 10:56:22,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:56:22,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:23,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:56:24,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:25,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:25,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:25,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:56:25,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:56:25,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:56:25,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:56:25,882 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 10:56:25,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:56:25,897 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:25,966 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:56:25,980 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 10:56:25,982 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:56:25,983 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:56:26,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:26,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:56:26,346 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:26,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:56:26,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:56:26,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:56:26,403 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:56:26,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:56:26,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:56:26,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:56:26,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:56:26,486 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 10:56:26,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:56:26,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:29,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:56:30,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:30,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:31,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:31,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:56:31,255 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:56:31,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:56:31,308 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:56:31,322 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 10:56:31,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:56:31,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:31,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:56:31,427 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 10:56:31,427 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:56:31,430 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 10:56:31,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:31,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 10:56:31,822 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:31,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 10:56:31,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 10:56:31,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 10:56:31,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 10:56:31,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 10:56:31,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 10:56:31,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 10:56:31,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:56:31,944 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 10:56:31,971 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:56:31,989 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:32,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:56:32,022 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:32,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:32,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:56:32,065 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 10:56:32,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:56:32,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:32,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:56:32,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:32,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:56:32,186 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:32,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:32,257 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:32,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:32,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:32,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:32,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:32,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:32,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:32,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:56:32,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:32,540 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:32,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:32,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:32,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:32,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:32,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:32,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:32,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:32,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:56:32,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:32,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:32,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:32,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:32,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:32,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:32,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:32,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:32,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:32,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:56:32,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:32,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:32,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:32,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:32,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:32,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:32,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:33,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:33,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:33,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:56:33,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:33,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:33,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:33,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:33,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:33,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:33,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:33,191 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:33,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:33,224 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:56:33,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:33,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:33,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:33,290 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:33,308 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:33,323 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:33,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:33,357 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:33,374 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:33,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:56:33,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:33,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:33,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:33,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:33,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:33,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:33,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:33,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:33,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:33,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:56:33,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:33,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:33,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:33,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:33,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:33,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:33,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:33,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:33,720 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 10:56:33,723 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:56:33,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:34,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:56:35,038 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:35,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:56:35,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:56:35,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:56:35,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:56:35,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:56:35,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:56:35,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:56:36,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:56:36,221 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 10:56:36,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:56:36,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:41,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:56:41,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:42,043 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:42,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:56:42,346 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 10:56:42,346 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:56:42,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:42,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:56:42,434 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 10:56:42,434 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:56:42,437 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 10:56:42,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:42,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 10:56:42,809 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:42,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 10:56:42,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 10:56:42,834 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 10:56:42,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 10:56:42,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 10:56:42,874 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 10:56:42,887 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 10:56:42,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:56:42,914 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 10:56:42,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:56:42,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:42,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:56:42,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:43,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:43,017 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:56:43,030 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 10:56:43,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:56:43,043 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:43,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:56:43,071 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:43,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:56:43,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:43,173 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:43,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:43,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:43,280 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:43,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:43,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:43,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:43,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:43,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:56:43,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:43,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:43,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:43,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:43,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:43,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:43,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:43,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:43,579 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:43,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:56:43,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:43,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:43,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:43,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:43,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:43,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:43,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:43,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:43,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:43,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:56:43,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:43,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:43,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:43,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:43,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:43,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:43,900 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:43,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:43,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:43,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:56:43,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:43,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:43,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:44,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:44,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:44,038 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:44,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:44,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:44,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:44,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:56:44,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:44,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:44,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:44,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:44,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:44,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:44,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:44,229 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:44,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:44,257 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:56:44,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:44,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:44,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:44,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:44,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:44,346 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:44,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:44,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:44,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:44,410 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:56:44,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:44,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:44,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:44,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:44,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:44,510 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:44,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:44,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:44,562 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 10:56:44,565 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:56:44,668 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:44,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:56:44,979 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:44,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:56:44,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:56:45,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:56:45,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:56:45,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:56:45,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:56:45,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:56:45,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:56:45,109 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 10:56:45,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:56:45,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:47,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:56:48,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:49,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:49,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 14086 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:56:50,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:56:50,132 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:56:50,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:56:50,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:56:50,278 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 10:56:50,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:56:50,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:50,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:56:50,368 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 10:56:50,368 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:56:50,371 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:56:50,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:50,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:56:50,773 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:50,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:56:50,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:56:50,808 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:56:50,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:56:50,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:56:50,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:56:50,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:56:50,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:56:50,910 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 10:56:50,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:56:50,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:52,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:56:53,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:53,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:54,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:54,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:56:54,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:56:54,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:56:54,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:56:54,484 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 10:56:54,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:56:54,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:54,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:56:54,584 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 10:56:54,584 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:56:54,586 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 10:56:54,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:54,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 10:56:54,960 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:54,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 10:56:54,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 10:56:54,987 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 10:56:55,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 10:56:55,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 10:56:55,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 10:56:55,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 10:56:55,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:56:55,098 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 10:56:55,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:56:55,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:55,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:56:55,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:55,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:55,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:56:55,219 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 10:56:55,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:56:55,232 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:55,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:56:55,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:55,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:56:55,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:55,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:55,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:55,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:55,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:55,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:55,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:55,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:55,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:55,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:56:55,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:55,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:55,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:55,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:55,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:55,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:55,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:55,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:55,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:55,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:56:55,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:55,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:55,839 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:55,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:55,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:55,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:55,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:55,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:55,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:55,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:56:55,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:55,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:56,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:56,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:56,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:56,061 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:56,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:56,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:56,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:56,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:56:56,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:56,173 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:56,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:56,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:56,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:56,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:56,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:56,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:56,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:56,306 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:56:56,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:56,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:56,357 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:56,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:56,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:56,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:56,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:56,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:56,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:56,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:56:56,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:56,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:56,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:56,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:56,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:56,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:56,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:56,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:56,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:56,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:56:56,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:56,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:56,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:56,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:56,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:56,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:56,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:56,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:56,809 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 10:56:56,811 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 10:56:56,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:57,102 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 10:56:57,177 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:57,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 10:56:57,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 10:56:57,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 10:56:57,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 10:56:57,232 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 10:56:57,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 10:56:57,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 10:56:57,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:56:57,282 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 10:56:57,313 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:56:57,326 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:57,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:56:57,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:57,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:56:57,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:56:57,388 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 10:56:57,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:56:57,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:56:57,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:56:57,426 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:57,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:56:57,494 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:57,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:57,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:57,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:57,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:57,660 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:57,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:57,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:56:57,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:57,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:56:57,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:57,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:57,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:57,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:57,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:57,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:57,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:57,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:56:57,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:57,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:56:57,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:57,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:57,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:57,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:58,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:58,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:58,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:58,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:56:58,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:58,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:56:58,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:58,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:58,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:58,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:58,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:58,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:58,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:58,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:56:58,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:58,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:56:58,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:58,290 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:58,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:58,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:58,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:58,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:58,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:58,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:56:58,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:58,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:56:58,428 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:58,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:58,459 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:58,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:58,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:58,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:58,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:58,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:56:58,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:58,556 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:56:58,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:58,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:58,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:58,614 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:58,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:58,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:58,655 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:58,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:56:58,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:58,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:56:58,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:58,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:58,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:58,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:58,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:58,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:58,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:58,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:56:58,835 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 10:56:58,836 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:56:58,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:56:59,147 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:56:59,210 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:56:59,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:56:59,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:56:59,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:56:59,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:56:59,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:56:59,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:56:59,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:56:59,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:56:59,334 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 10:56:59,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:56:59,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:57:01,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:57:02,170 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:57:02,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:57:03,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:57:03,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:57:03,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:57:03,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:57:03,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:57:03,295 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 10:57:03,295 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:57:03,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:57:03,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:57:03,375 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 10:57:03,377 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:57:03,380 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:57:03,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:57:03,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:57:03,769 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:57:03,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:57:03,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:57:03,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:57:03,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:57:03,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:57:03,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:57:03,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:57:03,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:57:03,888 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 10:57:03,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:57:03,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:57:09,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:57:09,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:57:10,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:57:10,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:57:10,733 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 10:57:10,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:57:10,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:57:10,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:57:10,815 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 10:57:10,816 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:57:10,819 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:57:11,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:57:11,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:57:11,657 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:57:11,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:57:11,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:57:11,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:57:11,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:57:11,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:57:11,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:57:11,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:57:11,778 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:57:11,792 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 10:57:11,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:57:11,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:57:14,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:57:14,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:57:15,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:57:15,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:57:15,319 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 10:57:15,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:57:15,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:57:15,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:57:15,394 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 10:57:15,394 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:57:15,397 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:57:15,474 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:57:15,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:57:15,753 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:57:15,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:57:15,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:57:15,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:57:15,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:57:15,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:57:15,834 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:57:15,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:57:15,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:57:15,887 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 10:57:15,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:57:15,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:57:17,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:57:18,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:57:19,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:57:19,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:57:19,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:57:19,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:57:19,989 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:57:20,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:57:20,089 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 10:57:20,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:57:20,102 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:57:20,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:57:20,137 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 10:57:20,138 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:57:20,141 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 10:57:20,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:57:20,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 10:57:20,528 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:57:20,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 10:57:20,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 10:57:20,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 10:57:20,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 10:57:20,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 10:57:20,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 10:57:20,607 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 10:57:20,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:57:20,634 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 10:57:20,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:57:20,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:57:20,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:57:20,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:57:20,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:57:20,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:57:20,749 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 10:57:20,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:57:20,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:57:20,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:57:20,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:57:20,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:57:20,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:57:20,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:57:20,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:57:20,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:57:20,987 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:57:21,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:57:21,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:57:21,086 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:57:21,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:57:21,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:57:21,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:57:21,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:57:21,186 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:57:21,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:57:21,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:57:21,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:57:21,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:57:21,265 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:57:21,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:57:21,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:57:21,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:57:21,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:57:21,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:57:21,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:57:21,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:57:21,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:57:21,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:57:21,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:57:21,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:57:21,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:57:21,469 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:57:21,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:57:21,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:57:21,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:57:21,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:57:21,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:57:21,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:57:21,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:57:21,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:57:21,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:57:21,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:57:21,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:57:21,647 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:57:21,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:57:21,676 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:57:21,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:57:21,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:57:21,721 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:57:21,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:57:21,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:57:21,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:57:21,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:57:21,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:57:21,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:57:21,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:57:21,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:57:21,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:57:21,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:57:21,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:57:21,900 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:57:21,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:57:21,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:57:21,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:57:21,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:57:21,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:57:21,989 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:57:22,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:57:22,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:57:22,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:57:22,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:57:22,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:57:22,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:57:22,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:57:22,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:57:22,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:57:22,160 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:57:22,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:57:22,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:57:22,207 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 10:57:22,207 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:57:22,210 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:57:22,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:57:22,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:57:22,539 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:57:22,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:57:22,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:57:22,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:57:22,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:57:22,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:57:22,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:57:22,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:57:22,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:23,056 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:57:23,056 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.848999977112. 2022-02-08 10:57:23,056 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:57:23,059 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:57:23,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:57:23,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:57:23,855 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:57:23,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:57:23,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:57:23,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:57:23,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:57:23,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:57:23,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:57:23,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:57:23,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:24,180 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:57:24,181 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 10:57:26,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:26,325 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:57:26,325 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 10:57:28,326 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:28,474 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:57:28,476 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 10:57:30,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:30,627 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:57:30,628 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 10:57:32,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:32,778 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:57:32,779 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 10:57:34,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:34,930 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:57:34,930 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 10:57:36,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:37,069 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:57:37,069 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 10:57:39,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:39,209 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:57:39,209 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 10:57:41,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:41,351 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:57:41,351 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 10:57:43,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:43,490 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:57:43,492 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 10:57:45,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:45,628 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:57:45,628 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 10:57:47,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:47,767 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 10:57:47,767 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 10:57:49,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 14096 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:57:49,911 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:57:49,911 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 26.8540000916. 2022-02-08 10:57:49,911 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:57:49,914 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:57:50,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:57:50,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:57:50,308 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:57:50,308 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:57:50,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:57:50,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:57:50,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:57:50,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:57:50,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:57:50,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:57:50,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:50,635 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:57:50,637 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.725000143051. 2022-02-08 10:57:50,637 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:57:50,638 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:57:51,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:57:51,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:57:51,595 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:57:51,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:57:51,614 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:57:51,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:57:51,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:57:51,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:57:51,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:57:51,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:57:51,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:52,219 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:57:52,220 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.58299994469. 2022-02-08 10:57:52,220 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:57:52,223 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:57:52,910 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:57:53,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:57:53,233 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:57:53,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:57:53,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:57:53,265 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:57:53,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:57:53,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:57:53,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:57:53,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:57:53,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:53,608 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:57:53,609 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.38999986649. 2022-02-08 10:57:53,609 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:57:53,611 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:57:54,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:57:54,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:57:54,496 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:57:54,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:57:54,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:57:54,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:57:54,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:57:54,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:57:54,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:57:54,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:57:54,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:55,163 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:57:55,164 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.55300021172. 2022-02-08 10:57:55,164 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:57:55,165 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:57:55,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:57:56,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:57:56,111 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:57:56,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:57:56,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:57:56,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:57:56,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:57:56,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:57:56,186 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:57:56,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:57:56,227 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:56,674 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:57:56,683 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.51900005341. 2022-02-08 10:57:56,684 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:57:56,687 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:57:56,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:57:57,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:57:57,315 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:57:57,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:57:57,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:57:57,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:57:57,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:57:57,403 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:57:57,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:57:57,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:57:57,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:57,720 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:57:57,721 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.03799986839. 2022-02-08 10:57:57,721 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 10:57:57,723 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:57:58,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:57:58,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:57:58,809 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:57:58,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:57:58,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:57:58,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:57:58,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:57:58,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:57:58,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:57:58,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:57:58,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 10:57:59,160 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:57:59,161 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.4390001297. ############################################################ # # RAM FootPrint: 4144.07421875 # ############################################################ 2022-02-08 10:57:59,211 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 10:57:59,213 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 64 seconds. Reason: Iteration:5 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14118 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:59:03,219 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 10:59:03,220 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 10:59:08,223 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:59:08,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:08,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:59:08,680 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:08,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:59:08,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:59:08,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:59:08,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:59:08,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:59:08,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:59:08,795 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:59:08,832 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:59:08,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 10:59:08,926 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['53638 memcached\n'], []) 2022-02-08 10:59:08,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 10:59:08,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 10:59:08,971 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['53638 memcached\n'], []) 2022-02-08 10:59:08,971 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:59:08,974 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:59:09,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:09,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:59:09,421 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:09,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:59:09,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:59:09,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:59:09,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:59:09,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:59:09,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:59:09,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:59:09,546 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:59:09,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 10:59:09,607 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['106827 memcached\n'], []) 2022-02-08 10:59:09,607 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 10:59:09,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 10:59:09,665 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['106827 memcached\n'], []) 2022-02-08 10:59:09,665 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:59:09,668 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:59:09,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:09,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:59:10,053 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:10,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:59:10,071 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:59:10,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:59:10,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:59:10,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:59:10,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:59:10,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:59:10,171 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:59:10,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 10:59:10,213 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['90533 memcached\n'], []) 2022-02-08 10:59:10,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 10:59:10,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 10:59:10,272 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['90533 memcached\n'], []) 2022-02-08 10:59:10,273 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:59:10,276 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:59:10,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:10,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:59:10,653 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:10,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:59:10,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:59:10,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:59:10,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:59:10,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:59:10,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:59:10,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:59:10,753 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:59:10,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 10:59:10,857 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['117698 memcached\n'], []) 2022-02-08 10:59:10,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 10:59:10,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 10:59:10,911 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['117698 memcached\n'], []) 2022-02-08 10:59:10,911 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:59:10,914 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:59:10,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:11,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:59:11,280 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:11,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:59:11,295 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:59:11,308 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:59:11,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:59:11,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:59:11,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:59:11,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:59:11,378 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:59:11,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 10:59:11,430 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['111596 memcached\n'], []) 2022-02-08 10:59:11,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 10:59:11,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 10:59:11,476 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['111596 memcached\n'], []) 2022-02-08 10:59:11,476 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:59:11,477 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:59:11,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:11,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:59:11,851 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:11,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:59:11,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:59:11,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:59:11,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:59:11,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:59:11,987 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:59:12,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:59:12,013 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:59:12,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 10:59:12,101 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['16000 memcached\n'], []) 2022-02-08 10:59:12,102 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 10:59:12,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 10:59:12,157 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['16000 memcached\n'], []) 2022-02-08 10:59:12,157 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 10:59:12,158 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:59:12,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:12,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:59:12,529 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:12,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:59:12,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:59:12,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:59:12,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:59:12,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:59:12,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:59:12,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:59:12,657 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:59:12,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 10:59:12,706 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['80391 memcached\n'], []) 2022-02-08 10:59:12,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 10:59:12,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 10:59:12,752 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['80391 memcached\n'], []) 2022-02-08 10:59:12,752 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:59:12,755 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:59:13,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:13,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:59:13,549 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:13,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:59:13,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:59:13,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:59:13,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:59:13,604 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:59:13,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:59:13,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:59:13,641 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:59:13,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 10:59:13,710 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['95598 memcached\n'], []) 2022-02-08 10:59:13,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 10:59:13,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 10:59:13,762 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['95598 memcached\n'], []) 2022-02-08 10:59:13,763 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:59:13,765 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:59:13,829 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:14,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:59:14,118 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:14,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:59:14,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:59:14,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:59:14,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:59:14,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:59:14,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:59:14,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:59:14,221 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 10:59:14,221 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 10:59:14,269 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['31519 memcached\n'], []) 2022-02-08 10:59:14,270 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 10:59:14,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 10:59:14,331 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['31519 memcached\n'], []) 2022-02-08 10:59:14,332 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:59:14,341 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 10:59:14,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:14,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 10:59:14,742 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:14,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 10:59:14,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 10:59:14,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 10:59:14,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 10:59:14,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 10:59:14,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 10:59:14,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 10:59:14,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:59:14,861 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 10:59:14,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:59:14,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:20,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:59:21,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:22,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:22,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:23,198 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:59:23,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:59:23,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:59:23,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:59:23,408 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 10:59:23,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:59:23,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:23,487 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:59:23,510 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 10:59:23,510 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 10:59:23,513 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 10:59:23,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:23,808 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 10:59:23,878 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:23,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 10:59:23,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 10:59:23,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 10:59:23,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 10:59:23,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 10:59:23,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 10:59:23,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 10:59:23,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:59:23,992 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 10:59:24,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:59:24,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:25,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:59:26,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:27,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:27,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:27,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:59:27,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:59:27,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:59:27,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:59:27,869 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 10:59:27,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:59:27,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:27,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:59:27,966 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 10:59:27,966 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 10:59:27,969 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 10:59:28,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:28,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 10:59:28,371 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:28,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 10:59:28,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 10:59:28,403 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 10:59:28,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 10:59:28,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 10:59:28,454 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 10:59:28,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 10:59:28,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:59:28,509 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 10:59:28,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:59:28,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:28,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:59:28,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:28,596 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:28,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:59:28,625 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 10:59:28,627 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:59:28,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:28,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:59:28,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:28,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:59:28,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:28,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:28,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:28,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:28,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:28,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:28,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:28,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:29,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:29,043 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:59:29,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:29,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:29,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:29,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:29,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:29,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:29,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:29,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:29,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:29,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:59:29,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:29,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:29,278 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:29,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:29,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:29,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:29,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:29,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:29,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:29,413 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:59:29,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:29,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:29,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:29,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:29,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:29,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:29,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:29,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:29,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:29,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:59:29,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:29,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:29,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:29,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:29,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:29,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:29,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:29,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:29,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:29,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:59:29,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:29,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:29,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:29,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:29,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:29,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:29,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:29,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:29,910 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:29,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:59:29,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:29,971 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:29,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:30,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:30,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:30,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:30,056 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:30,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:30,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:30,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:59:30,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:30,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:30,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:30,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:30,191 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:30,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:30,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:30,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:30,266 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 10:59:30,269 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 10:59:30,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:30,887 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 10:59:31,477 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:31,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 10:59:31,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 10:59:31,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 10:59:31,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 10:59:31,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 10:59:31,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 10:59:31,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 10:59:31,982 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:59:32,111 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 10:59:32,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:59:32,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:33,096 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:59:34,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:34,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:35,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:59:35,112 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 10:59:35,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:59:35,127 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:35,198 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:59:35,226 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 10:59:35,227 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 10:59:35,230 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 10:59:35,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:35,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 10:59:35,585 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:35,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 10:59:35,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 10:59:35,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 10:59:35,624 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 10:59:35,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 10:59:35,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 10:59:35,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 10:59:35,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:59:35,694 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 10:59:35,721 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:59:35,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:35,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:59:35,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:35,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:35,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:59:35,802 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 10:59:35,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:59:35,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:35,829 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:59:35,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:35,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:59:35,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:35,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:35,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:36,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:36,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:36,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:36,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:36,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:36,186 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:36,201 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:59:36,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:36,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:36,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:36,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:36,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:36,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:36,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:36,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:36,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:36,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:59:36,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:36,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:36,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:36,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:36,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:36,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:36,482 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:36,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:36,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:36,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:59:36,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:36,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:36,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:36,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:36,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:36,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:36,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:36,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:36,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:36,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:59:36,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:36,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:36,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:36,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:36,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:36,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:36,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:36,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:36,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:36,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:59:36,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:36,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:36,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:36,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:36,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:36,956 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:36,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:36,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:37,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:37,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:59:37,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:37,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:37,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:37,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:37,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:37,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:37,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:37,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:37,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:37,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:59:37,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:37,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:37,227 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:37,244 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:37,260 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:37,280 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:37,308 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:37,326 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:37,342 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 10:59:37,345 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 10:59:37,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:37,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 10:59:37,700 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:37,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 10:59:37,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 10:59:37,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 10:59:37,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 10:59:37,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 10:59:37,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 10:59:37,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 10:59:37,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:59:37,809 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 10:59:37,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:59:37,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:38,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:59:39,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:39,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:40,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:40,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:59:40,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:59:40,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:59:40,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:59:40,711 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 10:59:40,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:59:40,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:40,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:59:40,790 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 10:59:40,790 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 10:59:40,793 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 10:59:40,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:41,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 10:59:41,160 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:41,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 10:59:41,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 10:59:41,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 10:59:41,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 10:59:41,224 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 10:59:41,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 10:59:41,255 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 10:59:41,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:59:41,286 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 10:59:41,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:59:41,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:43,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:59:43,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:44,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:44,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:44,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:59:44,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:59:44,956 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:59:44,987 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:59:45,003 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 10:59:45,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:59:45,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:45,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:59:45,118 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 10:59:45,118 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 10:59:45,121 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 10:59:45,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:45,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 10:59:45,500 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:45,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 10:59:45,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 10:59:45,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 10:59:45,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 10:59:45,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 10:59:45,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 10:59:45,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 10:59:45,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:59:45,615 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 10:59:45,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:59:45,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:45,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:59:45,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:45,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:45,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:59:45,726 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 10:59:45,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:59:45,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:45,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:59:45,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:45,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:59:45,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:45,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:45,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:45,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:45,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:46,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:46,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:46,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:46,096 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:46,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:59:46,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:46,147 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:46,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:46,178 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:46,196 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:46,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:46,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:46,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:46,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:46,280 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:59:46,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:46,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:46,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:46,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:46,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:46,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:46,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:46,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:46,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:46,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:59:46,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:46,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:46,507 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:46,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:46,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:46,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:46,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:46,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:46,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:46,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:59:46,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:46,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:46,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:46,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:46,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:46,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:46,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:46,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:46,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:46,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:59:46,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:46,834 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:46,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:46,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:46,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:46,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:46,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:46,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:46,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:46,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:59:46,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:47,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:47,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:47,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:47,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:47,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:47,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:47,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:47,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:47,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:59:47,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:47,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:47,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:47,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:47,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:47,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:47,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:47,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:47,283 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 10:59:47,286 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 10:59:47,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:47,607 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 10:59:47,671 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:47,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 10:59:47,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 10:59:47,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 10:59:47,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 10:59:47,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 10:59:47,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 10:59:47,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 10:59:47,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:59:47,790 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 10:59:47,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:59:47,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:47,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:59:47,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:47,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:47,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:59:47,904 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 10:59:47,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:59:47,918 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:47,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:59:47,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:47,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 10:59:48,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:48,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:48,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:48,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:48,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:48,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:48,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:48,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 10:59:48,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:48,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 10:59:48,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:48,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:48,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:48,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:48,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:48,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:48,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:48,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 10:59:48,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:48,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 10:59:48,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:48,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:48,573 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:48,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:48,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:48,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:48,640 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:48,655 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 10:59:48,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:48,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 10:59:48,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:48,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:48,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:48,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:48,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:48,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:48,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:48,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 10:59:48,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:48,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 10:59:48,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:48,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:48,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:48,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:48,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:48,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:48,971 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:48,987 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 10:59:49,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:49,017 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 10:59:49,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:49,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:49,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:49,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:49,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:49,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:49,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:49,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 10:59:49,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:49,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 10:59:49,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:49,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:49,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:49,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:49,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:49,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:49,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:49,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 10:59:49,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:49,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 10:59:49,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:49,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:49,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:49,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:49,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:49,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:49,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:49,474 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 10:59:49,492 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 10:59:49,494 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 10:59:49,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 14132 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 10:59:49,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 10:59:49,882 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:49,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 10:59:49,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 10:59:49,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 10:59:49,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 10:59:49,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 10:59:49,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 10:59:49,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 10:59:50,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:59:50,026 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 10:59:50,071 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:59:50,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:50,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:59:51,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:52,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:52,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:52,729 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:59:52,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:59:52,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 10:59:52,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:59:52,839 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 10:59:52,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:59:52,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:52,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:59:52,924 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 10:59:52,924 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 10:59:52,926 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 10:59:53,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:53,255 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 10:59:53,315 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:53,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 10:59:53,329 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 10:59:53,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 10:59:53,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 10:59:53,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 10:59:53,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 10:59:53,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 10:59:53,413 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:59:53,428 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 10:59:53,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:59:53,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:54,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:59:54,655 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:55,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:55,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:59:55,437 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 10:59:55,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:59:55,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:55,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:59:55,483 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 10:59:55,483 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 10:59:55,486 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 10:59:56,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:56,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 10:59:56,497 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:56,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 10:59:56,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 10:59:56,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 10:59:56,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 10:59:56,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 10:59:56,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 10:59:56,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 10:59:56,647 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:59:56,663 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 10:59:56,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:59:56,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:57,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 10:59:57,969 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:58,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 10:59:58,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 10:59:58,660 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 10:59:58,660 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 10:59:58,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 10:59:58,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 10:59:58,740 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 10:59:58,740 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 10:59:58,743 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 10:59:58,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 10:59:59,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 10:59:59,148 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 10:59:59,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 10:59:59,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 10:59:59,178 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 10:59:59,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 10:59:59,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 10:59:59,229 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 10:59:59,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 10:59:59,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 10:59:59,278 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 10:59:59,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 10:59:59,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:00:00,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:00:00,938 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:00:01,446 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:00:01,897 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:00:04,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:00:04,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:00:04,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:00:04,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:00:04,430 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 11:00:04,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:00:04,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:00:04,469 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:00:04,490 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 11:00:04,490 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:00:04,494 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 11:00:04,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:00:04,829 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 11:00:04,897 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:00:04,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 11:00:04,913 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 11:00:04,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 11:00:04,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 11:00:04,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 11:00:04,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 11:00:04,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 11:00:05,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:00:05,026 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 11:00:05,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:00:05,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:00:05,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:00:05,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:00:05,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:00:05,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:00:05,153 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 11:00:05,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:00:05,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:00:05,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:00:05,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:00:05,227 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:00:05,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:00:05,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:00:05,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:00:05,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:00:05,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:00:05,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:00:05,497 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:00:05,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:00:05,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:00:05,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:00:05,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:00:05,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:00:05,640 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:00:05,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:00:05,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:00:05,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:00:05,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:00:05,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:00:05,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:00:05,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:00:05,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:00:05,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:00:05,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:00:05,839 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:00:05,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:00:05,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:00:05,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:00:05,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:00:05,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:00:05,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:00:05,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:00:05,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:00:05,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:00:06,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:00:06,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:00:06,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:00:06,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:00:06,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:00:06,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:00:06,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:00:06,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:00:06,170 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:00:06,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:00:06,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:00:06,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:00:06,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:00:06,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:00:06,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:00:06,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:00:06,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:00:06,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:00:06,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:00:06,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:00:06,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:00:06,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:00:06,410 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:00:06,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:00:06,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:00:06,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:00:06,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:00:06,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:00:06,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:00:06,535 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:00:06,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:00:06,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:00:06,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:00:06,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:00:06,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:00:06,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:00:06,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:00:06,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:00:06,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:00:06,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:00:06,729 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:00:06,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:00:06,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:00:06,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:00:06,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:00:06,836 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 11:00:06,838 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:00:06,839 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:00:06,915 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:00:07,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:00:07,230 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:00:07,232 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:00:07,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:00:07,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:00:07,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:00:07,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:00:07,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:00:07,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:00:07,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:07,848 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:00:07,851 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.01300001144. 2022-02-08 11:00:07,851 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:00:07,852 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:00:07,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:00:08,191 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:00:08,257 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:00:08,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:00:08,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:00:08,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:00:08,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:00:08,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:00:08,331 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:00:08,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:00:08,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:08,505 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:08,506 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:10,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:10,655 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:10,655 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:12,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:12,806 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:12,806 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:14,808 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:14,963 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:14,963 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:16,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:17,111 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:17,112 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:19,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:19,352 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:19,354 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:21,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:21,559 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:21,559 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:23,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:23,707 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:23,707 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:25,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:25,861 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:25,861 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:27,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:28,026 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:28,026 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:30,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:30,214 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:30,216 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:32,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:32,380 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:32,381 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:34,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:34,559 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:34,559 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:36,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:36,799 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:36,799 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:38,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:38,960 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:38,961 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:40,961 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:41,147 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:41,147 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:43,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:43,335 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:43,335 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:45,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:45,510 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:45,512 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:47,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:47,676 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:47,676 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:49,676 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 14136 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:00:49,842 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:49,842 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:51,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:52,059 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:52,059 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:54,061 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:54,242 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:54,242 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:56,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:56,407 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:56,407 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:00:58,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:00:58,638 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:00:58,638 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:01:00,640 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:01:00,926 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:01:00,926 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:01:02,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:01:03,111 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:01:03,111 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 55.2599999905. 2022-02-08 11:01:03,111 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:01:03,114 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:01:03,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:01:04,002 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:01:04,131 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:01:04,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:01:04,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:01:04,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:01:04,196 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:01:04,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:01:04,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:01:04,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:01:04,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:01:04,582 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:01:04,582 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.47099995613. 2022-02-08 11:01:04,582 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:01:04,585 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:01:05,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:01:05,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:01:05,608 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:01:05,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:01:05,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:01:05,839 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:01:05,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:01:05,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:01:05,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:01:05,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:01:06,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:01:06,740 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:01:06,740 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.126 2022-02-08 11:01:08,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:01:08,934 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:01:08,934 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.35199999809. 2022-02-08 11:01:08,934 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:01:08,937 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:01:09,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:01:09,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:01:10,033 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:01:10,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:01:10,056 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:01:10,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:01:10,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:01:10,198 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:01:10,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:01:10,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:01:10,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:01:10,730 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:01:10,730 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.79600000381. 2022-02-08 11:01:10,730 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:01:10,733 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:01:11,505 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:01:11,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:01:11,997 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:01:11,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:01:12,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:01:12,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:01:12,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:01:12,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:01:12,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:01:12,298 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:01:12,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:01:12,776 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:01:12,776 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 11:01:14,778 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:01:14,947 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:01:14,947 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 11:01:16,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:01:17,170 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:01:17,171 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.44000005722. 2022-02-08 11:01:17,171 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:01:17,173 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:01:17,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:01:17,923 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:01:18,045 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:01:18,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:01:18,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:01:18,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:01:18,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:01:18,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:01:18,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:01:18,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:01:18,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:01:18,595 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:01:18,596 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.42599987984. 2022-02-08 11:01:18,596 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:01:18,598 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:01:18,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:01:19,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:01:19,414 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:01:19,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:01:19,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:01:19,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:01:19,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:01:19,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:01:19,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:01:19,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:01:19,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:01:20,168 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:01:20,170 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.57300019264. 2022-02-08 11:01:20,170 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:01:20,171 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:01:20,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:01:20,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:01:21,174 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:01:21,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:01:21,224 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:01:21,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:01:21,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:01:21,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:01:21,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:01:21,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:01:21,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:01:21,785 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:01:21,785 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 11:01:23,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:01:24,516 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:01:24,517 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 11:01:26,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:01:26,967 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:01:26,967 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 11:01:28,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:01:29,286 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:01:29,286 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 11:01:31,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:01:31,473 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:01:31,473 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 11:01:33,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:01:33,614 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:01:33,614 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 13.4440000057. ############################################################ # # RAM FootPrint: 4705.10546875 # ############################################################ 2022-02-08 11:01:33,661 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 11:01:33,661 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 113 seconds. Reason: Iteration:6 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14146 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 14171 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:03:26,667 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 11:03:26,667 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 11:03:31,671 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:03:31,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:03:32,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:03:32,160 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:03:32,160 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:03:32,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:03:32,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:03:32,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:03:32,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:03:32,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:03:32,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:03:32,345 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:03:32,346 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:03:32,398 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['54277 memcached\n'], []) 2022-02-08 11:03:32,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 11:03:32,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:03:32,448 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['54277 memcached\n'], []) 2022-02-08 11:03:32,450 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:03:32,453 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:03:32,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:03:33,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:03:33,096 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:03:33,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:03:33,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:03:33,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:03:33,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:03:33,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:03:33,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:03:33,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:03:33,210 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:03:33,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:03:33,263 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['107623 memcached\n'], []) 2022-02-08 11:03:33,265 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 11:03:33,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:03:33,308 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['107623 memcached\n'], []) 2022-02-08 11:03:33,308 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:03:33,309 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:03:33,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:03:33,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:03:33,721 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:03:33,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:03:33,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:03:33,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:03:33,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:03:33,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:03:33,816 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:03:33,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:03:33,848 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:03:33,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:03:33,891 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['91286 memcached\n'], []) 2022-02-08 11:03:33,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 11:03:33,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:03:33,940 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['91286 memcached\n'], []) 2022-02-08 11:03:33,940 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:03:33,941 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:03:34,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:03:34,257 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:03:34,322 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:03:34,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:03:34,336 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:03:34,349 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:03:34,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:03:34,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:03:34,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:03:34,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:03:34,421 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:03:34,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:03:34,467 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['118351 memcached\n'], []) 2022-02-08 11:03:34,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 11:03:34,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:03:34,522 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['118351 memcached\n'], []) 2022-02-08 11:03:34,522 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:03:34,525 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:03:34,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:03:34,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:03:34,884 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:03:34,885 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:03:34,900 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:03:34,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:03:34,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:03:34,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:03:34,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:03:34,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:03:34,980 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:03:34,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:03:35,036 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['112218 memcached\n'], []) 2022-02-08 11:03:35,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 11:03:35,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:03:35,079 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['112218 memcached\n'], []) 2022-02-08 11:03:35,079 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:03:35,082 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:03:35,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:03:35,482 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:03:35,548 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:03:35,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:03:35,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:03:35,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:03:35,594 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:03:35,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:03:35,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:03:35,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:03:35,647 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:03:35,647 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:03:35,698 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['16652 memcached\n'], []) 2022-02-08 11:03:35,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 11:03:35,721 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:03:35,742 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['16652 memcached\n'], []) 2022-02-08 11:03:35,742 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:03:35,743 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:03:35,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:03:36,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:03:36,098 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:03:36,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:03:36,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:03:36,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:03:36,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:03:36,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:03:36,163 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:03:36,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:03:36,187 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:03:36,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:03:36,230 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['81027 memcached\n'], []) 2022-02-08 11:03:36,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 11:03:36,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:03:36,273 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['81027 memcached\n'], []) 2022-02-08 11:03:36,273 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:03:36,276 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:03:36,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:03:36,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:03:36,707 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:03:36,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:03:36,721 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:03:36,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:03:36,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:03:36,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:03:36,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:03:36,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:03:36,806 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:03:36,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:03:36,872 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['96222 memcached\n'], []) 2022-02-08 11:03:36,874 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 11:03:36,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:03:36,921 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['96222 memcached\n'], []) 2022-02-08 11:03:36,923 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:03:36,924 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:03:37,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:03:37,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:03:38,148 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:03:38,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:03:38,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:03:38,186 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:03:38,198 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:03:38,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:03:38,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:03:38,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:03:38,275 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:03:38,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:03:38,334 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['32185 memcached\n'], []) 2022-02-08 11:03:38,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 11:03:38,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:03:38,378 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['32185 memcached\n'], []) 2022-02-08 11:03:38,380 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:03:38,381 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:03:38,474 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:03:38,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:03:38,770 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:03:38,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:03:38,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:03:38,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:03:38,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:03:38,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:03:38,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:03:38,885 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:03:38,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:03:38,918 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 11:03:38,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:03:38,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:03:45,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:03:46,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:03:46,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:03:47,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:03:47,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:03:47,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:03:47,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:03:47,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:03:47,938 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 11:03:47,938 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:03:47,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:03:48,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:03:48,042 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 11:03:48,042 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:03:48,043 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:03:48,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:03:48,336 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:03:48,394 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:03:48,395 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:03:48,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:03:48,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:03:48,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:03:48,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:03:48,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:03:48,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:03:48,497 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:03:48,519 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 11:03:48,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:03:48,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 14194 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:03:53,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:03:54,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:03:55,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:03:55,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:03:56,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:03:56,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:03:56,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:03:56,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:03:56,252 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 11:03:56,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:03:56,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:03:56,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:03:56,345 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 11:03:56,346 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:03:56,351 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 11:03:56,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:03:56,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 11:03:56,812 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:03:56,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 11:03:56,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 11:03:56,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 11:03:56,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 11:03:56,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 11:03:56,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 11:03:56,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 11:03:56,923 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:03:56,940 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 11:03:56,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:03:56,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:03:57,002 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:03:57,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:03:57,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:03:57,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:03:57,065 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 11:03:57,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:03:57,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:03:57,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:03:57,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:03:57,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:03:57,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:03:57,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:03:57,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:03:57,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:03:57,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:03:57,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:03:57,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:03:57,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:03:57,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:03:57,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:03:57,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:03:57,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:03:57,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:03:57,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:03:57,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:03:57,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:03:57,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:03:57,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:03:57,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:03:57,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:03:57,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:03:57,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:03:57,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:03:57,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:03:57,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:03:57,816 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:03:57,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:03:57,854 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:03:57,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:03:57,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:03:57,910 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:03:57,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:03:57,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:03:57,961 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:03:57,979 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:03:57,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:03:58,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:03:58,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:03:58,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:03:58,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:03:58,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:03:58,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:03:58,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:03:58,132 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:03:58,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:03:58,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:03:58,183 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:03:58,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:03:58,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:03:58,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:03:58,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:03:58,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:03:58,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:03:58,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:03:58,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:03:58,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:03:58,349 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:03:58,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:03:58,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:03:58,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:03:58,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:03:58,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:03:58,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:03:58,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:03:58,482 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:03:58,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:03:58,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:03:58,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:03:58,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:03:58,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:03:58,581 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:03:58,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:03:58,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:03:58,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:03:58,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:03:58,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:03:58,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:03:58,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:03:58,717 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 11:03:58,719 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:03:58,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:03:59,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:04:00,058 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:04:00,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:04:00,201 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:04:00,357 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:04:00,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:04:00,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:04:00,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:04:00,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:04:00,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:04:00,954 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 11:04:00,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:04:01,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:03,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:04:04,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:04,811 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:05,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:04:05,171 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 11:04:05,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:04:05,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:05,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:04:05,266 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 11:04:05,266 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:04:05,269 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 11:04:05,362 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:04:05,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 11:04:05,703 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:04:05,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 11:04:05,721 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 11:04:05,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 11:04:05,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 11:04:05,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 11:04:05,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 11:04:05,795 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 11:04:05,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:04:05,823 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 11:04:05,854 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:04:05,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:05,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:04:05,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:05,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:05,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:04:05,938 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 11:04:05,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:04:05,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:05,966 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:04:05,979 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:05,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:04:06,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:06,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:06,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:06,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:06,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:06,071 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:06,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:06,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:06,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:06,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:04:06,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:06,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:06,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:06,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:06,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:06,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:06,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:06,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:06,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:06,285 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:04:06,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:06,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:06,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:06,357 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:06,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:06,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:06,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:06,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:06,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:06,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:04:06,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:06,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:06,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:06,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:06,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:06,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:06,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:06,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:06,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:06,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:04:06,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:06,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:06,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:06,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:06,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:06,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:06,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:06,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:06,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:06,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:04:06,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:06,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:06,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:06,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:06,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:06,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:06,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:06,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:06,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:06,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:04:06,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:06,982 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:06,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:07,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:07,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:07,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:07,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:07,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:07,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:07,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:04:07,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:07,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:07,147 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:07,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:07,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:07,191 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:07,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:07,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:07,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:07,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:04:07,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:07,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:07,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:07,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:07,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:07,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:07,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:07,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:07,407 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 11:04:07,411 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:04:07,497 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:04:07,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:04:07,783 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:04:07,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:04:07,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:04:07,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:04:07,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:04:07,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:04:07,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:04:07,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:04:07,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:04:07,903 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 11:04:07,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:04:07,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:11,604 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:04:12,454 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:12,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:13,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:14,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:04:14,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:04:14,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:04:14,658 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:04:14,676 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 11:04:14,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:04:14,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:14,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:04:14,766 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 11:04:14,766 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:04:14,769 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:04:14,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:04:15,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:04:15,201 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:04:15,201 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:04:15,221 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:04:15,237 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:04:15,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:04:15,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:04:15,295 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:04:15,313 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:04:15,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:04:15,354 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 11:04:15,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:04:15,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:20,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:04:21,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:21,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:22,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:22,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:04:22,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:04:22,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:04:22,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:04:22,645 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 11:04:22,645 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:04:22,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:22,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:04:22,753 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 11:04:22,753 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:04:22,756 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 11:04:22,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:04:23,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 11:04:23,151 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:04:23,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 11:04:23,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 11:04:23,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 11:04:23,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 11:04:23,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 11:04:23,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 11:04:23,244 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 11:04:23,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:04:23,276 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 11:04:23,306 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:04:23,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:23,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:04:23,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:23,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:23,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:04:23,401 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 11:04:23,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:04:23,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:23,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:04:23,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:23,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:04:23,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:23,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:23,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:23,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:23,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:23,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:23,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:23,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:23,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:23,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:04:23,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:23,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:23,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:23,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:23,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:23,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:23,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:23,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:24,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:24,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:04:24,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:24,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:24,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:24,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:24,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:24,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:24,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:24,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:24,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:24,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:04:24,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:24,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:24,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:24,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:24,295 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:24,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:24,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:24,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:24,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:24,397 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:04:24,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:24,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:24,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:24,469 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:24,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:24,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:24,520 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:24,540 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:24,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:24,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:04:24,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:24,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:24,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:24,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:24,668 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:24,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:24,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:24,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:24,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:24,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:04:24,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:24,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:24,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:24,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:24,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:24,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:24,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:24,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:24,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:24,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:04:24,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:24,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:24,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:24,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:25,007 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:25,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:25,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:25,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:25,076 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 11:04:25,078 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 11:04:25,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:04:25,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 11:04:25,447 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:04:25,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 11:04:25,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 11:04:25,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 11:04:25,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 11:04:25,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 11:04:25,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 11:04:25,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 11:04:25,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:04:25,569 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 11:04:25,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:04:25,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:25,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:04:25,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:25,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:25,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:04:25,684 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 11:04:25,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:04:25,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:25,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:04:25,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:25,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:04:25,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:25,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:25,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:25,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:25,816 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:25,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:25,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:25,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:04:25,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:25,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:04:25,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:25,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:25,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:25,971 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:25,987 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:26,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:26,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:26,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:04:26,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:26,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:04:26,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:26,127 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:26,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:26,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:26,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:26,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:26,214 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:26,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:04:26,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:26,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:04:26,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:26,306 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:26,326 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:26,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:26,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:26,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:26,397 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:26,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:04:26,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:26,459 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:04:26,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:26,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:26,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:26,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:26,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:26,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:26,581 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:26,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:04:26,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:26,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:04:26,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:26,668 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:26,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:26,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:26,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:26,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:26,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:26,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:04:26,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:26,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:04:26,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:26,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:26,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:26,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:26,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:26,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:26,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:26,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:04:26,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:26,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:04:26,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:26,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:27,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:27,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:27,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:27,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:27,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:27,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:04:27,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:27,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:04:27,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:27,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:27,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:27,191 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:27,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:27,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:27,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:27,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:04:27,279 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 11:04:27,282 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:04:27,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:04:27,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:04:27,690 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:04:27,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:04:27,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:04:27,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:04:27,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:04:27,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:04:27,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:04:27,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:04:27,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:04:27,831 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 11:04:27,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:04:27,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:33,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:04:34,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:34,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:35,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:35,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:04:35,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:04:35,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:04:35,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:04:35,463 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 11:04:35,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:04:35,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:35,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:04:35,549 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 11:04:35,551 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:04:35,552 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:04:36,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:04:36,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:04:36,951 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:04:36,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:04:37,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:04:37,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:04:37,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:04:37,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:04:37,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:04:37,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:04:37,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:04:37,424 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 11:04:37,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:04:37,505 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:43,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:04:44,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:44,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:45,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:04:45,309 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 11:04:45,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:04:45,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:45,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:04:45,414 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 11:04:45,415 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:04:45,420 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:04:45,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:04:46,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:04:46,302 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:04:46,303 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:04:46,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:04:46,336 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:04:46,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:04:46,374 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:04:46,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:04:46,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:04:46,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:04:46,456 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 11:04:46,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:04:46,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:48,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:04:49,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:49,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 14197 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:04:50,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:04:50,154 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 11:04:50,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:04:50,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:50,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:04:50,244 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 11:04:50,246 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:04:50,247 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:04:50,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:04:50,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:04:50,788 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:04:50,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:04:50,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:04:50,839 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:04:50,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:04:50,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:04:50,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:04:50,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:04:50,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:04:51,000 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 11:04:51,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:04:51,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:57,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:04:58,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:58,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:04:59,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:59,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:04:59,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:04:59,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:04:59,729 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:04:59,749 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 11:04:59,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:04:59,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:04:59,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:04:59,802 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 11:04:59,803 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:04:59,805 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 11:04:59,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:05:00,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 11:05:00,190 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:05:00,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 11:05:00,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 11:05:00,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 11:05:00,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 11:05:00,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 11:05:00,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 11:05:00,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 11:05:00,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:05:00,305 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 11:05:00,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:05:00,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:05:00,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:05:00,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:05:00,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:05:00,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:05:00,427 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 11:05:00,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:05:00,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:05:00,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:05:00,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:05:00,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:05:00,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:05:00,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:05:00,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:05:00,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:05:00,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:05:00,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:05:00,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:05:00,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:05:00,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:05:00,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:05:00,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:05:00,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:05:00,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:05:00,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:05:00,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:05:00,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:05:00,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:05:01,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:05:01,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:05:01,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:05:01,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:05:01,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:05:01,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:05:01,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:05:01,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:05:01,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:05:01,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:05:01,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:05:01,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:05:01,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:05:01,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:05:01,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:05:01,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:05:01,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:05:01,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:05:01,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:05:01,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:05:01,349 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:05:01,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:05:01,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:05:01,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:05:01,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:05:01,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:05:01,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:05:01,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:05:01,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:05:01,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:05:01,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:05:01,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:05:01,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:05:01,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:05:01,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:05:01,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:05:01,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:05:01,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:05:01,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:05:01,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:05:01,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:05:01,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:05:01,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:05:01,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:05:01,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:05:01,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:05:01,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:05:01,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:05:01,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:05:01,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:05:01,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:05:01,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:05:01,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:05:01,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:05:01,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:05:01,915 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:05:01,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:05:01,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:05:01,969 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:05:01,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:05:02,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:05:02,019 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 11:05:02,019 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:05:02,022 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:05:02,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:05:02,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:05:02,388 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:05:02,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:05:02,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:05:02,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:05:02,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:05:02,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:05:02,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:05:02,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:05:02,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:02,943 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:05:02,944 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.924000024796. 2022-02-08 11:05:02,944 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:05:02,946 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:05:03,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:05:03,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:05:03,894 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:05:03,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:05:03,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:05:03,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:05:04,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:05:04,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:05:04,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:05:04,186 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:05:04,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:04,943 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:05:04,943 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:05:06,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:07,247 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:05:07,249 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:05:09,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:09,467 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:05:09,467 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:05:11,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:11,778 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:05:11,779 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:05:13,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:13,944 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:05:13,944 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:05:15,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:16,130 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:05:16,131 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:05:18,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:18,276 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:05:18,276 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 15.3329999447. 2022-02-08 11:05:18,276 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:05:18,279 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:05:18,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:05:19,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:05:19,296 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:05:19,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:05:19,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:05:19,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:05:19,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:05:19,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:05:19,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:05:19,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:05:19,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:20,158 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:05:20,160 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.88300013542. 2022-02-08 11:05:20,160 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:05:20,161 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:05:21,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:05:21,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:05:21,818 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:05:21,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:05:21,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:05:21,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:05:21,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:05:21,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:05:21,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:05:21,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:05:21,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:22,349 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:05:22,351 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.19099998474. 2022-02-08 11:05:22,351 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:05:22,352 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:05:22,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:05:23,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:05:23,456 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:05:23,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:05:23,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:05:23,497 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:05:23,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:05:23,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:05:23,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:05:23,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:05:23,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:24,272 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:05:24,272 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.92100000381. 2022-02-08 11:05:24,273 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:05:24,275 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:05:24,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:05:25,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:05:25,171 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:05:25,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:05:25,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:05:25,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:05:25,214 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:05:25,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:05:25,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:05:25,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:05:25,308 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:25,582 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:05:25,584 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.31100010872. 2022-02-08 11:05:25,584 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:05:25,585 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:05:26,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:05:26,834 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:05:26,999 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:05:27,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:05:27,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:05:27,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:05:27,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:05:27,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:05:27,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:05:27,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:05:27,119 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:27,572 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:05:27,573 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.98999977112. 2022-02-08 11:05:27,573 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:05:27,575 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:05:27,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:05:28,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:05:28,158 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:05:28,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:05:28,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:05:28,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:05:28,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:05:28,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:05:28,232 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:05:28,257 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:05:28,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:28,720 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:05:28,724 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.15000009537. 2022-02-08 11:05:28,724 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:05:28,726 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:05:28,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:05:29,306 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:05:29,622 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:05:29,624 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:05:29,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:05:29,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:05:29,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:05:29,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:05:29,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:05:30,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:05:30,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:30,457 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:05:30,457 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 11:05:32,459 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:32,729 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:05:32,729 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 11:05:34,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:35,138 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:05:35,138 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 11:05:37,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:37,572 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:05:37,572 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 11:05:39,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:40,119 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:05:40,121 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 11:05:42,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:42,880 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:05:42,880 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 11:05:44,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:45,249 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:05:45,249 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 11:05:47,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:05:47,783 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:05:47,785 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 19.0600001812. ############################################################ # # RAM FootPrint: 4757.29296875 # ############################################################ 2022-02-08 11:05:47,822 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 11:05:47,823 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 95 seconds. Reason: Iteration:7 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14209 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 14235 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:07:22,832 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 11:07:22,832 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 11:07:27,835 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:07:28,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:07:28,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:07:28,348 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:07:28,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:07:28,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:07:28,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:07:28,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:07:28,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:07:28,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:07:28,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:07:28,533 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:07:28,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:07:28,611 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['55063 memcached\n'], []) 2022-02-08 11:07:28,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 11:07:28,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:07:28,677 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['55063 memcached\n'], []) 2022-02-08 11:07:28,677 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:07:28,680 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:07:29,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:07:29,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:07:29,371 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:07:29,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:07:29,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:07:29,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:07:29,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:07:29,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:07:29,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:07:29,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:07:29,483 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:07:29,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:07:29,562 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['108713 memcached\n'], []) 2022-02-08 11:07:29,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 11:07:29,594 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:07:29,615 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['108713 memcached\n'], []) 2022-02-08 11:07:29,615 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:07:29,621 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:07:29,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:07:29,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:07:30,010 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:07:30,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:07:30,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:07:30,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:07:30,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:07:30,079 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:07:30,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:07:30,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:07:30,128 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:07:30,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:07:30,171 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['92181 memcached\n'], []) 2022-02-08 11:07:30,173 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 11:07:30,198 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:07:30,221 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['92181 memcached\n'], []) 2022-02-08 11:07:30,221 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:07:30,223 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:07:30,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:07:30,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:07:30,592 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:07:30,594 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:07:30,607 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:07:30,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:07:30,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:07:30,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:07:30,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:07:30,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:07:30,691 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:07:30,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:07:30,763 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['119194 memcached\n'], []) 2022-02-08 11:07:30,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 11:07:30,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:07:30,813 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['119194 memcached\n'], []) 2022-02-08 11:07:30,813 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:07:30,816 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:07:30,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:07:31,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:07:31,157 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:07:31,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:07:31,173 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:07:31,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:07:31,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:07:31,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:07:31,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:07:31,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:07:31,282 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:07:31,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:07:31,341 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['113039 memcached\n'], []) 2022-02-08 11:07:31,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 11:07:31,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:07:31,395 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['113039 memcached\n'], []) 2022-02-08 11:07:31,395 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:07:31,398 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:07:31,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:07:31,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:07:31,766 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:07:31,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:07:31,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:07:31,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:07:31,808 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:07:31,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:07:31,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:07:31,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:07:31,861 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:07:31,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:07:31,904 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['17464 memcached\n'], []) 2022-02-08 11:07:31,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 11:07:31,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:07:31,960 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['17464 memcached\n'], []) 2022-02-08 11:07:31,960 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:07:31,963 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:07:32,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:07:32,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:07:32,319 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:07:32,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:07:32,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:07:32,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:07:32,357 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:07:32,374 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:07:32,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:07:32,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:07:32,417 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:07:32,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:07:32,489 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['81792 memcached\n'], []) 2022-02-08 11:07:32,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 11:07:32,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:07:32,591 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['81792 memcached\n'], []) 2022-02-08 11:07:32,592 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:07:32,594 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:07:32,660 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:07:32,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:07:32,938 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:07:32,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:07:32,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:07:32,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:07:32,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:07:32,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:07:33,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:07:33,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:07:33,075 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:07:33,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:07:33,137 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['97025 memcached\n'], []) 2022-02-08 11:07:33,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 11:07:33,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:07:33,203 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['97025 memcached\n'], []) 2022-02-08 11:07:33,203 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:07:33,207 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:07:33,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:07:34,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:07:34,220 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:07:34,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:07:34,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:07:34,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:07:34,272 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:07:34,290 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:07:34,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:07:34,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:07:34,344 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:07:34,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:07:34,411 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['33149 memcached\n'], []) 2022-02-08 11:07:34,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 11:07:34,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:07:34,457 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['33149 memcached\n'], []) 2022-02-08 11:07:34,457 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:07:34,461 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:07:34,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:07:34,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:07:34,826 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:07:34,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:07:34,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:07:34,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:07:34,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:07:34,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:07:34,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:07:34,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:07:34,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:07:34,974 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 11:07:35,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:07:35,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:07:35,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:07:36,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:07:37,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:07:37,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:07:37,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:07:38,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:07:38,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:07:38,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:07:38,217 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 11:07:38,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:07:38,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:07:38,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:07:38,336 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 11:07:38,338 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:07:38,341 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:07:38,426 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:07:38,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:07:38,696 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:07:38,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:07:38,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:07:38,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:07:38,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:07:38,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:07:38,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:07:38,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:07:38,795 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:07:38,809 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 11:07:38,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:07:38,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:07:41,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:07:41,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:07:42,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:07:42,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:07:43,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:07:43,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:07:43,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:07:43,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:07:43,375 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 11:07:43,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:07:43,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:07:43,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:07:43,487 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 11:07:43,487 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:07:43,490 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 11:07:43,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:07:43,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 11:07:43,877 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:07:43,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 11:07:43,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 11:07:43,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 11:07:43,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 11:07:43,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 11:07:43,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 11:07:43,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 11:07:43,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:07:44,002 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 11:07:44,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:07:44,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:07:44,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:07:44,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:07:44,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:07:44,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:07:44,125 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 11:07:44,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:07:44,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:07:44,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:07:44,170 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:44,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:07:44,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:44,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:44,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:44,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:44,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:44,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:44,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:44,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:44,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:44,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:07:44,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:44,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:44,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:44,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:44,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:44,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:44,668 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:44,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:44,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:44,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:07:44,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:44,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:44,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:44,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:44,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:44,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:44,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:44,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:44,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:44,938 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:07:44,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:44,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:44,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:45,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:45,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:45,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:45,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:45,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:45,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:45,119 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:07:45,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:45,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:45,178 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:45,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:45,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:45,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:45,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:45,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:45,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:45,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:07:45,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:45,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:45,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:45,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:45,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:45,410 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:45,426 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:45,443 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:45,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:45,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:07:45,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:45,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:45,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:45,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:45,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:45,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:45,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:45,627 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:45,645 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:45,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:07:45,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:45,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:45,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:45,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:45,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:45,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:45,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:45,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:45,828 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 11:07:45,832 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:07:46,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:07:46,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:07:47,437 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:07:47,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:07:47,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:07:47,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:07:47,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:07:47,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:07:48,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:07:48,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:07:48,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:07:48,338 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 11:07:48,362 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:07:48,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 14261 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:07:52,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:07:53,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:07:53,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:07:53,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:07:53,783 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 11:07:53,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:07:53,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:07:53,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:07:53,880 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 11:07:53,880 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:07:53,884 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 11:07:53,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:07:54,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 11:07:54,226 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:07:54,227 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 11:07:54,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 11:07:54,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 11:07:54,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 11:07:54,278 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 11:07:54,290 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 11:07:54,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 11:07:54,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:07:54,329 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 11:07:54,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:07:54,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:07:54,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:07:54,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:07:54,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:07:54,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:07:54,444 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 11:07:54,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:07:54,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:07:54,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:07:54,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:54,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:07:54,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:54,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:54,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:54,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:54,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:54,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:54,607 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:54,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:07:54,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:54,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:07:54,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:54,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:54,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:54,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:54,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:54,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:54,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:54,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:07:54,808 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:54,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:07:54,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:54,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:54,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:54,897 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:54,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:54,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:54,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:54,966 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:07:54,982 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:54,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:07:55,017 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:55,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:55,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:55,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:55,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:55,096 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:55,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:55,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:07:55,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:55,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:07:55,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:55,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:55,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:55,221 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:55,237 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:55,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:55,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:55,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:07:55,298 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:55,313 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:07:55,329 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:55,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:55,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:55,374 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:55,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:55,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:55,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:55,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:07:55,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:55,469 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:07:55,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:55,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:55,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:55,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:55,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:55,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:55,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:55,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:07:55,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:55,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:07:55,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:55,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:55,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:55,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:55,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:55,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:55,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:55,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:07:55,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:07:55,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:07:55,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:07:55,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:07:55,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:07:55,836 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:07:55,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:07:55,874 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:07:55,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:07:55,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:07:55,928 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 11:07:55,930 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:07:56,007 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:07:56,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:07:56,276 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:07:56,278 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:07:56,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:07:56,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:07:56,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:07:56,336 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:07:56,349 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:07:56,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:07:56,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:07:56,394 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 11:07:56,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:07:56,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:07:59,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:08:00,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:01,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:01,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:01,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:08:01,793 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:08:01,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:08:01,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:08:01,944 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 11:08:01,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:08:01,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:02,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:08:02,025 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 11:08:02,026 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:08:02,028 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:08:02,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:08:02,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:08:02,440 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:08:02,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:08:02,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:08:02,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:08:02,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:08:02,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:08:02,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:08:02,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:08:02,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:08:02,604 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 11:08:02,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:08:02,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:06,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:08:07,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:07,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:08,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:08,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:08:09,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:08:09,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:08:09,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:08:09,470 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 11:08:09,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:08:09,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:09,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:08:09,601 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 11:08:09,601 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:08:09,605 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 11:08:09,696 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:08:09,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 11:08:10,046 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:08:10,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 11:08:10,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 11:08:10,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 11:08:10,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 11:08:10,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 11:08:10,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 11:08:10,163 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 11:08:10,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:08:10,200 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 11:08:10,227 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:08:10,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:10,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:08:10,280 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:10,298 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:10,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:08:10,332 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 11:08:10,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:08:10,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:10,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:08:10,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:10,426 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:08:10,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:10,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:10,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:10,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:10,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:10,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:10,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:10,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:10,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:10,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:08:10,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:10,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:10,910 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:10,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:10,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:10,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:11,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:11,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:11,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:11,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:08:11,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:11,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:11,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:11,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:11,170 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:11,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:11,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:11,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:11,255 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:11,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:08:11,298 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:11,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:11,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:11,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:11,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:11,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:11,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:11,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:11,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:11,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:08:11,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:11,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:11,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:11,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:11,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:11,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:11,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:11,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:11,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:11,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:08:11,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:11,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:11,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:11,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:11,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:11,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:11,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:11,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:11,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:11,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:08:11,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:11,923 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:11,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:11,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:11,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:12,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:12,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:12,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:12,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:12,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:08:12,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:12,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:12,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:12,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:12,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:12,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:12,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:12,237 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:12,260 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 11:08:12,263 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 11:08:12,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:08:12,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 11:08:12,624 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:08:12,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 11:08:12,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 11:08:12,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 11:08:12,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 11:08:12,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 11:08:12,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 11:08:12,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 11:08:12,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:08:12,736 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 11:08:12,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:08:12,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:12,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:08:12,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:12,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:12,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:08:12,851 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 11:08:12,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:08:12,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:12,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:08:12,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:12,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:08:12,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:12,938 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:12,956 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:12,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:12,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:13,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:13,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:13,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:13,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:13,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:08:13,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:13,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:13,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:13,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:13,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:13,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:13,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:13,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:13,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:13,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:08:13,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:13,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:13,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:13,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:13,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:13,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:13,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:13,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:13,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:13,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:08:13,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:13,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:13,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:13,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:13,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:13,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:13,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:13,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:13,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:13,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:08:13,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:13,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:13,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:13,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:13,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:13,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:13,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:13,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:13,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:13,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:08:13,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:13,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:13,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:13,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:13,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:13,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:13,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:13,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:13,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:13,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:08:13,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:13,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:13,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:13,989 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:14,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:14,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:14,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:14,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:14,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:14,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:08:14,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:14,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:14,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:14,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:14,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:14,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:14,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:14,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:14,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:14,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:08:14,260 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:14,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:14,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:14,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:14,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:14,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:14,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:14,374 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:14,390 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 11:08:14,391 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:08:14,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:08:14,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:08:14,832 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:08:14,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:08:14,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:08:14,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:08:14,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:08:14,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:08:14,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:08:14,938 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:08:14,956 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:08:14,990 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 11:08:15,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:08:15,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:16,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:08:17,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:17,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:18,336 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:18,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:08:18,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:08:18,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:08:18,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:08:18,615 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 11:08:18,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:08:18,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:18,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:08:18,700 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 11:08:18,700 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:08:18,703 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:08:18,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:08:19,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:08:19,075 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:08:19,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:08:19,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:08:19,102 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:08:19,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:08:19,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:08:19,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:08:19,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:08:19,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:08:19,188 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 11:08:19,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:08:19,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:19,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:08:20,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:20,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:21,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:08:21,451 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 11:08:21,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:08:21,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:21,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:08:21,496 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 11:08:21,496 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:08:21,499 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:08:22,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:08:22,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:08:22,480 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:08:22,482 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:08:22,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:08:22,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:08:22,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:08:22,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:08:22,591 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:08:22,624 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:08:22,647 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:08:22,664 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 11:08:22,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:08:22,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:23,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:08:24,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:24,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:24,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:08:24,740 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 11:08:24,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:08:24,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:24,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:08:24,832 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 11:08:24,832 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:08:24,835 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:08:24,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:08:25,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:08:25,216 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:08:25,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:08:25,232 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:08:25,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:08:25,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:08:25,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:08:25,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:08:25,308 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:08:25,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:08:25,342 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 11:08:25,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:08:25,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:27,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:08:27,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:28,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:28,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:30,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:08:30,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:08:30,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:08:30,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:08:30,592 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 11:08:30,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:08:30,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:30,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:08:30,681 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 11:08:30,683 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:08:30,684 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 11:08:30,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:08:31,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 11:08:31,098 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:08:31,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 11:08:31,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 11:08:31,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 11:08:31,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 11:08:31,163 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 11:08:31,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 11:08:31,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 11:08:31,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:08:31,221 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 11:08:31,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:08:31,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:31,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:08:31,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:31,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:08:31,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:08:31,332 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 11:08:31,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:08:31,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:08:31,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:08:31,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:31,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:08:31,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:31,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:31,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:31,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:31,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:31,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:31,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:31,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:08:31,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:31,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:08:31,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:31,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:31,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:31,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:31,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:31,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:31,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:31,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:08:31,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:31,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:08:31,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:31,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:31,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:31,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:31,793 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:31,811 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:31,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:31,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:08:31,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:31,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:08:31,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:31,918 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:31,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:31,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:31,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:31,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:32,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:32,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:08:32,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:32,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:08:32,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:32,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:32,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:32,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:32,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:32,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:32,186 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:32,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:08:32,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:32,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:08:32,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:32,285 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:32,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:32,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:32,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:32,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:32,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:32,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:08:32,397 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:32,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:08:32,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:32,443 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:32,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:32,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:32,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:32,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:32,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:32,540 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:08:32,556 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:32,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:08:32,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:32,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:32,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:32,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:32,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:32,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:32,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:32,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:08:32,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:32,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:08:32,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:32,778 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:32,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:32,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:32,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:32,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:32,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:32,887 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:08:32,905 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 11:08:32,905 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:08:32,908 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:08:33,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:08:33,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:08:33,359 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:08:33,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:08:33,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:08:33,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:08:33,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:08:33,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:08:33,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:08:33,459 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:08:33,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:08:34,015 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:08:34,016 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.11000013351. 2022-02-08 11:08:34,016 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:08:34,017 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:08:34,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:08:34,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:08:34,440 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:08:34,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:08:34,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:08:34,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:08:34,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:08:34,510 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:08:34,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:08:34,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:08:34,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:08:34,713 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:08:34,713 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:08:36,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:08:36,862 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:08:36,862 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:08:38,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:08:39,026 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:08:39,026 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:08:41,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:08:41,184 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:08:41,184 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:08:43,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:08:43,357 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:08:43,358 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:08:45,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:08:45,532 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:08:45,532 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:08:47,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:08:47,696 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:08:47,697 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:08:49,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 14261 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:08:49,865 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:08:49,865 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:08:51,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:08:52,043 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:08:52,045 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:08:54,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:08:54,210 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:08:54,210 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:08:56,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:08:56,430 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:08:56,430 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:08:58,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:08:58,648 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:08:58,648 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:09:00,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:00,864 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:09:00,864 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:09:02,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:03,193 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:09:03,194 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:09:05,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:05,417 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:09:05,417 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:09:07,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:07,645 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:09:07,647 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:09:09,647 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:09,839 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:09:09,839 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:09:11,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:12,013 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:09:12,015 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:09:14,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:14,196 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:09:14,196 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:09:16,196 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:16,346 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:09:16,348 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:09:18,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:18,575 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:09:18,575 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:09:20,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:20,903 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:09:20,904 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:09:22,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:23,283 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:09:23,283 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:09:25,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:25,418 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:09:25,418 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 51.4029998779. 2022-02-08 11:09:25,418 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:09:25,421 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:09:26,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:09:27,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:09:27,295 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:09:27,295 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:09:27,313 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:09:27,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:09:27,349 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:09:27,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:09:27,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:09:27,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:09:27,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:32,361 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:09:32,361 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.9430000782. 2022-02-08 11:09:32,361 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:09:32,364 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:09:32,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:09:33,298 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:09:33,562 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:09:33,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:09:33,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:09:33,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:09:33,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:09:33,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:09:33,640 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:09:33,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:09:33,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:34,285 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:09:34,285 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.92300009727. 2022-02-08 11:09:34,285 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:09:34,286 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:09:34,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:09:35,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:09:35,197 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:09:35,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:09:35,214 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:09:35,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:09:35,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:09:35,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:09:35,280 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:09:35,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:09:35,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:36,698 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:09:36,698 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.41399979591. 2022-02-08 11:09:36,698 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:09:36,700 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:09:37,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:09:37,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:09:38,046 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:09:38,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:09:38,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:09:38,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:09:38,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:09:38,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:09:38,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:09:38,163 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:09:38,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:38,763 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:09:38,763 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.0640001297. 2022-02-08 11:09:38,763 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:09:38,765 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:09:39,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:09:39,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:09:40,006 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:09:40,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:09:40,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:09:40,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:09:40,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:09:40,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:09:40,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:09:40,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:09:40,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:40,578 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:09:40,579 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.81699991226. 2022-02-08 11:09:40,579 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:09:40,582 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:09:41,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:09:41,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:09:41,881 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:09:41,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:09:41,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:09:41,918 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:09:41,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:09:41,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:09:41,961 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:09:41,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:09:41,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:42,441 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:09:42,443 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.8630001545. 2022-02-08 11:09:42,443 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:09:42,444 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:09:42,971 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:09:43,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:09:43,279 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:09:43,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:09:43,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:09:43,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:09:43,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:09:43,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:09:43,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:09:43,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:09:43,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:09:44,256 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:09:44,256 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.81399989128. ############################################################ # # RAM FootPrint: 4738.57421875 # ############################################################ 2022-02-08 11:09:44,290 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 11:09:44,290 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 71 seconds. Reason: Iteration:8 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14271 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 14296 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:10:55,296 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 11:10:55,296 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 11:11:00,299 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:11:00,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:00,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:11:00,707 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:00,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:11:00,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:11:00,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:11:00,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:11:00,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:11:00,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:11:00,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:11:00,811 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:11:00,811 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:11:00,874 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['55758 memcached\n'], []) 2022-02-08 11:11:00,874 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 11:11:00,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:11:00,930 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['55758 memcached\n'], []) 2022-02-08 11:11:00,930 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:11:00,931 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:11:01,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:01,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:11:01,789 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:01,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:11:01,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:11:01,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:11:01,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:11:01,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:11:01,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:11:01,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:11:01,914 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:11:01,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:11:01,982 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['109541 memcached\n'], []) 2022-02-08 11:11:01,982 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 11:11:02,007 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:11:02,029 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['109541 memcached\n'], []) 2022-02-08 11:11:02,029 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:11:02,030 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:11:02,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:02,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:11:02,441 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:02,443 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:11:02,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:11:02,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:11:02,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:11:02,507 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:11:02,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:11:02,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:11:02,552 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:11:02,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:11:02,604 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['93035 memcached\n'], []) 2022-02-08 11:11:02,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 11:11:02,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:11:02,653 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['93035 memcached\n'], []) 2022-02-08 11:11:02,653 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:11:02,654 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:11:02,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:02,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:11:03,009 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:03,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:11:03,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:11:03,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:11:03,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:11:03,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:11:03,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:11:03,086 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:11:03,101 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:11:03,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:11:03,151 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['119944 memcached\n'], []) 2022-02-08 11:11:03,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 11:11:03,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:11:03,200 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['119944 memcached\n'], []) 2022-02-08 11:11:03,200 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:11:03,203 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:11:03,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:03,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:11:03,559 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:03,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:11:03,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:11:03,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:11:03,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:11:03,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:11:03,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:11:03,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:11:03,667 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:11:03,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:11:03,710 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['113792 memcached\n'], []) 2022-02-08 11:11:03,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 11:11:03,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:11:03,757 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['113792 memcached\n'], []) 2022-02-08 11:11:03,759 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:11:03,760 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:11:03,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:04,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:11:04,128 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:04,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:11:04,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:11:04,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:11:04,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:11:04,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:11:04,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:11:04,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:11:04,220 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:11:04,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:11:04,296 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['18220 memcached\n'], []) 2022-02-08 11:11:04,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 11:11:04,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:11:04,341 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['18220 memcached\n'], []) 2022-02-08 11:11:04,341 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:11:04,344 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:11:04,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:04,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:11:04,703 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:04,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:11:04,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:11:04,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:11:04,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:11:04,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:11:04,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:11:04,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:11:04,798 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:11:04,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:11:04,857 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['82515 memcached\n'], []) 2022-02-08 11:11:04,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 11:11:04,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:11:04,904 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['82515 memcached\n'], []) 2022-02-08 11:11:04,904 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:11:04,907 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:11:04,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:05,224 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:11:05,282 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:05,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:11:05,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:11:05,306 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:11:05,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:11:05,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:11:05,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:11:05,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:11:05,375 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:11:05,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:11:05,441 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['97749 memcached\n'], []) 2022-02-08 11:11:05,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 11:11:05,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:11:05,484 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['97749 memcached\n'], []) 2022-02-08 11:11:05,486 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:11:05,486 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:11:05,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:05,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:11:05,877 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:05,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:11:05,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:11:05,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:11:05,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:11:05,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:11:05,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:11:05,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:11:05,983 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:11:05,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:11:06,035 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['34121 memcached\n'], []) 2022-02-08 11:11:06,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 11:11:06,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:11:06,081 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['34121 memcached\n'], []) 2022-02-08 11:11:06,081 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:11:06,082 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:11:06,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:06,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:11:06,430 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:06,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:11:06,446 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:11:06,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:11:06,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:11:06,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:11:06,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:11:06,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:11:06,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:11:06,539 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 11:11:06,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:11:06,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:08,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:11:09,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:10,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:10,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:11,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:11:11,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:11:11,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:11:11,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:11:11,434 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 11:11:11,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:11:11,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:11,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:11:11,532 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 11:11:11,533 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:11:11,536 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:11:11,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:11,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:11:11,878 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:11,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:11:11,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:11:11,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:11:11,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:11:11,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:11:11,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:11:11,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:11:12,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:11:12,025 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 11:11:12,056 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:11:12,071 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:18,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:11:19,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:20,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:20,543 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:20,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:11:20,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:11:20,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:11:20,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:11:20,766 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 11:11:20,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:11:20,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:20,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:11:20,867 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 11:11:20,868 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:11:20,869 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 11:11:20,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:21,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 11:11:21,292 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:21,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 11:11:21,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 11:11:21,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 11:11:21,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 11:11:21,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 11:11:21,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 11:11:21,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 11:11:21,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:11:21,434 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 11:11:21,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:11:21,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:21,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:11:21,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:21,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:21,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:11:21,565 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 11:11:21,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:11:21,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:21,596 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:11:21,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:21,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:11:21,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:21,655 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:21,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:21,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:21,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:21,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:21,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:21,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:21,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:21,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:11:21,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:21,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:21,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:21,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:21,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:21,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:21,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:21,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:21,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:21,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:11:21,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:22,007 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:22,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:22,043 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:22,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:22,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:22,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:22,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:22,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:22,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:11:22,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:22,196 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:22,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:22,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:22,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:22,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:22,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:22,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:22,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:22,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:11:22,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:22,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:22,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:22,403 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:22,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:22,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:22,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:22,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:22,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:22,507 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:11:22,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:22,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:22,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:22,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:22,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:22,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:22,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:22,647 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:22,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:22,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:11:22,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:22,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:22,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:22,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:22,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:22,778 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:22,793 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:22,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:22,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:22,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:11:22,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:22,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:22,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:22,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:22,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:22,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:22,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:22,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:22,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:23,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:11:23,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:23,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:23,071 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:23,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:23,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:23,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:23,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:23,160 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:23,177 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 11:11:23,178 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:11:23,497 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:24,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:11:24,811 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:24,811 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:11:24,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:11:25,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:11:25,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:11:25,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:11:25,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:11:25,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:11:25,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:11:25,335 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 11:11:25,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:11:25,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:25,979 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:11:26,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:26,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:27,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:11:27,289 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 11:11:27,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:11:27,303 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:27,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:11:27,384 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 11:11:27,385 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:11:27,388 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 11:11:27,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:27,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 11:11:27,747 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:27,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 11:11:27,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 11:11:27,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 11:11:27,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 11:11:27,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 11:11:27,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 11:11:27,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 11:11:27,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:11:27,859 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 11:11:27,897 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:11:27,913 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:27,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:11:27,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:27,966 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:27,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:11:27,990 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 11:11:27,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:11:28,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:28,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:11:28,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:28,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:11:28,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:28,086 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:28,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:28,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:28,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:28,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:28,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:28,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:28,186 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:28,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:11:28,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:28,232 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:28,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:28,260 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:28,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:28,290 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:28,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:28,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:28,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:28,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:11:28,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:28,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:28,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:28,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:28,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:28,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:28,494 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:28,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:28,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:28,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:11:28,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:28,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:28,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:28,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:28,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:28,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:28,658 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:28,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:28,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:28,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:11:28,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:28,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:28,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:28,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:28,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:28,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:28,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:28,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:28,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:28,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:11:28,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:28,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:28,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:28,923 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:28,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:28,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:28,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:28,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:29,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:29,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:11:29,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:29,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:29,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:29,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:29,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:29,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:29,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:29,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:29,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:29,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:11:29,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:29,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:29,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:29,227 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:29,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:29,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:29,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:29,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:29,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:29,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:11:29,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:29,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:29,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:29,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:29,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:29,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:29,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:29,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:29,454 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 11:11:29,457 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:11:29,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:29,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:11:29,842 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:29,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:11:29,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:11:29,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:11:29,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:11:29,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:11:29,913 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:11:29,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:11:29,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:11:29,959 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 11:11:29,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:11:30,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:34,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:11:35,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:35,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:36,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:36,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:11:36,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:11:36,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:11:36,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:11:36,960 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 11:11:36,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:11:36,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:37,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:11:37,052 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 11:11:37,052 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:11:37,055 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:11:37,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:37,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:11:37,443 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:37,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:11:37,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:11:37,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:11:37,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:11:37,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:11:37,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:11:37,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:11:37,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:11:37,592 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 11:11:37,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:11:37,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:39,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:11:40,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:41,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:41,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:41,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:11:41,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:11:41,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:11:41,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:11:41,931 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 11:11:41,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:11:41,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:42,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:11:42,038 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 11:11:42,039 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:11:42,040 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 11:11:42,132 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:42,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 11:11:42,461 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:42,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 11:11:42,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 11:11:42,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 11:11:42,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 11:11:42,535 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 11:11:42,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 11:11:42,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 11:11:42,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:11:42,602 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 11:11:42,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:11:42,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:42,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:11:42,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:42,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:42,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:11:42,743 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 11:11:42,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:11:42,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:42,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:11:42,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:42,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:11:42,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:42,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:42,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:42,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:42,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:42,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:42,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:42,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:42,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:42,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:11:42,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:43,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:43,038 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:43,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:43,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:43,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:43,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:43,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:43,170 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:43,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:11:43,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:43,229 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:43,255 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:43,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:43,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:43,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:43,336 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:43,357 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:43,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:43,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:11:43,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:43,446 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:43,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:43,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:43,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:43,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:43,543 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:43,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:43,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:43,604 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:11:43,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:43,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:43,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:43,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:43,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:43,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:43,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:43,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:43,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:43,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:11:43,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:43,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:43,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:43,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:43,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:43,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:43,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:43,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:43,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:43,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:11:43,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:44,017 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:44,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:44,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:44,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:44,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:44,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:44,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:44,147 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:44,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:11:44,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:44,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:44,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:44,237 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:44,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:44,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:44,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:44,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:44,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:44,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:11:44,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:44,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:44,410 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:44,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:44,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:44,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:44,497 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:44,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:44,540 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 11:11:44,542 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 11:11:44,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:44,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 11:11:44,921 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:44,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 11:11:44,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 11:11:44,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 11:11:44,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 11:11:44,982 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 11:11:45,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 11:11:45,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 11:11:45,061 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:11:45,082 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 11:11:45,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:11:45,127 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:45,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:11:45,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:45,170 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:45,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:11:45,200 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 11:11:45,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:11:45,214 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:45,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:11:45,244 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:45,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:11:45,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:45,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:45,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:45,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:45,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:45,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:45,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:45,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:11:45,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:45,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:11:45,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:45,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:45,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:45,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:45,520 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:45,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:45,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:45,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:11:45,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:45,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:11:45,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:45,645 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:45,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:45,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:45,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:45,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:45,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:45,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:11:45,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:45,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:11:45,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:45,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:45,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:45,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:45,887 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:45,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:45,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:45,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:11:45,956 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:45,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:11:45,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:46,007 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:46,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:46,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:46,056 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:46,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:46,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:46,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:11:46,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:46,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:11:46,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:46,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:46,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:46,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:46,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:46,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:46,260 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:46,278 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:11:46,295 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:46,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:11:46,326 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:46,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:46,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:46,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:46,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:46,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:46,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:46,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:11:46,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:46,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:11:46,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:46,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:46,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:46,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:46,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:46,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:46,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:46,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:11:46,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:46,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:11:46,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:46,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:46,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:46,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:46,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:46,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:46,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:46,778 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:11:46,812 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 11:11:46,822 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:11:46,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:47,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:11:47,382 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:47,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:11:47,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:11:47,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:11:47,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:11:47,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:11:47,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:11:47,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:11:47,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:11:47,598 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 11:11:47,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:11:47,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 14307 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:11:52,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:11:53,520 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:54,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:11:54,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:54,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:11:55,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:11:55,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:11:55,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:11:55,157 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 11:11:55,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:11:55,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:11:55,224 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:11:55,249 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 11:11:55,250 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:11:55,252 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:11:55,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:11:55,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:11:55,680 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:11:55,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:11:55,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:11:55,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:11:55,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:11:55,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:11:55,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:11:55,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:11:55,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:11:55,818 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 11:11:55,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:11:55,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:12:00,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:12:01,428 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:12:01,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:12:02,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:12:02,276 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 11:12:02,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:12:02,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:12:02,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:12:02,336 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 11:12:02,338 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:12:02,341 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:12:02,936 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:12:03,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:12:03,316 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:12:03,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:12:03,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:12:03,349 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:12:03,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:12:03,395 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:12:03,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:12:03,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:12:03,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:12:03,473 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 11:12:03,507 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:12:03,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:12:06,147 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:12:06,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:12:07,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:12:07,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:12:07,710 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 11:12:07,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:12:07,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:12:07,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:12:07,792 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 11:12:07,792 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:12:07,795 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:12:07,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:12:08,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:12:08,493 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:12:08,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:12:08,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:12:08,647 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:12:08,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:12:08,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:12:08,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:12:08,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:12:08,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:12:08,789 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 11:12:08,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:12:08,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:12:13,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:12:14,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:12:15,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:12:15,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:12:16,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:12:16,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:12:16,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:12:16,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:12:16,581 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 11:12:16,581 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:12:16,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:12:16,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:12:16,697 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 11:12:16,697 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:12:16,700 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 11:12:16,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:12:17,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 11:12:17,098 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:12:17,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 11:12:17,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 11:12:17,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 11:12:17,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 11:12:17,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 11:12:17,178 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 11:12:17,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 11:12:17,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:12:17,223 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 11:12:17,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:12:17,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:12:17,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:12:17,298 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:12:17,313 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:12:17,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:12:17,342 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 11:12:17,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:12:17,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:12:17,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:12:17,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:12:17,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:12:17,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:12:17,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:12:17,487 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:12:17,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:12:17,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:12:17,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:12:17,556 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:12:17,573 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:12:17,591 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:12:17,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:12:17,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:12:17,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:12:17,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:12:17,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:12:17,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:12:17,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:12:17,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:12:17,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:12:17,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:12:17,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:12:17,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:12:17,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:12:17,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:12:17,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:12:17,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:12:17,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:12:17,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:12:17,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:12:17,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:12:17,969 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:12:17,987 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:12:18,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:12:18,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:12:18,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:12:18,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:12:18,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:12:18,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:12:18,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:12:18,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:12:18,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:12:18,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:12:18,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:12:18,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:12:18,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:12:18,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:12:18,257 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:12:18,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:12:18,295 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:12:18,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:12:18,329 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:12:18,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:12:18,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:12:18,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:12:18,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:12:18,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:12:18,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:12:18,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:12:18,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:12:18,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:12:18,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:12:18,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:12:18,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:12:18,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:12:18,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:12:18,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:12:18,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:12:18,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:12:18,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:12:18,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:12:18,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:12:18,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:12:18,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:12:18,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:12:18,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:12:18,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:12:18,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:12:18,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:12:18,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:12:18,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:12:18,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:12:18,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:12:18,900 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:12:18,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:12:18,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:12:18,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:12:18,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:12:18,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:12:19,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:12:19,036 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 11:12:19,036 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:12:19,039 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:12:19,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:12:19,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:12:19,424 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:12:19,426 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:12:19,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:12:19,454 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:12:19,469 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:12:19,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:12:19,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:12:19,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:12:19,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:20,019 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:12:20,019 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 0.982000112534. 2022-02-08 11:12:20,019 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:12:20,022 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:12:20,173 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:12:20,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:12:20,582 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:12:20,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:12:20,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:12:20,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:12:20,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:12:20,658 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:12:20,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:12:20,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:12:20,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:21,039 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:12:21,039 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:12:23,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:23,430 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:12:23,430 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:12:25,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:25,770 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:12:25,770 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:12:27,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:28,101 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:12:28,102 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:12:30,102 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:30,575 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:12:30,575 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:12:32,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:32,788 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:12:32,788 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:12:34,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:34,996 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:12:34,996 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:12:36,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:37,171 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:12:37,171 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:12:39,173 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:39,355 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:12:39,355 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:12:41,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:41,516 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:12:41,516 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:12:43,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:43,716 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:12:43,717 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:12:45,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:45,875 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:12:45,875 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:12:47,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:48,029 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:12:48,029 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 14314 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:12:50,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:50,177 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:12:50,177 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:12:52,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:52,329 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:12:52,329 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:12:54,331 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:54,470 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:12:54,470 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:12:56,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:12:56,615 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:12:56,617 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 36.5980000496. 2022-02-08 11:12:56,617 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:12:56,618 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:12:57,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:12:58,278 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:12:58,759 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:12:58,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:12:58,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:12:58,913 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:12:59,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:12:59,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:12:59,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:12:59,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:12:59,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:13:00,295 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:13:00,295 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 11:13:02,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:13:02,457 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:13:02,457 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.83999991417. 2022-02-08 11:13:02,457 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:13:02,460 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:13:03,655 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:13:04,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:13:04,404 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:13:04,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:13:04,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:13:04,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:13:04,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:13:04,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:13:04,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:13:04,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:13:04,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:13:06,089 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:13:06,091 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.63300013542. 2022-02-08 11:13:06,091 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:13:06,092 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:13:06,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:13:07,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:13:07,651 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:13:07,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:13:07,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:13:07,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:13:07,696 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:13:07,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:13:07,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:13:07,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:13:07,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:13:09,595 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:13:09,595 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.5039999485. 2022-02-08 11:13:09,595 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:13:09,596 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:13:10,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:13:10,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:13:10,608 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:13:10,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:13:10,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:13:10,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:13:10,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:13:10,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:13:10,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:13:10,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:13:10,729 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:13:11,349 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:13:11,349 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.75499987602. 2022-02-08 11:13:11,349 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:13:11,351 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:13:12,002 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:13:12,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:13:12,525 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:13:12,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:13:12,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:13:12,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:13:12,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:13:12,591 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:13:12,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:13:12,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:13:12,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:13:13,085 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:13:13,088 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.73900008202. 2022-02-08 11:13:13,088 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:13:13,091 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:13:13,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:13:14,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:13:14,634 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:13:14,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:13:14,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:13:14,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:13:14,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:13:14,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:13:14,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:13:14,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:13:14,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:13:16,292 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:13:16,292 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.20300006866. 2022-02-08 11:13:16,292 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:13:16,293 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:13:16,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:13:17,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:13:17,312 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:13:17,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:13:17,326 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:13:17,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:13:17,362 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:13:17,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:13:17,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:13:17,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:13:17,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:13:18,203 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:13:18,204 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.91100001335. ############################################################ # # RAM FootPrint: 4784.33984375 # ############################################################ 2022-02-08 11:13:18,246 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 11:13:18,246 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 107 seconds. Reason: Iteration:9 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14332 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 14359 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:15:05,253 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 11:15:05,253 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 11:15:10,256 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:15:10,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:10,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:15:10,707 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:10,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:15:10,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:15:10,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:15:10,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:15:10,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:15:10,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:15:10,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:15:10,839 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:15:10,839 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:15:10,917 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['56459 memcached\n'], []) 2022-02-08 11:15:10,918 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 11:15:10,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:15:10,977 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['56459 memcached\n'], []) 2022-02-08 11:15:10,977 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:15:10,980 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:15:11,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:11,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:15:11,730 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:11,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:15:11,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:15:11,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:15:11,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:15:11,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:15:11,829 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:15:11,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:15:11,857 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:15:11,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:15:11,921 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['110551 memcached\n'], []) 2022-02-08 11:15:11,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 11:15:11,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:15:11,970 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['110551 memcached\n'], []) 2022-02-08 11:15:11,970 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:15:11,971 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:15:12,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:12,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:15:12,484 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:12,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:15:12,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:15:12,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:15:12,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:15:12,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:15:12,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:15:12,579 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:15:12,595 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:15:12,596 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:15:12,640 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['93844 memcached\n'], []) 2022-02-08 11:15:12,640 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 11:15:12,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:15:12,687 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['93844 memcached\n'], []) 2022-02-08 11:15:12,688 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:15:12,690 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:15:12,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:13,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:15:13,075 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:13,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:15:13,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:15:13,102 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:15:13,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:15:13,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:15:13,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:15:13,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:15:13,170 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:15:13,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:15:13,242 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['120631 memcached\n'], []) 2022-02-08 11:15:13,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 11:15:13,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:15:13,290 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['120631 memcached\n'], []) 2022-02-08 11:15:13,292 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:15:13,295 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:15:13,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:13,591 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:15:13,650 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:13,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:15:13,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:15:13,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:15:13,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:15:13,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:15:13,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:15:13,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:15:13,746 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:15:13,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:15:13,789 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['114493 memcached\n'], []) 2022-02-08 11:15:13,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 11:15:13,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:15:13,842 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['114493 memcached\n'], []) 2022-02-08 11:15:13,842 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:15:13,845 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:15:13,938 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:14,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:15:14,209 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:14,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:15:14,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:15:14,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:15:14,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:15:14,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:15:14,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:15:14,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:15:14,319 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:15:14,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:15:14,371 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['18922 memcached\n'], []) 2022-02-08 11:15:14,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 11:15:14,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:15:14,423 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['18922 memcached\n'], []) 2022-02-08 11:15:14,424 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:15:14,426 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:15:14,497 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:14,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:15:14,772 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:14,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:15:14,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:15:14,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:15:14,811 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:15:14,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:15:14,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:15:14,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:15:14,868 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:15:14,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:15:14,934 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['83226 memcached\n'], []) 2022-02-08 11:15:14,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 11:15:14,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:15:14,979 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['83226 memcached\n'], []) 2022-02-08 11:15:14,980 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:15:14,982 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:15:15,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:15,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:15:15,349 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:15,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:15:15,367 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:15:15,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:15:15,395 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:15:15,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:15:15,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:15:15,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:15:15,453 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:15:15,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:15:15,516 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['98407 memcached\n'], []) 2022-02-08 11:15:15,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 11:15:15,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:15:15,565 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['98407 memcached\n'], []) 2022-02-08 11:15:15,565 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:15:15,568 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:15:15,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:15,900 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:15:16,016 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:16,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:15:16,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:15:16,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:15:16,056 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:15:16,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:15:16,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:15:16,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:15:16,118 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:15:16,119 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:15:16,165 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['34814 memcached\n'], []) 2022-02-08 11:15:16,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 11:15:16,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:15:16,210 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['34814 memcached\n'], []) 2022-02-08 11:15:16,211 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:15:16,213 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:15:16,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:16,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:15:16,631 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:16,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:15:16,645 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:15:16,660 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:15:16,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:15:16,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:15:16,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:15:16,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:15:16,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:15:16,772 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 11:15:16,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:15:16,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:21,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:15:22,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:22,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:23,170 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:23,303 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:15:23,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:15:23,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:15:23,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:15:23,575 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 11:15:23,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:15:23,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:23,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:15:23,678 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 11:15:23,680 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:15:23,683 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:15:23,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:24,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:15:24,121 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:24,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:15:24,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:15:24,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:15:24,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:15:24,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:15:24,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:15:24,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:15:24,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:15:24,233 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 11:15:24,265 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:15:24,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:27,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:15:28,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:28,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:29,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:29,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:15:29,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:15:29,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:15:29,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:15:29,954 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 11:15:29,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:15:29,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:30,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:15:30,065 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 11:15:30,065 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:15:30,068 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 11:15:30,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:30,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 11:15:30,471 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:30,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 11:15:30,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 11:15:30,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 11:15:30,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 11:15:30,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 11:15:30,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 11:15:30,579 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 11:15:30,594 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:15:30,611 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 11:15:30,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:15:30,658 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:30,676 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:15:30,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:30,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:30,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:15:30,737 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 11:15:30,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:15:30,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:30,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:15:30,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:30,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:15:30,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:30,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:30,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:30,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:30,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:30,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:30,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:30,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:30,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:30,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:15:30,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:30,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:31,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:31,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:31,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:31,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:31,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:31,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:31,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:31,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:15:31,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:31,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:31,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:31,201 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:31,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:31,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:31,255 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:31,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:31,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:31,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:15:31,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:31,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:31,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:31,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:31,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:31,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:31,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:31,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:31,497 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:31,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:15:31,535 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:31,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:31,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:31,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:31,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:31,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:31,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:31,655 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:31,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:31,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:15:31,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:31,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:31,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:31,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:31,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:31,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:31,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:31,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:31,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:31,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:15:31,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:31,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:31,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:31,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:31,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:31,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:32,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:32,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:32,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:32,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:15:32,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:32,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:32,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:32,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:32,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:32,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:32,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:32,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:32,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:32,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:15:32,232 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:32,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:32,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:32,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:32,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:32,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:32,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:32,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:32,371 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 11:15:32,374 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:15:32,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:33,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:15:33,802 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:33,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:15:33,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:15:34,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:15:34,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:15:34,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:15:34,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:15:34,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:15:34,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:15:34,802 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 11:15:34,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:15:34,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:36,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:15:36,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:37,127 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:37,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:15:37,453 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 11:15:37,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:15:37,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:37,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:15:37,555 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 11:15:37,555 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:15:37,556 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 11:15:37,645 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:37,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 11:15:37,924 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:37,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 11:15:37,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 11:15:37,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 11:15:37,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 11:15:37,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 11:15:37,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 11:15:38,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 11:15:38,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:15:38,038 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 11:15:38,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:15:38,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:38,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:15:38,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:38,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:38,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:15:38,151 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 11:15:38,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:15:38,163 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:38,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:15:38,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:38,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:15:38,221 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:38,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:38,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:38,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:38,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:38,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:38,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:38,329 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:38,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:38,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:15:38,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:38,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:38,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:38,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:38,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:38,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:38,469 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:38,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:38,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:38,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:15:38,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:38,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:38,594 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:38,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:38,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:38,645 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:38,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:38,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:38,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:38,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:15:38,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:38,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:38,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:38,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:38,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:38,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:38,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:38,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:38,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:38,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:15:38,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:38,915 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:38,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:38,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:38,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:38,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:38,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:39,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:39,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:39,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:15:39,061 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:39,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:39,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:39,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:39,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:39,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:39,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:39,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:39,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:39,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:15:39,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:39,237 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:39,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:39,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:39,280 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:39,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:39,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:39,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:39,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:39,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:15:39,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:39,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:39,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:39,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:39,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:39,446 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:39,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:39,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:39,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:39,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:15:39,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:39,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:39,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:39,573 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:39,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:39,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:39,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:39,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:39,657 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 11:15:39,660 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:15:39,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:39,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:15:40,010 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:40,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:15:40,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:15:40,038 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:15:40,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:15:40,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:15:40,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:15:40,096 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:15:40,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:15:40,125 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 11:15:40,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:15:40,173 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:43,295 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:15:44,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:44,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:45,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:45,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:15:45,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:15:45,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:15:45,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:15:45,663 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 11:15:45,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:15:45,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:45,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:15:45,739 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 11:15:45,739 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:15:45,740 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:15:45,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:46,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:15:46,089 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:46,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:15:46,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:15:46,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:15:46,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:15:46,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:15:46,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:15:46,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:15:46,201 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:15:46,219 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 11:15:46,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:15:46,265 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:47,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:15:48,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:49,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:49,645 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:49,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:15:49,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 5 queries submitted, 0 failed, 14374 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:15:49,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:15:49,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:15:49,944 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 11:15:49,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:15:49,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:50,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:15:50,046 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 11:15:50,046 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:15:50,049 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 11:15:50,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:50,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 11:15:50,450 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:50,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 11:15:50,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 11:15:50,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 11:15:50,494 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 11:15:50,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 11:15:50,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 11:15:50,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 11:15:50,556 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:15:50,572 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 11:15:50,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:15:50,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:50,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:15:50,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:50,668 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:50,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:15:50,701 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 11:15:50,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:15:50,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:50,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:15:50,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:50,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:15:50,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:50,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:50,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:50,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:50,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:50,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:50,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:50,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:50,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:50,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:15:50,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:50,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:50,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:51,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:51,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:51,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:51,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:51,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:51,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:51,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:15:51,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:51,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:51,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:51,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:51,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:51,232 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:51,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:51,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:51,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:51,306 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:15:51,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:51,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:51,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:51,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:51,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:51,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:51,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:51,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:51,474 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:51,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:15:51,510 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:51,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:51,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:51,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:51,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:51,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:51,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:51,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:51,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:51,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:15:51,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:51,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:51,721 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:51,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:51,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:51,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:51,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:51,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:51,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:51,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:15:51,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:51,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:51,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:51,956 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:51,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:51,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:52,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:52,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:52,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:52,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:15:52,086 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:52,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:52,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:52,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:52,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:52,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:52,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:52,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:52,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:52,265 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:15:52,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:52,306 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:52,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:52,346 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:52,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:52,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:52,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:52,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:52,444 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 11:15:52,447 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 11:15:52,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:52,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 11:15:52,803 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:52,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 11:15:52,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 11:15:52,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 11:15:52,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 11:15:52,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 11:15:52,874 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 11:15:52,887 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 11:15:52,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:15:52,915 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 11:15:52,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:15:52,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:52,971 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:15:52,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:52,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:53,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:15:53,025 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 11:15:53,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:15:53,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:53,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:15:53,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:53,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:15:53,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:53,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:53,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:53,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:53,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:53,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:53,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:53,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:15:53,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:53,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:15:53,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:53,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:53,303 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:53,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:53,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:53,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:53,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:53,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:15:53,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:53,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:15:53,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:53,443 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:53,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:53,474 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:53,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:53,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:53,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:53,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:15:53,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:53,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:15:53,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:53,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:53,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:53,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:53,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:53,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:53,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:53,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:15:53,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:53,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:15:53,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:53,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:53,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:53,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:53,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:53,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:53,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:53,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:15:53,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:53,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:15:53,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:53,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:53,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:53,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:53,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:53,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:54,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:54,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:15:54,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:54,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:15:54,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:54,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:54,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:54,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:54,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:54,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:54,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:54,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:15:54,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:54,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:15:54,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:54,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:54,265 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:54,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:54,295 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:54,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:54,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:54,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:15:54,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:54,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:15:54,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:54,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:54,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:54,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:54,454 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:54,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:54,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:54,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:15:54,519 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 11:15:54,522 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:15:54,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:15:54,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:15:54,973 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:15:54,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:15:54,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:15:55,007 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:15:55,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:15:55,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:15:55,061 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:15:55,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:15:55,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:15:55,131 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 11:15:55,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:15:55,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:57,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:15:58,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:58,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:15:59,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:59,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:15:59,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:15:59,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:15:59,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:15:59,678 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 11:15:59,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:15:59,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:15:59,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:15:59,763 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 11:15:59,765 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:15:59,767 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:15:59,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:16:00,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:16:00,170 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:16:00,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:16:00,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:16:00,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:16:00,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:16:00,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:16:00,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:16:00,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:16:00,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:16:00,305 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 11:16:00,331 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:16:00,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:16:01,056 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:16:01,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:16:02,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:16:02,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:16:02,562 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 11:16:02,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:16:02,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:16:02,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:16:02,654 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 11:16:02,654 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:16:02,657 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:16:03,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:16:03,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:16:03,648 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:16:03,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:16:03,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:16:03,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:16:03,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:16:03,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:16:03,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:16:03,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:16:03,778 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:16:03,793 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 11:16:03,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:16:03,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:16:04,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:16:05,265 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:16:05,655 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:16:06,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:16:06,039 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 11:16:06,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:16:06,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:16:06,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:16:06,121 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 11:16:06,122 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:16:06,125 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:16:06,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:16:06,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:16:06,493 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:16:06,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:16:06,507 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:16:06,520 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:16:06,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:16:06,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:16:06,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:16:06,579 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:16:06,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:16:06,617 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 11:16:06,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:16:06,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:16:09,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:16:09,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:16:10,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:16:10,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:16:12,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:16:12,313 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:16:12,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:16:12,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:16:12,536 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 11:16:12,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:16:12,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:16:12,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:16:12,627 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 11:16:12,628 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:16:12,634 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 11:16:12,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:16:12,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 11:16:13,026 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:16:13,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 11:16:13,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 11:16:13,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 11:16:13,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 11:16:13,086 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 11:16:13,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 11:16:13,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 11:16:13,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:16:13,150 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 11:16:13,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:16:13,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:16:13,214 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:16:13,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:16:13,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:16:13,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:16:13,275 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 11:16:13,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:16:13,290 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:16:13,308 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:16:13,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:16:13,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:16:13,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:16:13,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:16:13,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:16:13,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:16:13,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:16:13,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:16:13,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:16:13,474 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:16:13,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:16:13,507 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:16:13,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:16:13,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:16:13,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:16:13,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:16:13,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:16:13,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:16:13,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:16:13,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:16:13,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:16:13,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:16:13,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:16:13,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:16:13,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:16:13,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:16:13,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:16:13,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:16:13,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:16:13,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:16:13,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:16:13,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:16:13,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:16:13,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:16:13,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:16:13,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:16:13,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:16:13,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:16:14,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:16:14,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:16:14,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:16:14,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:16:14,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:16:14,096 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:16:14,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:16:14,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:16:14,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:16:14,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:16:14,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:16:14,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:16:14,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:16:14,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:16:14,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:16:14,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:16:14,285 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:16:14,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:16:14,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:16:14,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:16:14,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:16:14,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:16:14,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:16:14,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:16:14,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:16:14,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:16:14,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:16:14,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:16:14,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:16:14,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:16:14,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:16:14,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:16:14,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:16:14,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:16:14,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:16:14,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:16:14,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:16:14,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:16:14,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:16:14,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:16:14,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:16:14,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:16:14,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:16:14,721 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:16:14,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:16:14,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:16:14,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:16:14,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:16:14,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:16:14,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:16:14,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:16:14,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:16:14,888 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 11:16:14,890 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:16:14,892 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:16:15,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:16:15,244 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:16:15,309 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:16:15,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:16:15,323 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:16:15,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:16:15,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:16:15,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:16:15,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:16:15,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:16:15,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:15,973 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:16:15,973 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.08399987221. 2022-02-08 11:16:15,973 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:16:15,976 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:16:16,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:16:16,285 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:16:16,352 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:16:16,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:16:16,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:16:16,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:16:16,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:16:16,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:16:16,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:16:16,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:16:16,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:16,713 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:16,714 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:18,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:18,861 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:18,862 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:20,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:21,043 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:21,043 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:23,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:23,190 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:23,190 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:25,191 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:25,371 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:25,372 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:27,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:27,535 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:27,536 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:29,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:29,694 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:29,694 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:31,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:31,848 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:31,849 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:33,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:34,000 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:34,000 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:36,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:36,167 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:36,167 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:38,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:38,424 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:38,424 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:40,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:40,592 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:40,592 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:42,594 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:42,757 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:42,757 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:44,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:44,917 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:44,918 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:46,918 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:47,128 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:47,128 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:49,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:49,286 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:49,286 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 14378 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:16:51,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:51,446 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:51,447 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:53,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:53,628 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:53,628 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:55,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:55,786 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:55,786 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:16:57,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:16:58,062 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:16:58,062 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:17:00,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:17:00,236 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:17:00,236 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:17:02,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:17:02,391 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:17:02,392 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:17:04,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:17:04,749 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:17:04,749 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:17:06,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:17:06,967 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:17:06,967 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:17:08,969 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:17:09,170 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:17:09,171 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:17:11,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:17:11,444 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:17:11,444 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 55.4700000286. 2022-02-08 11:17:11,444 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:17:11,446 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:17:12,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:17:12,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:17:12,950 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:17:12,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:17:12,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:17:12,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:17:13,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:17:13,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:17:13,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:17:13,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:17:13,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:17:13,415 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:17:13,415 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.97199988365. 2022-02-08 11:17:13,415 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:17:13,417 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:17:14,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:17:14,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:17:14,651 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:17:14,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:17:14,668 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:17:14,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:17:14,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:17:14,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:17:14,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:17:14,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:17:14,834 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:17:15,460 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:17:15,460 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.04500007629. 2022-02-08 11:17:15,460 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:17:15,463 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:17:16,270 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:17:16,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:17:16,671 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:17:16,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:17:16,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:17:16,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:17:16,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:17:16,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:17:16,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:17:16,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:17:16,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:17:17,743 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:17:17,743 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.28299999237. 2022-02-08 11:17:17,743 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:17:17,746 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:17:18,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:17:19,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:17:19,345 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:17:19,346 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:17:19,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:17:19,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:17:19,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:17:19,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:17:19,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:17:19,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:17:19,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:17:21,479 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:17:21,480 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.73600006104. 2022-02-08 11:17:21,480 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:17:21,482 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:17:22,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:17:22,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:17:22,535 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:17:22,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:17:22,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:17:22,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:17:22,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:17:22,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:17:22,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:17:22,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:17:22,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:17:23,155 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:17:23,155 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.67599987984. 2022-02-08 11:17:23,155 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:17:23,157 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:17:23,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:17:23,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:17:23,861 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:17:23,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:17:23,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:17:23,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:17:24,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:17:24,022 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:17:24,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:17:24,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:17:24,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:17:25,167 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:17:25,167 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.0110001564. 2022-02-08 11:17:25,167 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:17:25,168 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:17:25,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:17:26,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:17:26,200 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:17:26,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:17:26,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:17:26,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:17:26,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:17:26,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:17:26,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:17:26,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:17:26,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:17:26,888 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:17:26,890 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.72199988365. ############################################################ # # RAM FootPrint: 4748.05859375 # ############################################################ 2022-02-08 11:17:26,924 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 11:17:26,924 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 90 seconds. Reason: Iteration:10 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14390 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 14416 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:18:56,931 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 11:18:56,931 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 11:19:01,934 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:19:02,183 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:02,482 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:19:02,578 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:02,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:19:02,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:19:02,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:19:02,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:19:02,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:19:02,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:19:02,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:19:02,762 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:19:02,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:19:02,861 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['57196 memcached\n'], []) 2022-02-08 11:19:02,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 11:19:02,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:19:02,963 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['57196 memcached\n'], []) 2022-02-08 11:19:02,963 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:19:02,966 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:19:03,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:03,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:19:03,984 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:03,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:19:04,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:19:04,022 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:19:04,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:19:04,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:19:04,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:19:04,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:19:04,108 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:19:04,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:19:04,155 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['111505 memcached\n'], []) 2022-02-08 11:19:04,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 11:19:04,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:19:04,200 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['111505 memcached\n'], []) 2022-02-08 11:19:04,200 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:19:04,203 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:19:04,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:04,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:19:04,562 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:04,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:19:04,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:19:04,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:19:04,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:19:04,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:19:04,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:19:04,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:19:04,687 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:19:04,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:19:04,743 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['94756 memcached\n'], []) 2022-02-08 11:19:04,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 11:19:04,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:19:04,792 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['94756 memcached\n'], []) 2022-02-08 11:19:04,793 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:19:04,796 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:19:04,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:05,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:19:05,170 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:05,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:19:05,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:19:05,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:19:05,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:19:05,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:19:05,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:19:05,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:19:05,263 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:19:05,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:19:05,303 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['121393 memcached\n'], []) 2022-02-08 11:19:05,303 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 11:19:05,329 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:19:05,349 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['121393 memcached\n'], []) 2022-02-08 11:19:05,351 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:19:05,352 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:19:05,443 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:05,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:19:05,721 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:05,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:19:05,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:19:05,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:19:05,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:19:05,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:19:05,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:19:05,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:19:05,818 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:19:05,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:19:05,868 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['115258 memcached\n'], []) 2022-02-08 11:19:05,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 11:19:05,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:19:05,928 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['115258 memcached\n'], []) 2022-02-08 11:19:05,930 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:19:05,931 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:19:06,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:06,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:19:06,295 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:06,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:19:06,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:19:06,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:19:06,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:19:06,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:19:06,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:19:06,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:19:06,391 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:19:06,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:19:06,456 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['19715 memcached\n'], []) 2022-02-08 11:19:06,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 11:19:06,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:19:06,503 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['19715 memcached\n'], []) 2022-02-08 11:19:06,503 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:19:06,506 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:19:06,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:06,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:19:06,871 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:06,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:19:06,887 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:19:06,900 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:19:06,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:19:06,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:19:06,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:19:06,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:19:06,966 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:19:06,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:19:07,012 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['83959 memcached\n'], []) 2022-02-08 11:19:07,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 11:19:07,038 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:19:07,059 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['83959 memcached\n'], []) 2022-02-08 11:19:07,059 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:19:07,061 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:19:07,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:07,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:19:07,428 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:07,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:19:07,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:19:07,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:19:07,469 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:19:07,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:19:07,497 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:19:07,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:19:07,526 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:19:07,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:19:07,604 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['99143 memcached\n'], []) 2022-02-08 11:19:07,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 11:19:07,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:19:07,664 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['99143 memcached\n'], []) 2022-02-08 11:19:07,664 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:19:07,667 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:19:07,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:07,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:19:08,019 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:08,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:19:08,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:19:08,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:19:08,061 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:19:08,079 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:19:08,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:19:08,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:19:08,119 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:19:08,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:19:08,161 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['35621 memcached\n'], []) 2022-02-08 11:19:08,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 11:19:08,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:19:08,207 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['35621 memcached\n'], []) 2022-02-08 11:19:08,207 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:19:08,209 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:19:08,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:08,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:19:08,595 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:08,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:19:08,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:19:08,624 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:19:08,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:19:08,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:19:08,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:19:08,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:19:08,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:19:08,714 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 11:19:08,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:19:08,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:11,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:19:12,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:13,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:13,543 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:14,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:19:14,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:19:14,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:19:14,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:19:14,622 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 11:19:14,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:19:14,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:14,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:19:14,799 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 11:19:14,799 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:19:14,802 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:19:14,887 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:15,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:19:15,144 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:15,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:19:15,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:19:15,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:19:15,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:19:15,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:19:15,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:19:15,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:19:15,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:19:15,250 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 11:19:15,280 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:19:15,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:22,413 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:19:23,272 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:23,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:24,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:24,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:19:24,428 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:19:24,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:19:24,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:19:24,522 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 11:19:24,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:19:24,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:24,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:19:24,624 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 11:19:24,625 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:19:24,627 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 11:19:24,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:24,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 11:19:25,012 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:25,012 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 11:19:25,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 11:19:25,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 11:19:25,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 11:19:25,079 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 11:19:25,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 11:19:25,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 11:19:25,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:19:25,138 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 11:19:25,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:19:25,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:25,201 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:19:25,221 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:25,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:25,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:19:25,265 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 11:19:25,265 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:19:25,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:25,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:19:25,308 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:25,323 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:19:25,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:25,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:25,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:25,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:25,403 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:25,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:25,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:25,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:25,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:25,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:19:25,505 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:25,520 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:25,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:25,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:25,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:25,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:25,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:25,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:25,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:25,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:19:25,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:25,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:25,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:25,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:25,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:25,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:25,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:25,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:25,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:25,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:19:25,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:25,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:25,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:25,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:25,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:25,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:25,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:25,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:25,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:26,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:19:26,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:26,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:26,071 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:26,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:26,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:26,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:26,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:26,170 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:26,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:26,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:19:26,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:26,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:26,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:26,270 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:26,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:26,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:26,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:26,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:26,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:26,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:19:26,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:26,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:26,426 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:26,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:26,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:26,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:26,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:26,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:26,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:26,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:19:26,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:26,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:26,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:26,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:26,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:26,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:26,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:26,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:26,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:26,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:19:26,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:26,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:26,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:26,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:26,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:26,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:26,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:26,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:26,852 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 11:19:26,855 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:19:27,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:27,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:19:27,681 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:27,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:19:27,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:19:27,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:19:27,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:19:27,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:19:27,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:19:27,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:19:27,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:19:27,799 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 11:19:27,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:19:27,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:28,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:19:29,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:29,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:29,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:19:29,730 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 11:19:29,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:19:29,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:29,811 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:19:29,825 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 11:19:29,825 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:19:29,828 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 11:19:29,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:30,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 11:19:30,219 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:30,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 11:19:30,232 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 11:19:30,244 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 11:19:30,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 11:19:30,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 11:19:30,298 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 11:19:30,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 11:19:30,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:19:30,339 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 11:19:30,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:19:30,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:30,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:19:30,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:30,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:30,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:19:30,446 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 11:19:30,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:19:30,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:30,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:19:30,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:30,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:19:30,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:30,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:30,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:30,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:30,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:30,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:30,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:30,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:30,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:30,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:19:30,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:30,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:30,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:30,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:30,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:30,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:30,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:30,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:30,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:30,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:19:30,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:30,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:30,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:30,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:30,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:30,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:30,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:30,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:30,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:30,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:19:30,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:30,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:31,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:31,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:31,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:31,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:31,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:31,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:31,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:31,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:19:31,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:31,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:31,170 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:31,186 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:31,201 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:31,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:31,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:31,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:31,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:31,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:19:31,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:31,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:31,367 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:31,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:31,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:31,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:31,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:31,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:31,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:31,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:19:31,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:31,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:31,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:31,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:31,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:31,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:31,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:31,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:31,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:31,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:19:31,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:31,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:31,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:31,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:31,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:31,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:31,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:31,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:31,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:31,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:19:31,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:31,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:31,834 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:31,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:31,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:31,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:31,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:31,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:31,926 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 11:19:31,927 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:19:32,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:32,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:19:32,290 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:32,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:19:32,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:19:32,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:19:32,329 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:19:32,346 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:19:32,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:19:32,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:19:32,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:19:32,411 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 11:19:32,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:19:32,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:35,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:19:36,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:37,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:37,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:38,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:19:38,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:19:38,836 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:19:38,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:19:38,924 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 11:19:38,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:19:38,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:38,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:19:39,009 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 11:19:39,009 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:19:39,012 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:19:39,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:39,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:19:39,377 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:39,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:19:39,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:19:39,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:19:39,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:19:39,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:19:39,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:19:39,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:19:39,487 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:19:39,506 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 11:19:39,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:19:39,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:45,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:19:46,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:47,295 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:47,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:48,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:19:48,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:19:48,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:19:48,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:19:48,436 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 11:19:48,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:19:48,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:48,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:19:48,568 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 11:19:48,568 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:19:48,569 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 11:19:48,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:48,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 11:19:49,017 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:49,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 11:19:49,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 11:19:49,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 11:19:49,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 11:19:49,096 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 11:19:49,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 11:19:49,132 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 11:19:49,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:19:49,173 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 11:19:49,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:19:49,221 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:49,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:19:49,257 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:49,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:49,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:19:49,305 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 11:19:49,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:19:49,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:49,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:19:49,357 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:49,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:19:49,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:49,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:49,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:49,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:49,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:49,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:49,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:49,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:49,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:49,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:19:49,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:49,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:49,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:49,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:49,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:49,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:49,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:49,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:49,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:49,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:19:49,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:49,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:49,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 5 queries submitted, 0 failed, 14433 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:19:49,836 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:49,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:49,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:49,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:49,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:49,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:49,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:19:50,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:50,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:50,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:50,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:50,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:50,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:50,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:50,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:50,178 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:50,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:19:50,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:50,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:50,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:50,280 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:50,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:50,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:50,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:50,362 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:50,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:50,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:19:50,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:50,446 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:50,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:50,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:50,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:50,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:50,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:50,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:50,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:50,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:19:50,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:50,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:50,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:50,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:50,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:50,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:50,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:50,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:50,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:50,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:19:50,839 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:50,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:50,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:50,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:50,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:50,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:50,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:50,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:51,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:51,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:19:51,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:51,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:51,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:51,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:51,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:51,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:51,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:51,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:51,207 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 11:19:51,210 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 11:19:51,290 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:51,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 11:19:51,598 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:51,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 11:19:51,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 11:19:51,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 11:19:51,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 11:19:51,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 11:19:51,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 11:19:51,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 11:19:51,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:19:51,711 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 11:19:51,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:19:51,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:51,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:19:51,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:51,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:19:51,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:19:51,822 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 11:19:51,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:19:51,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:19:51,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:19:51,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:51,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:19:51,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:51,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:51,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:51,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:51,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:51,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:52,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:52,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:19:52,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:52,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:19:52,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:52,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:52,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:52,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:52,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:52,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:52,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:52,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:19:52,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:52,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:19:52,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:52,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:52,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:52,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:52,323 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:52,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:52,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:52,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:19:52,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:52,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:19:52,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:52,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:52,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:52,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:52,505 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:52,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:52,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:52,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:19:52,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:52,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:19:52,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:52,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:52,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:52,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:52,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:52,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:52,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:52,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:19:52,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:52,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:19:52,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:52,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:52,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:52,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:52,836 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:52,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:52,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:52,887 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:19:52,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:52,918 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:19:52,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:52,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:52,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:52,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:53,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:53,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:53,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:53,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:19:53,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:53,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:19:53,096 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:53,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:53,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:53,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:53,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:53,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:53,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:53,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:19:53,232 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:53,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:19:53,270 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:53,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:53,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:53,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:53,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:53,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:53,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:53,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:19:53,404 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 11:19:53,405 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:19:53,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:19:54,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:19:54,928 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:19:54,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:19:55,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:19:55,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:19:55,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:19:55,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:19:55,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:19:55,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:19:55,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:19:56,118 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 11:19:56,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:19:56,178 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:20:02,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:20:03,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:20:04,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:20:04,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:20:04,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:20:05,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:20:05,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:20:05,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:20:05,210 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 11:20:05,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:20:05,227 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:20:05,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:20:05,325 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 11:20:05,325 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:20:05,326 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:20:05,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:20:05,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:20:05,819 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:20:05,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:20:05,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:20:05,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:20:05,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:20:05,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:20:06,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:20:06,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:20:06,147 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:20:06,186 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 11:20:06,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:20:06,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:20:06,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:20:07,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:20:08,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:20:08,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:20:08,661 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 11:20:08,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:20:08,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:20:08,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:20:08,742 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 11:20:08,743 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:20:08,744 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:20:09,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:20:09,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:20:09,806 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:20:09,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:20:09,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:20:09,836 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:20:09,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:20:09,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:20:09,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:20:09,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:20:09,956 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:20:09,971 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 11:20:10,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:20:10,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:20:12,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:20:12,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:20:13,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:20:13,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:20:13,505 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 11:20:13,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:20:13,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:20:13,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:20:13,585 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 11:20:13,586 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:20:13,588 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:20:13,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:20:13,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:20:14,032 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:20:14,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:20:14,061 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:20:14,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:20:14,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:20:14,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:20:14,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:20:14,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:20:14,183 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:20:14,203 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 11:20:14,232 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:20:14,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:20:18,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:20:19,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:20:20,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:20:20,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:20:20,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:20:20,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:20:21,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:20:21,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:20:21,239 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 11:20:21,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:20:21,257 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:20:21,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:20:21,315 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 11:20:21,315 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:20:21,316 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 11:20:21,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:20:21,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 11:20:21,697 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:20:21,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 11:20:21,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 11:20:21,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 11:20:21,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 11:20:21,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 11:20:21,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 11:20:21,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 11:20:21,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:20:21,825 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 11:20:21,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:20:21,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:20:21,885 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:20:21,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:20:21,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:20:21,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:20:21,950 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 11:20:21,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:20:21,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:20:21,979 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:20:21,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:20:22,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:20:22,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:20:22,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:20:22,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:20:22,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:20:22,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:20:22,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:20:22,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:20:22,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:20:22,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:20:22,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:20:22,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:20:22,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:20:22,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:20:22,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:20:22,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:20:22,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:20:22,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:20:22,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:20:22,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:20:22,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:20:22,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:20:22,403 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:20:22,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:20:22,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:20:22,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:20:22,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:20:22,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:20:22,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:20:22,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:20:22,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:20:22,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:20:22,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:20:22,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:20:22,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:20:22,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:20:22,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:20:22,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:20:22,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:20:22,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:20:22,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:20:22,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:20:22,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:20:22,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:20:22,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:20:22,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:20:22,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:20:22,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:20:22,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:20:22,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:20:22,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:20:22,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:20:22,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:20:22,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:20:22,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:20:23,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:20:23,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:20:23,043 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:20:23,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:20:23,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:20:23,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:20:23,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:20:23,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:20:23,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:20:23,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:20:23,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:20:23,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:20:23,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:20:23,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:20:23,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:20:23,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:20:23,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:20:23,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:20:23,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:20:23,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:20:23,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:20:23,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:20:23,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:20:23,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:20:23,403 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:20:23,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:20:23,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:20:23,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:20:23,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:20:23,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:20:23,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:20:23,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:20:23,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:20:23,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:20:23,612 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 11:20:23,614 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:20:23,615 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:20:23,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:20:23,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:20:23,996 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:20:23,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:20:24,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:20:24,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:20:24,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:20:24,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:20:24,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:20:24,086 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:20:24,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:20:24,638 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:20:24,638 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.02499985695. 2022-02-08 11:20:24,638 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:20:24,641 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:20:24,969 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:20:25,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:20:25,440 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:20:25,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:20:25,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:20:25,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:20:25,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:20:25,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:20:25,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:20:25,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:20:25,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:20:26,464 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:20:26,464 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:20:28,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:20:28,697 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:20:28,697 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:20:30,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:20:30,986 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:20:30,986 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:20:32,987 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:20:33,183 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:20:33,184 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:20:35,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:20:35,483 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:20:35,483 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:20:37,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:20:37,756 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:20:37,756 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:20:39,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:20:39,997 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:20:39,999 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:20:41,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:20:42,211 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:20:42,213 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:20:44,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:20:44,467 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:20:44,467 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:20:46,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:20:46,813 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:20:46,813 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:20:48,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:20:49,038 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:20:49,038 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 14433 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:20:51,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:20:51,203 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:20:51,204 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:20:53,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:20:53,371 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:20:53,371 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:20:55,374 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:20:55,517 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:20:55,529 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 30.8910000324. 2022-02-08 11:20:55,529 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:20:55,532 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:20:56,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:20:56,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:20:56,836 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:20:56,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:20:57,038 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:20:57,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:20:57,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:20:57,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:20:57,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:20:57,627 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:20:57,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:20:58,466 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:20:58,466 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 11:21:00,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:21:00,648 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:21:00,648 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 11:21:02,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:21:02,823 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:21:02,825 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 11:21:04,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:21:04,986 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:21:04,986 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 9.45700001717. 2022-02-08 11:21:04,987 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:21:04,990 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:21:05,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:21:05,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:21:06,032 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:21:06,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:21:06,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:21:06,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:21:06,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:21:06,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:21:06,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:21:06,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:21:06,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:21:06,732 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:21:06,733 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.74500012398. 2022-02-08 11:21:06,733 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:21:06,736 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:21:07,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:21:08,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:21:08,292 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:21:08,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:21:08,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:21:08,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:21:08,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:21:08,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:21:08,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:21:08,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:21:08,446 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:21:09,369 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:21:09,371 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.63800001144. 2022-02-08 11:21:09,371 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:21:09,372 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:21:10,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:21:10,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:21:10,713 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:21:10,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:21:10,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:21:10,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:21:10,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:21:10,795 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:21:10,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:21:10,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:21:10,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:21:14,269 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:21:14,269 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.89800000191. 2022-02-08 11:21:14,269 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:21:14,270 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:21:14,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:21:15,002 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:21:15,094 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:21:15,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:21:15,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:21:15,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:21:15,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:21:15,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:21:15,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:21:15,196 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:21:15,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:21:15,772 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:21:15,773 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.5039999485. 2022-02-08 11:21:15,773 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:21:15,775 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:21:16,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:21:16,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:21:16,703 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:21:16,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:21:16,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:21:16,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:21:16,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:21:16,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:21:16,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:21:16,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:21:16,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:21:17,588 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:21:17,588 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.81599998474. 2022-02-08 11:21:17,588 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:21:17,591 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:21:18,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:21:18,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:21:18,601 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:21:18,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:21:18,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:21:18,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:21:18,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:21:18,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:21:18,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:21:18,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:21:18,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:21:20,177 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:21:20,178 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.58999991417. ############################################################ # # RAM FootPrint: 4828.17578125 # ############################################################ 2022-02-08 11:21:20,210 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 11:21:20,210 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 88 seconds. Reason: Iteration:11 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14452 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:22:48,217 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 11:22:48,217 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 5 queries submitted, 0 failed, 14477 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:22:53,220 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:22:53,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:22:53,579 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:22:53,663 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:22:53,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:22:53,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:22:53,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:22:53,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:22:53,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:22:53,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:22:53,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:22:53,854 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:22:53,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:22:53,940 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['58036 memcached\n'], []) 2022-02-08 11:22:53,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 11:22:53,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:22:54,017 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['58036 memcached\n'], []) 2022-02-08 11:22:54,019 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:22:54,022 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:22:54,729 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:22:54,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:22:55,107 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:22:55,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:22:55,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:22:55,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:22:55,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:22:55,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:22:55,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:22:55,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:22:55,221 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:22:55,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:22:55,282 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['112553 memcached\n'], []) 2022-02-08 11:22:55,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 11:22:55,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:22:55,335 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['112553 memcached\n'], []) 2022-02-08 11:22:55,336 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:22:55,338 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:22:55,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:22:56,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:22:56,309 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:22:56,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:22:56,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:22:56,395 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:22:56,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:22:56,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:22:56,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:22:56,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:22:56,476 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:22:56,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:22:56,543 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['95575 memcached\n'], []) 2022-02-08 11:22:56,543 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 11:22:56,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:22:56,592 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['95575 memcached\n'], []) 2022-02-08 11:22:56,594 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:22:56,595 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:22:56,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:22:56,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:22:56,969 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:22:56,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:22:56,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:22:57,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:22:57,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:22:57,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:22:57,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:22:57,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:22:57,092 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:22:57,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:22:57,138 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['122143 memcached\n'], []) 2022-02-08 11:22:57,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 11:22:57,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:22:57,207 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['122143 memcached\n'], []) 2022-02-08 11:22:57,207 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:22:57,211 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:22:57,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:22:57,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:22:57,565 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:22:57,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:22:57,581 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:22:57,594 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:22:57,607 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:22:57,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:22:57,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:22:57,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:22:57,670 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:22:57,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:22:57,716 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['116005 memcached\n'], []) 2022-02-08 11:22:57,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 11:22:57,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:22:57,759 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['116005 memcached\n'], []) 2022-02-08 11:22:57,760 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:22:57,763 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:22:57,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:22:58,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:22:58,138 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:22:58,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:22:58,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:22:58,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:22:58,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:22:58,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:22:58,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:22:58,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:22:58,240 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:22:58,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:22:58,286 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['20433 memcached\n'], []) 2022-02-08 11:22:58,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 11:22:58,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:22:58,332 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['20433 memcached\n'], []) 2022-02-08 11:22:58,334 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:22:58,335 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:22:58,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:22:58,647 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:22:58,711 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:22:58,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:22:58,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:22:58,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:22:58,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:22:58,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:22:58,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:22:58,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:22:58,805 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:22:58,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:22:58,851 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['84680 memcached\n'], []) 2022-02-08 11:22:58,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 11:22:58,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:22:58,900 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['84680 memcached\n'], []) 2022-02-08 11:22:58,901 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:22:58,903 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:22:58,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:22:59,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:22:59,273 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:22:59,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:22:59,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:22:59,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:22:59,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:22:59,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:22:59,346 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:22:59,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:22:59,374 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:22:59,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:22:59,436 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['99865 memcached\n'], []) 2022-02-08 11:22:59,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 11:22:59,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:22:59,482 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['99865 memcached\n'], []) 2022-02-08 11:22:59,483 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:22:59,483 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:22:59,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:22:59,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:22:59,842 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:22:59,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:22:59,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:22:59,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:22:59,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:22:59,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:22:59,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:22:59,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:22:59,948 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:22:59,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:23:00,012 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['36324 memcached\n'], []) 2022-02-08 11:23:00,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 11:23:00,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:23:00,078 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['36324 memcached\n'], []) 2022-02-08 11:23:00,078 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:23:00,082 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:23:00,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:23:00,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:23:00,444 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:23:00,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:23:00,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:23:00,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:23:00,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:23:00,505 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:23:00,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:23:00,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:23:00,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:23:00,576 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 11:23:00,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:23:00,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:05,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:23:06,428 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:06,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:07,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:07,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:23:07,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:23:07,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:23:07,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:23:07,842 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 11:23:07,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:23:07,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:07,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:23:07,951 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 11:23:07,951 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:23:07,956 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:23:08,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:23:08,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:23:08,319 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:23:08,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:23:08,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:23:08,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:23:08,362 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:23:08,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:23:08,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:23:08,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:23:08,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:23:08,436 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 11:23:08,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:23:08,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:11,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:23:12,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:12,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:13,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:13,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:23:13,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:23:13,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:23:13,897 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:23:13,910 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 11:23:13,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:23:13,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:14,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:23:14,028 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 11:23:14,029 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:23:14,032 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 11:23:14,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:23:14,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 11:23:14,427 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:23:14,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 11:23:14,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 11:23:14,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 11:23:14,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 11:23:14,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 11:23:14,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 11:23:14,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 11:23:14,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:23:14,549 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 11:23:14,579 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:23:14,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:14,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:23:14,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:14,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:14,660 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:23:14,678 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 11:23:14,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:23:14,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:14,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:23:14,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:14,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:23:14,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:14,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:14,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:14,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:14,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:14,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:14,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:14,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:14,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:14,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:23:14,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:14,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:14,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:14,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:15,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:15,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:15,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:15,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:15,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:15,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:23:15,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:15,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:15,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:15,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:15,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:15,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:15,244 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:15,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:15,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:15,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:23:15,323 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:15,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:15,362 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:15,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:15,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:15,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:15,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:15,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:15,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:15,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:23:15,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:15,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:15,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:15,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:15,579 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:15,596 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:15,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:15,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:15,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:15,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:23:15,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:15,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:15,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:15,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:15,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:15,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:15,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:15,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:15,839 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:15,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:23:15,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:15,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:15,910 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:15,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:15,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:15,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:15,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:15,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:16,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:16,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:23:16,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:16,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:16,071 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:16,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:16,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:16,127 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:16,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:16,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:16,173 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:16,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:23:16,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:16,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:16,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:16,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:16,272 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:16,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:16,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:16,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:16,342 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 11:23:16,345 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:23:16,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:23:16,971 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:23:17,548 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:23:17,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:23:17,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:23:17,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:23:18,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:23:18,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:23:18,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:23:18,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:23:18,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:23:18,414 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 11:23:18,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:23:18,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:20,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:23:21,323 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:21,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:22,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:23:22,032 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 11:23:22,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:23:22,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:22,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:23:22,134 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 11:23:22,134 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:23:22,137 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 11:23:22,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:23:22,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 11:23:22,502 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:23:22,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 11:23:22,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 11:23:22,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 11:23:22,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 11:23:22,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 11:23:22,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 11:23:22,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 11:23:22,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:23:22,605 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 11:23:22,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:23:22,645 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:22,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:23:22,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:22,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:22,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:23:22,710 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 11:23:22,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:23:22,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:22,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:23:22,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:22,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:23:22,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:22,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:22,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:22,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:22,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:22,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:22,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:22,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:22,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:22,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:23:22,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:22,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:22,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:22,989 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:23,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:23,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:23,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:23,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:23,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:23,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:23:23,096 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:23,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:23,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:23,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:23,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:23,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:23,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:23,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:23,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:23,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:23:23,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:23,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:23,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:23,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:23,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:23,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:23,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:23,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:23,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:23,395 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:23:23,413 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:23,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:23,443 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:23,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:23,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:23,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:23,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:23,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:23,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:23,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:23:23,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:23,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:23,596 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:23,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:23,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:23,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:23,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:23,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:23,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:23,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:23:23,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:23,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:23,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:23,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:23,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:23,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:23,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:23,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:23,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:23,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:23:23,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:23,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:23,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:23,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:23,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:23,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:23,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:23,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:24,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:24,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:23:24,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:24,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:24,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:24,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:24,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:24,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:24,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:24,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:24,171 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 11:23:24,173 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:23:24,265 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:23:24,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:23:24,532 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:23:24,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:23:24,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:23:24,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:23:24,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:23:24,596 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:23:24,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:23:24,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:23:24,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:23:24,658 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 11:23:24,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:23:24,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:28,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:23:29,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:29,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:30,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:30,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:23:30,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:23:30,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:23:30,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:23:30,894 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 11:23:30,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:23:30,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:30,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:23:30,974 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 11:23:30,974 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:23:30,977 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:23:31,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:23:31,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:23:31,359 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:23:31,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:23:31,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:23:31,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:23:31,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:23:31,428 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:23:31,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:23:31,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:23:31,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:23:31,496 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 11:23:31,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:23:31,543 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:34,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:23:35,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:36,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:36,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:37,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:23:37,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:23:37,505 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:23:37,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:23:37,585 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 11:23:37,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:23:37,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:37,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:23:37,710 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 11:23:37,710 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:23:37,713 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 11:23:37,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:23:38,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 11:23:38,131 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:23:38,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 11:23:38,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 11:23:38,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 11:23:38,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 11:23:38,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 11:23:38,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 11:23:38,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 11:23:38,265 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:23:38,285 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 11:23:38,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:23:38,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:38,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:23:38,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:38,397 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:38,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:23:38,434 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 11:23:38,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:23:38,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:38,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:23:38,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:38,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:23:38,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:38,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:38,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:38,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:38,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:38,647 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:38,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:38,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:38,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:38,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:23:38,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:38,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:38,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:38,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:38,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:38,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:38,885 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:38,910 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:38,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:38,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:23:38,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:38,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:39,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:39,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:39,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:39,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:39,119 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:39,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:39,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:39,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:23:39,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:39,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:39,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:39,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:39,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:39,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:39,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:39,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:39,395 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:39,426 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:23:39,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:39,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:39,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:39,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:39,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:39,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:39,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:39,596 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:39,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:39,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:23:39,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:39,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:39,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:39,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:39,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:39,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:39,795 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:39,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:39,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:39,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:23:39,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:39,897 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:39,915 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:39,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:39,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:39,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:39,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:40,017 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:40,038 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:40,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:23:40,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:40,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:40,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:40,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:40,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:40,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:40,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:40,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:40,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:40,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:23:40,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:40,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:40,346 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:40,367 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:40,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:40,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:40,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:40,459 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:40,486 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 11:23:40,487 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 11:23:40,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:23:40,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 11:23:40,871 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:23:40,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 11:23:40,887 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 11:23:40,900 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 11:23:40,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 11:23:40,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 11:23:40,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 11:23:40,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 11:23:40,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:23:40,993 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 11:23:41,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:23:41,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:41,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:23:41,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:41,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:41,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:23:41,108 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 11:23:41,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:23:41,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:41,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:23:41,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:41,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:23:41,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:41,214 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:41,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:41,255 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:41,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:41,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:41,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:41,329 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:23:41,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:41,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:23:41,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:41,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:41,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:41,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:41,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:41,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:41,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:41,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:23:41,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:41,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:23:41,556 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:41,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:41,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:41,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:41,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:41,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:41,658 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:41,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:23:41,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:41,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:23:41,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:41,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:41,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:41,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:41,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:41,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:41,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:41,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:23:41,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:41,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:23:41,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:41,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:41,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:41,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:42,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:42,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:42,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:42,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:23:42,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:42,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:23:42,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:42,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:42,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:42,160 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:42,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:42,191 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:42,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:42,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:23:42,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:42,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:23:42,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:42,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:42,306 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:42,323 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:42,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:42,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:42,374 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:42,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:23:42,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:42,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:23:42,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:42,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:42,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:42,487 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:42,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:42,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:42,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:42,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:23:42,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:42,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:23:42,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:42,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:42,640 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:42,655 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:42,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:42,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:42,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:42,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:23:42,743 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 11:23:42,746 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:23:42,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:23:43,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:23:43,201 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:23:43,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:23:43,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:23:43,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:23:43,270 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:23:43,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:23:43,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:23:43,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:23:43,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:23:43,378 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 11:23:43,403 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:23:43,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:45,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:23:46,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:47,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:47,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:47,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:23:48,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:23:48,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:23:48,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:23:48,213 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 11:23:48,214 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:23:48,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:48,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:23:48,319 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 11:23:48,319 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:23:48,321 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:23:48,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:23:48,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:23:48,969 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:23:48,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:23:48,989 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:23:49,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:23:49,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:23:49,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:23:49,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:23:49,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:23:49,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:23:49,209 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 11:23:49,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:23:49,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 14487 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:23:51,839 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:23:52,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:53,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:53,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:23:53,585 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 11:23:53,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:23:53,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:53,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:23:53,667 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 11:23:53,668 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:23:53,671 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:23:54,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:23:54,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:23:54,638 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:23:54,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:23:54,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:23:54,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:23:54,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:23:54,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:23:54,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:23:54,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:23:54,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:23:54,825 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 11:23:54,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:23:54,874 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:55,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:23:56,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:56,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:23:57,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:23:57,203 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 11:23:57,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:23:57,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:57,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:23:57,286 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 11:23:57,288 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:23:57,289 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:23:57,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:23:57,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:23:57,723 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:23:57,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:23:57,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:23:57,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:23:57,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:23:57,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:23:57,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:23:57,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:23:57,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:23:57,858 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 11:23:57,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:23:57,900 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:23:59,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:24:00,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:24:01,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:24:01,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:24:03,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:24:03,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:24:03,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:24:03,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:24:03,826 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 11:24:03,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:24:03,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:24:03,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:24:03,966 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 11:24:03,967 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:24:03,970 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 11:24:04,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:24:04,285 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 11:24:04,355 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:24:04,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 11:24:04,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 11:24:04,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 11:24:04,395 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 11:24:04,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 11:24:04,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 11:24:04,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 11:24:04,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:24:04,467 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 11:24:04,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:24:04,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:24:04,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:24:04,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:24:04,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:24:04,573 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:24:04,588 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 11:24:04,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:24:04,604 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:24:04,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:24:04,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:24:04,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:24:04,668 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:24:04,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:24:04,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:24:04,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:24:04,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:24:04,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:24:04,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:24:04,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:24:04,839 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:24:04,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:24:04,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:24:04,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:24:04,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:24:04,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:24:04,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:24:04,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:24:04,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:24:05,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:24:05,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:24:05,043 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:24:05,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:24:05,079 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:24:05,096 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:24:05,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:24:05,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:24:05,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:24:05,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:24:05,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:24:05,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:24:05,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:24:05,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:24:05,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:24:05,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:24:05,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:24:05,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:24:05,331 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:24:05,349 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:24:05,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:24:05,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:24:05,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:24:05,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:24:05,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:24:05,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:24:05,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:24:05,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:24:05,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:24:05,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:24:05,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:24:05,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:24:05,584 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:24:05,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:24:05,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:24:05,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:24:05,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:24:05,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:24:05,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:24:05,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:24:05,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:24:05,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:24:05,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:24:05,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:24:05,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:24:05,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:24:05,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:24:05,834 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:24:05,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:24:05,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:24:05,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:24:05,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:24:05,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:24:05,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:24:05,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:24:05,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:24:05,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:24:06,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:24:06,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:24:06,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:24:06,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:24:06,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:24:06,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:24:06,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:24:06,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:24:06,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:24:06,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:24:06,183 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:24:06,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:24:06,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:24:06,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:24:06,255 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 11:24:06,256 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:24:06,259 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:24:06,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:24:06,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:24:06,653 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:24:06,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:24:06,668 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:24:06,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:24:06,696 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:24:06,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:24:06,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:24:06,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:24:06,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:07,256 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:24:07,257 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.00199985504. 2022-02-08 11:24:07,257 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:24:07,259 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:24:07,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:24:07,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:24:07,750 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:24:07,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:24:07,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:24:07,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:24:07,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:24:07,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:24:07,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:24:07,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:24:07,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:08,131 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:08,131 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:10,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:10,290 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:10,290 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:12,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:12,447 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:12,447 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:14,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:14,601 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:14,601 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:16,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:16,782 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:16,782 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:18,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:18,980 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:18,980 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:20,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:21,155 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:21,155 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:23,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:23,338 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:23,338 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:25,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:25,517 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:25,517 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:27,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:27,720 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:27,720 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:29,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:29,917 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:29,917 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:31,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:32,082 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:32,082 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:34,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:34,266 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:34,266 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:36,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:36,443 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:36,444 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:38,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:38,617 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:38,617 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:40,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:40,786 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:40,786 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:42,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:42,980 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:42,980 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:44,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:45,220 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:45,220 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:47,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:47,433 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:47,434 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:24:49,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:49,621 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:24:49,621 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 42.364000082. 2022-02-08 11:24:49,621 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:24:49,624 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 5 queries submitted, 0 failed, 14497 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:24:50,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:24:50,834 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:24:50,947 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:24:50,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:24:50,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:24:50,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:24:51,038 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:24:51,096 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:24:51,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:24:51,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:24:51,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:51,671 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:24:51,673 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 11:24:53,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:53,839 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:24:53,841 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.21900010109. 2022-02-08 11:24:53,841 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:24:53,842 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:24:54,579 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:24:54,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:24:55,003 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:24:55,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:24:55,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:24:55,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:24:55,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:24:55,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:24:55,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:24:55,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:24:55,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:55,706 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:24:55,706 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.86500000954. 2022-02-08 11:24:55,706 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:24:55,707 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:24:56,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:24:57,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:24:57,476 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:24:57,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:24:57,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:24:57,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:24:57,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:24:57,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:24:57,573 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:24:57,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:24:57,624 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:24:58,536 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:24:58,536 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.83099985123. 2022-02-08 11:24:58,538 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:24:58,539 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:24:59,196 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:24:59,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:24:59,680 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:24:59,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:24:59,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:24:59,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:24:59,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:24:59,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:24:59,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:24:59,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:24:59,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:25:00,608 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:25:00,608 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.0720000267. 2022-02-08 11:25:00,608 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:25:00,611 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:25:01,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:25:01,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:25:02,032 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:25:02,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:25:02,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:25:02,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:25:02,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:25:02,102 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:25:02,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:25:02,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:25:02,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:25:02,654 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:25:02,655 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.04699993134. 2022-02-08 11:25:02,655 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:25:02,657 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:25:03,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:25:04,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:25:04,476 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:25:04,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:25:04,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:25:04,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:25:04,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:25:04,556 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:25:04,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:25:04,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:25:04,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:25:05,158 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:25:05,160 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.50400018692. 2022-02-08 11:25:05,160 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:25:05,161 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:25:06,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:25:06,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:25:06,822 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:25:06,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:25:06,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:25:06,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:25:06,900 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:25:06,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:25:06,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:25:06,969 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:25:06,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:25:08,207 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:25:08,207 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.04799985886. ############################################################ # # RAM FootPrint: 4807.953125 # ############################################################ 2022-02-08 11:25:08,240 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 11:25:08,240 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 88 seconds. Reason: Iteration:12 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14520 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:26:36,255 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 11:26:36,255 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 11:26:41,257 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:26:41,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:26:42,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:26:42,230 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:26:42,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:26:42,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:26:42,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:26:42,280 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:26:42,298 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:26:42,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:26:42,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:26:42,355 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:26:42,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:26:42,421 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['58887 memcached\n'], []) 2022-02-08 11:26:42,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 11:26:42,446 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:26:42,467 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['58887 memcached\n'], []) 2022-02-08 11:26:42,467 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:26:42,470 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:26:42,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:26:43,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:26:43,374 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:26:43,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:26:43,397 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:26:43,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:26:43,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:26:43,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:26:43,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:26:43,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:26:43,510 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:26:43,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:26:43,566 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['113426 memcached\n'], []) 2022-02-08 11:26:43,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 11:26:43,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:26:43,614 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['113426 memcached\n'], []) 2022-02-08 11:26:43,615 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:26:43,617 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:26:43,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:26:43,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:26:43,990 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:26:43,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:26:44,007 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:26:44,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:26:44,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:26:44,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:26:44,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:26:44,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:26:44,108 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:26:44,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:26:44,151 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['96476 memcached\n'], []) 2022-02-08 11:26:44,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 11:26:44,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:26:44,203 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['96476 memcached\n'], []) 2022-02-08 11:26:44,203 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:26:44,206 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:26:44,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:26:44,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:26:44,588 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:26:44,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:26:44,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:26:44,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:26:44,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:26:44,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:26:44,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:26:44,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:26:44,691 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:26:44,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:26:44,739 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['122891 memcached\n'], []) 2022-02-08 11:26:44,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 11:26:44,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:26:44,798 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['122891 memcached\n'], []) 2022-02-08 11:26:44,799 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:26:44,801 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:26:44,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:26:45,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:26:45,151 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:26:45,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:26:45,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:26:45,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:26:45,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:26:45,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:26:45,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:26:45,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:26:45,246 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:26:45,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:26:45,309 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['116755 memcached\n'], []) 2022-02-08 11:26:45,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 11:26:45,336 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:26:45,359 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['116755 memcached\n'], []) 2022-02-08 11:26:45,361 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:26:45,362 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:26:45,459 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:26:45,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:26:45,721 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:26:45,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:26:45,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:26:45,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:26:45,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:26:45,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:26:45,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:26:45,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:26:45,818 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:26:45,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:26:45,884 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['21181 memcached\n'], []) 2022-02-08 11:26:45,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 11:26:45,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:26:45,930 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['21181 memcached\n'], []) 2022-02-08 11:26:45,930 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:26:45,934 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:26:46,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:26:46,229 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:26:46,286 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:26:46,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:26:46,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:26:46,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:26:46,323 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:26:46,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:26:46,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:26:46,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:26:46,378 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:26:46,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:26:46,450 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['85409 memcached\n'], []) 2022-02-08 11:26:46,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 11:26:46,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:26:46,496 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['85409 memcached\n'], []) 2022-02-08 11:26:46,496 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:26:46,499 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:26:46,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:26:46,793 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:26:46,848 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:26:46,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:26:46,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:26:46,874 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:26:46,885 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:26:46,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:26:46,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:26:46,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:26:46,946 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:26:46,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:26:47,056 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['100585 memcached\n'], []) 2022-02-08 11:26:47,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 11:26:47,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:26:47,105 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['100585 memcached\n'], []) 2022-02-08 11:26:47,105 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:26:47,108 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:26:47,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:26:47,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:26:47,490 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:26:47,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:26:47,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:26:47,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:26:47,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:26:47,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:26:47,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:26:47,581 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:26:47,601 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:26:47,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:26:47,654 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['37081 memcached\n'], []) 2022-02-08 11:26:47,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 11:26:47,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:26:47,706 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['37081 memcached\n'], []) 2022-02-08 11:26:47,707 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:26:47,710 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:26:47,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:26:48,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:26:48,082 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:26:48,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:26:48,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:26:48,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:26:48,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:26:48,147 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:26:48,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:26:48,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:26:48,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:26:48,213 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 11:26:48,244 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:26:48,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 14541 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:26:50,854 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:26:51,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:26:52,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:26:52,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:26:53,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:26:53,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:26:53,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:26:53,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:26:53,881 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 11:26:53,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:26:53,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:26:53,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:26:53,983 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 11:26:53,983 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:26:53,986 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:26:54,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:26:54,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:26:54,362 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:26:54,362 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:26:54,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:26:54,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:26:54,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:26:54,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:26:54,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:26:54,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:26:54,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:26:54,483 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 11:26:54,510 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:26:54,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:26:59,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:27:00,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:01,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:01,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:02,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:27:02,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:27:02,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:27:02,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:27:02,255 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 11:27:02,255 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:27:02,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:02,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:27:02,378 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 11:27:02,378 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:27:02,380 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 11:27:02,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:27:02,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 11:27:02,770 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:27:02,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 11:27:02,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 11:27:02,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 11:27:02,816 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 11:27:02,836 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 11:27:02,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 11:27:02,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 11:27:02,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:27:02,901 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 11:27:02,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:27:02,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:02,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:27:02,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:02,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:03,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:27:03,023 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 11:27:03,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:27:03,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:03,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:27:03,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:03,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:27:03,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:03,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:03,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:03,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:03,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:03,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:03,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:03,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:03,265 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:03,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:27:03,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:03,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:03,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:03,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:03,367 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:03,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:03,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:03,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:03,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:03,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:27:03,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:03,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:03,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:03,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:03,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:03,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:03,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:03,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:03,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:03,627 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:27:03,645 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:03,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:03,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:03,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:03,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:03,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:03,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:03,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:03,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:03,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:27:03,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:03,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:03,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:03,887 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:03,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:03,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:03,938 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:03,956 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:03,971 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:03,989 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:27:04,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:04,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:04,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:04,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:04,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:04,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:04,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:04,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:04,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:04,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:27:04,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:04,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:04,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:04,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:04,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:04,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:04,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:04,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:04,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:04,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:27:04,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:04,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:04,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:04,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:04,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:04,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:04,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:04,474 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:04,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:04,510 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:27:04,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:04,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:04,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:04,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:04,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:04,624 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:04,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:04,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:04,678 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 11:27:04,680 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:27:04,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:27:05,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:27:05,836 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:27:05,836 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:27:05,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:27:06,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:27:06,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:27:06,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:27:06,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:27:06,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:27:06,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:27:07,142 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 11:27:07,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:27:07,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:09,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:27:10,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:10,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:10,989 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:27:11,007 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 11:27:11,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:27:11,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:11,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:27:11,108 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 11:27:11,108 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:27:11,111 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 11:27:11,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:27:11,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 11:27:11,460 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:27:11,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 11:27:11,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 11:27:11,487 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 11:27:11,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 11:27:11,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 11:27:11,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 11:27:11,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 11:27:11,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:27:11,569 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 11:27:11,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:27:11,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:11,627 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:27:11,640 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:11,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:11,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:27:11,678 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 11:27:11,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:27:11,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:11,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:27:11,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:11,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:27:11,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:11,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:11,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:11,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:11,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:11,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:11,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:11,874 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:11,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:11,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:27:11,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:11,936 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:11,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:11,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:11,982 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:12,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:12,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:12,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:12,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:12,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:27:12,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:12,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:12,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:12,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:12,173 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:12,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:12,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:12,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:12,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:12,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:27:12,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:12,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:12,303 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:12,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:12,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:12,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:12,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:12,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:12,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:12,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:27:12,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:12,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:12,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:12,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:12,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:12,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:12,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:12,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:12,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:12,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:27:12,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:12,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:12,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:12,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:12,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:12,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:12,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:12,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:12,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:12,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:27:12,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:12,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:12,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:12,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:12,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:12,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:12,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:12,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:12,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:12,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:27:12,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:12,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:12,936 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:12,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:12,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:12,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:12,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:13,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:13,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:13,043 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:27:13,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:13,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:13,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:13,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:13,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:13,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:13,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:13,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:13,188 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 11:27:13,191 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:27:13,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:27:13,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:27:13,528 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:27:13,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:27:13,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:27:13,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:27:13,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:27:13,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:27:13,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:27:13,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:27:13,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:27:13,644 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 11:27:13,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:27:13,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:17,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:27:18,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:18,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:19,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:20,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:27:20,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:27:20,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:27:20,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:27:20,881 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 11:27:20,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:27:20,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:20,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:27:20,964 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 11:27:20,966 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:27:20,967 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:27:21,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:27:21,308 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:27:21,372 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:27:21,374 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:27:21,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:27:21,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:27:21,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:27:21,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:27:21,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:27:21,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:27:21,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:27:21,513 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 11:27:21,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:27:21,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:27,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:27:28,987 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:29,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:30,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:31,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:27:31,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:27:31,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:27:31,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:27:31,822 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 11:27:31,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:27:31,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:31,966 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:27:31,990 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 11:27:31,990 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:27:31,993 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 11:27:32,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:27:32,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 11:27:32,493 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:27:32,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 11:27:32,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 11:27:32,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 11:27:32,556 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 11:27:32,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 11:27:32,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 11:27:32,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 11:27:32,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:27:32,664 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 11:27:32,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:27:32,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:32,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:27:32,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:32,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:32,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:27:32,811 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 11:27:32,811 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:27:32,829 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:32,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:27:32,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:32,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:27:32,913 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:32,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:32,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:32,979 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:33,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:33,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:33,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:33,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:33,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:33,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:27:33,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:33,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:33,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:33,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:33,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:33,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:33,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:33,326 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:33,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:33,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:27:33,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:33,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:33,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:33,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:33,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:33,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:33,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:33,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:33,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:33,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:27:33,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:33,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:33,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:33,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:33,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:33,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:33,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:33,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:33,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:33,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:27:33,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:33,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:33,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:33,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:33,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:34,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:34,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:34,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:34,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:34,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:27:34,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:34,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:34,198 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:34,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:34,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:34,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:34,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:34,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:34,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:34,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:27:34,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:34,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:34,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:34,443 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:34,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:34,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:34,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:34,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:34,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:34,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:27:34,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:34,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:34,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:34,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:34,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:34,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:34,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:34,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:34,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:34,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:27:34,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:34,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:34,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:34,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:34,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:34,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:34,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:34,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:34,996 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 11:27:35,000 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 11:27:35,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:27:35,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 11:27:35,372 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:27:35,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 11:27:35,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 11:27:35,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 11:27:35,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 11:27:35,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 11:27:35,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 11:27:35,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 11:27:35,494 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:27:35,525 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 11:27:35,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:27:35,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:35,594 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:27:35,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:35,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:35,640 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:27:35,655 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 11:27:35,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:27:35,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:35,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:27:35,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:35,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:27:35,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:35,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:35,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:35,795 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:35,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:35,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:35,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:35,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:27:35,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:35,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:27:35,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:35,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:35,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:35,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:36,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:36,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:36,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:36,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:27:36,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:36,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:27:36,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:36,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:36,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:36,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:36,183 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:36,201 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:36,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:36,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:27:36,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:36,272 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:27:36,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:36,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:36,331 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:36,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:36,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:36,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:36,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:36,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:27:36,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:36,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:27:36,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:36,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:36,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:36,543 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:36,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:36,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:36,594 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:36,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:27:36,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:36,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:27:36,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:36,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:36,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:36,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:36,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:36,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:36,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:36,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:27:36,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:36,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:27:36,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:36,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:36,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:36,885 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:36,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:36,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:36,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:36,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:27:36,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:36,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:27:37,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:37,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:37,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:37,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:37,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:37,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:37,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:37,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:27:37,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:37,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:27:37,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:37,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:37,224 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:37,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:37,257 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:37,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:37,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:37,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:27:37,325 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 11:27:37,328 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:27:37,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:27:37,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:27:37,858 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:27:37,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:27:37,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:27:37,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:27:37,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:27:37,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:27:37,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:27:38,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:27:38,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:27:38,088 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 11:27:38,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:27:38,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:43,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:27:44,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:45,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:45,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:45,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:27:45,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:27:45,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:27:45,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:27:45,960 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 11:27:45,961 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:27:45,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:46,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:27:46,058 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 11:27:46,058 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:27:46,062 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:27:46,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:27:46,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:27:46,473 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:27:46,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:27:46,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:27:46,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:27:46,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:27:46,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:27:46,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:27:46,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:27:46,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:27:46,621 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 11:27:46,645 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:27:46,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:49,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 5 queries submitted, 0 failed, 14547 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:27:50,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:51,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:51,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:27:51,509 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 11:27:51,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:27:51,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:51,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:27:51,582 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 11:27:51,582 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:27:51,585 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:27:52,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:27:52,579 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:27:52,693 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:27:52,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:27:52,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:27:52,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:27:52,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:27:52,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:27:52,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:27:52,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:27:52,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:27:52,835 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 11:27:52,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:27:52,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:54,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:27:55,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:55,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:27:56,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:27:56,048 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 11:27:56,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:27:56,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:27:56,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:27:56,128 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 11:27:56,128 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:27:56,131 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:27:56,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:27:56,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:27:56,694 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:27:56,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:27:56,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:27:56,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:27:56,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:27:56,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:27:56,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:27:56,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:27:56,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:27:56,905 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 11:27:56,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:27:57,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:28:03,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:28:04,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:28:05,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:28:05,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:28:06,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:28:07,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:28:07,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:28:07,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:28:07,532 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 11:28:07,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:28:07,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:28:07,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:28:07,683 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 11:28:07,684 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:28:07,686 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 11:28:07,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:28:08,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 11:28:08,098 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:28:08,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 11:28:08,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 11:28:08,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 11:28:08,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 11:28:08,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 11:28:08,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 11:28:08,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 11:28:08,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:28:08,240 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 11:28:08,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:28:08,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:28:08,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:28:08,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:28:08,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:28:08,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:28:08,362 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 11:28:08,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:28:08,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:28:08,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:28:08,410 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:28:08,428 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:28:08,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:28:08,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:28:08,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:28:08,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:28:08,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:28:08,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:28:08,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:28:08,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:28:08,604 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:28:08,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:28:08,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:28:08,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:28:08,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:28:08,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:28:08,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:28:08,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:28:08,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:28:08,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:28:08,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:28:08,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:28:08,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:28:08,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:28:08,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:28:08,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:28:08,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:28:08,923 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:28:08,943 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:28:08,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:28:08,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:28:08,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:28:09,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:28:09,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:28:09,056 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:28:09,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:28:09,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:28:09,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:28:09,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:28:09,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:28:09,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:28:09,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:28:09,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:28:09,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:28:09,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:28:09,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:28:09,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:28:09,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:28:09,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:28:09,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:28:09,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:28:09,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:28:09,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:28:09,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:28:09,426 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:28:09,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:28:09,459 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:28:09,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:28:09,494 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:28:09,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:28:09,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:28:09,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:28:09,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:28:09,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:28:09,596 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:28:09,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:28:09,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:28:09,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:28:09,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:28:09,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:28:09,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:28:09,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:28:09,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:28:09,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:28:09,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:28:09,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:28:09,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:28:09,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:28:09,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:28:09,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:28:09,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:28:09,885 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:28:09,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:28:09,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:28:09,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:28:09,969 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:28:09,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:28:10,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:28:10,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:28:10,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:28:10,059 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 11:28:10,059 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:28:10,062 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:28:10,183 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:28:10,403 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:28:10,467 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:28:10,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:28:10,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:28:10,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:28:10,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:28:10,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:28:10,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:28:10,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:28:10,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:11,072 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:28:11,072 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.01300001144. 2022-02-08 11:28:11,072 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:28:11,075 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:28:11,272 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:28:11,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:28:11,658 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:28:11,658 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:28:11,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:28:11,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:28:11,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:28:11,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:28:11,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:28:11,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:28:11,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:11,957 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:28:11,957 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:28:13,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:14,135 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:28:14,137 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:28:16,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:16,305 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:28:16,305 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:28:18,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:18,648 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:28:18,648 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:28:20,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:21,108 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:28:21,108 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:28:23,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:23,387 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:28:23,387 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:28:25,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:25,637 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:28:25,638 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:28:27,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:27,841 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:28:27,842 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:28:29,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:30,115 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:28:30,117 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:28:32,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:32,344 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:28:32,344 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:28:34,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:34,555 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:28:34,555 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:28:36,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:36,746 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:28:36,746 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:28:38,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:38,920 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:28:38,920 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:28:40,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:41,062 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:28:41,062 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 29.9900000095. 2022-02-08 11:28:41,062 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:28:41,065 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:28:41,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:28:42,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:28:42,302 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:28:42,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:28:42,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:28:42,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:28:42,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:28:42,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:28:42,397 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:28:42,428 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:28:42,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:43,092 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:28:43,092 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.02999997139. 2022-02-08 11:28:43,094 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:28:43,095 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:28:43,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:28:44,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:28:44,181 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:28:44,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:28:44,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:28:44,214 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:28:44,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:28:44,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:28:44,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:28:44,295 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:28:44,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:45,869 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:28:45,869 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.77599978447. 2022-02-08 11:28:45,871 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:28:45,871 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:28:46,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:28:46,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:28:46,944 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:28:46,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:28:46,961 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:28:47,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:28:47,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:28:47,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:28:47,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:28:47,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:28:47,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:47,713 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:28:47,713 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.84200000763. 2022-02-08 11:28:47,713 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:28:47,716 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:28:48,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:28:49,224 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:28:49,355 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:28:49,357 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:28:49,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:28:49,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:28:49,410 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:28:49,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:28:49,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:28:49,482 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:28:49,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 14554 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:28:51,303 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:28:51,303 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.58999991417. 2022-02-08 11:28:51,305 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:28:51,306 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:28:51,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:28:52,163 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:28:52,257 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:28:52,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:28:52,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:28:52,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:28:52,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:28:52,323 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:28:52,336 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:28:52,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:28:52,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:52,816 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:28:52,818 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.51300001144. 2022-02-08 11:28:52,818 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:28:52,819 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:28:53,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:28:53,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:28:53,770 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:28:53,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:28:53,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:28:53,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:28:53,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:28:53,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:28:53,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:28:53,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:28:53,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:54,552 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:28:54,552 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.7349998951. 2022-02-08 11:28:54,553 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:28:54,556 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:28:55,201 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:28:55,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:28:55,671 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:28:55,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:28:55,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:28:55,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:28:55,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:28:55,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:28:55,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:28:55,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:28:55,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:28:56,332 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:28:56,334 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.78000020981. ############################################################ # # RAM FootPrint: 4870.36328125 # ############################################################ 2022-02-08 11:28:56,371 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 11:28:56,371 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 94 seconds. Reason: Iteration:13 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14580 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:30:30,380 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 11:30:30,380 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 11:30:35,382 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:30:35,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:30:36,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:30:36,305 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:30:36,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:30:36,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:30:36,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:30:36,349 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:30:36,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:30:36,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:30:36,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:30:36,436 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:30:36,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:30:36,506 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['59606 memcached\n'], []) 2022-02-08 11:30:36,507 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 11:30:36,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:30:36,585 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['59606 memcached\n'], []) 2022-02-08 11:30:36,586 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:30:36,589 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:30:37,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:30:37,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:30:37,552 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:30:37,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:30:37,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:30:37,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:30:37,607 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:30:37,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:30:37,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:30:37,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:30:37,684 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:30:37,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:30:37,740 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['114398 memcached\n'], []) 2022-02-08 11:30:37,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 11:30:37,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:30:37,801 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['114398 memcached\n'], []) 2022-02-08 11:30:37,802 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:30:37,805 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:30:37,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:30:38,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:30:38,203 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:30:38,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:30:38,221 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:30:38,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:30:38,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:30:38,272 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:30:38,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:30:38,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:30:38,316 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:30:38,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:30:38,361 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['97335 memcached\n'], []) 2022-02-08 11:30:38,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 11:30:38,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:30:38,413 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['97335 memcached\n'], []) 2022-02-08 11:30:38,413 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:30:38,417 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:30:38,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:30:38,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:30:38,773 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:30:38,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:30:38,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:30:38,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:30:38,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:30:38,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:30:38,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:30:38,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:30:38,869 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:30:38,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:30:38,926 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['123634 memcached\n'], []) 2022-02-08 11:30:38,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 11:30:38,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:30:38,973 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['123634 memcached\n'], []) 2022-02-08 11:30:38,973 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:30:38,976 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:30:39,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:30:39,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:30:39,336 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:30:39,336 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:30:39,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:30:39,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:30:39,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:30:39,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:30:39,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:30:39,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:30:39,440 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:30:39,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:30:39,507 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['117616 memcached\n'], []) 2022-02-08 11:30:39,507 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 11:30:39,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:30:39,556 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['117616 memcached\n'], []) 2022-02-08 11:30:39,556 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:30:39,561 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:30:39,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:30:40,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:30:40,101 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:30:40,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:30:40,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:30:40,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:30:40,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:30:40,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:30:40,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:30:40,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:30:40,197 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:30:40,198 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:30:40,256 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['22042 memcached\n'], []) 2022-02-08 11:30:40,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 11:30:40,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:30:40,303 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['22042 memcached\n'], []) 2022-02-08 11:30:40,303 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:30:40,306 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:30:40,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:30:40,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:30:40,789 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:30:40,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:30:40,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:30:40,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:30:40,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:30:40,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:30:40,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:30:40,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:30:40,888 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:30:40,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:30:40,944 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['86084 memcached\n'], []) 2022-02-08 11:30:40,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 11:30:40,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:30:40,992 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['86084 memcached\n'], []) 2022-02-08 11:30:40,992 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:30:40,994 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:30:41,086 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:30:41,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:30:41,378 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:30:41,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:30:41,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:30:41,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:30:41,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:30:41,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:30:41,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:30:41,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:30:41,486 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:30:41,487 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:30:41,549 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['101301 memcached\n'], []) 2022-02-08 11:30:41,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 11:30:41,584 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:30:41,614 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['101301 memcached\n'], []) 2022-02-08 11:30:41,614 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:30:41,617 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:30:41,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:30:41,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:30:41,976 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:30:41,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:30:41,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:30:42,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:30:42,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:30:42,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:30:42,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:30:42,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:30:42,082 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:30:42,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:30:42,121 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['37826 memcached\n'], []) 2022-02-08 11:30:42,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 11:30:42,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:30:42,174 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['37826 memcached\n'], []) 2022-02-08 11:30:42,174 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:30:42,177 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:30:42,272 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:30:42,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:30:42,568 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:30:42,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:30:42,584 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:30:42,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:30:42,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:30:42,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:30:42,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:30:42,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:30:42,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:30:42,700 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 11:30:42,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:30:42,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:30:46,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:30:47,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:30:48,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:30:48,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:30:49,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:30:49,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:30:49,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:30:49,395 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:30:49,411 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 11:30:49,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:30:49,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:30:49,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:30:49,516 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 11:30:49,517 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:30:49,520 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:30:49,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:30:49,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 5 queries submitted, 0 failed, 14604 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:30:49,877 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:30:49,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:30:49,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:30:49,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:30:49,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:30:49,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:30:49,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:30:49,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:30:49,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:30:49,992 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 11:30:50,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:30:50,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:30:55,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:30:56,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:30:56,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:30:57,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:30:57,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:30:57,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:30:58,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:30:58,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:30:58,102 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 11:30:58,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:30:58,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:30:58,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:30:58,206 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 11:30:58,207 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:30:58,210 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 11:30:58,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:30:58,540 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 11:30:58,605 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:30:58,607 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 11:30:58,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 11:30:58,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 11:30:58,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 11:30:58,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 11:30:58,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 11:30:58,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 11:30:58,721 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:30:58,740 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 11:30:58,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:30:58,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:30:58,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:30:58,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:30:58,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:30:58,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:30:58,869 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 11:30:58,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:30:58,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:30:58,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:30:58,915 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:30:58,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:30:58,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:30:58,971 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:30:58,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:30:59,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:30:59,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:30:59,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:30:59,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:30:59,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:30:59,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:30:59,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:30:59,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:30:59,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:30:59,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:30:59,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:30:59,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:30:59,229 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:30:59,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:30:59,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:30:59,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:30:59,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:30:59,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:30:59,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:30:59,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:30:59,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:30:59,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:30:59,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:30:59,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:30:59,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:30:59,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:30:59,474 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:30:59,494 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:30:59,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:30:59,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:30:59,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:30:59,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:30:59,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:30:59,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:30:59,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:30:59,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:30:59,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:30:59,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:30:59,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:30:59,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:30:59,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:30:59,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:30:59,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:30:59,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:30:59,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:30:59,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:30:59,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:30:59,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:30:59,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:30:59,915 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:30:59,936 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:30:59,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:30:59,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:30:59,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:00,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:00,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:00,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:31:00,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:00,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:00,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:00,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:00,127 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:00,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:00,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:00,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:00,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:00,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:31:00,229 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:00,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:00,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:00,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:00,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:00,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:00,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:00,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:00,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:00,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:31:00,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:00,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:00,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:00,459 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:00,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:00,494 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:00,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:00,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:00,549 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 11:31:00,552 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:31:00,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:31:01,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:31:01,121 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:31:01,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:31:01,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:31:01,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:31:01,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:31:01,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:31:01,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:31:01,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:31:01,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:31:01,252 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 11:31:01,285 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:31:01,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:02,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:31:03,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:03,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:04,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:31:04,062 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 11:31:04,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:31:04,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:04,147 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:31:04,174 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 11:31:04,174 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:31:04,177 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 11:31:04,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:31:04,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 11:31:04,549 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:31:04,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 11:31:04,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 11:31:04,579 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 11:31:04,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 11:31:04,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 11:31:04,627 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 11:31:04,640 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 11:31:04,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:31:04,671 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 11:31:04,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:31:04,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:04,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:31:04,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:04,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:04,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:31:04,795 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 11:31:04,795 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:31:04,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:04,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:31:04,836 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:04,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:31:04,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:04,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:04,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:04,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:04,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:04,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:04,982 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:05,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:05,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:05,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:31:05,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:05,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:05,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:05,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:05,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:05,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:05,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:05,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:05,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:05,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:31:05,201 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:05,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:05,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:05,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:05,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:05,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:05,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:05,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:05,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:05,349 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:31:05,367 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:05,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:05,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:05,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:05,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:05,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:05,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:05,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:05,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:05,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:31:05,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:05,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:05,581 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:05,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:05,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:05,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:05,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:05,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:05,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:05,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:31:05,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:05,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:05,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:05,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:05,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:05,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:05,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:05,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:05,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:05,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:31:05,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:05,936 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:05,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:05,966 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:05,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:06,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:06,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:06,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:06,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:06,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:31:06,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:06,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:06,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:06,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:06,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:06,173 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:06,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:06,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:06,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:06,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:31:06,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:06,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:06,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:06,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:06,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:06,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:06,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:06,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:06,380 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 11:31:06,382 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:31:06,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:31:06,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:31:06,749 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:31:06,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:31:06,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:31:06,778 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:31:06,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:31:06,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:31:06,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:31:06,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:31:06,854 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:31:06,868 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 11:31:06,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:31:06,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:10,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:31:11,163 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:11,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:12,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:12,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:31:12,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:31:13,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:31:13,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:31:13,140 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 11:31:13,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:31:13,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:13,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:31:13,267 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 11:31:13,269 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:31:13,272 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:31:13,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:31:13,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:31:13,721 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:31:13,721 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:31:13,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:31:13,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:31:13,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:31:13,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:31:13,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:31:13,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:31:13,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:31:13,855 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 11:31:13,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:31:13,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:18,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:31:19,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:20,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:21,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:21,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:31:21,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:31:21,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:31:22,017 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:31:22,043 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 11:31:22,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:31:22,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:22,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:31:22,183 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 11:31:22,184 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:31:22,187 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 11:31:22,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:31:22,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 11:31:22,625 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:31:22,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 11:31:22,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 11:31:22,658 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 11:31:22,676 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 11:31:22,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 11:31:22,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 11:31:22,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 11:31:22,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:31:22,776 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 11:31:22,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:31:22,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:22,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:31:22,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:22,887 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:22,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:31:22,921 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 11:31:22,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:31:22,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:22,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:31:22,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:23,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:31:23,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:23,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:23,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:23,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:23,119 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:23,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:23,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:23,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:23,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:23,232 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:31:23,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:23,270 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:23,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:23,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:23,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:23,346 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:23,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:23,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:23,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:23,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:31:23,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:23,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:23,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:23,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:23,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:23,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:23,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:23,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:23,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:23,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:31:23,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:23,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:23,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:23,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:23,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:23,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:23,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:23,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:23,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:23,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:31:23,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:23,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:23,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:23,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:23,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:23,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:24,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:24,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:24,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:24,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:31:24,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:24,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:24,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:24,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:24,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:24,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:24,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:24,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:24,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:24,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:31:24,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:24,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:24,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:24,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:24,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:24,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:24,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:24,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:24,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:24,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:31:24,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:24,494 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:24,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:24,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:24,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:24,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:24,591 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:24,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:24,627 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:24,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:31:24,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:24,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:24,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:24,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:24,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:24,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:24,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:24,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:24,831 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 11:31:24,834 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 11:31:24,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:31:25,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 11:31:25,259 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:31:25,260 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 11:31:25,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 11:31:25,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 11:31:25,303 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 11:31:25,323 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 11:31:25,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 11:31:25,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 11:31:25,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:31:25,387 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 11:31:25,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:31:25,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:25,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:31:25,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:25,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:25,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:31:25,517 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 11:31:25,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:31:25,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:25,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:31:25,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:25,584 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:31:25,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:25,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:25,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:25,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:25,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:25,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:25,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:25,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:25,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:25,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:31:25,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:25,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:25,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:25,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:25,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:25,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:25,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:25,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:25,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:25,971 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:31:25,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:26,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:26,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:26,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:26,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:26,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:26,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:26,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:26,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:26,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:31:26,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:26,196 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:26,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:26,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:26,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:26,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:26,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:26,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:26,323 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:26,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:31:26,362 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:26,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:26,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:26,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:26,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:26,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:26,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:26,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:26,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:26,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:31:26,540 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:26,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:26,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:26,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:26,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:26,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:26,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:26,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:26,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:26,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:31:26,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:26,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:26,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:26,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:26,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:26,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:26,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:26,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:26,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:26,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:31:26,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:26,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:26,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:26,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:26,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:26,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:26,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:26,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:27,022 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:27,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:31:27,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:27,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:27,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:27,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:27,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:27,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:27,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:27,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:27,206 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 11:31:27,209 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:31:27,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:31:27,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:31:27,834 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:31:27,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:31:27,854 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:31:27,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:31:27,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:31:27,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:31:27,961 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:31:27,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:31:28,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:31:28,058 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 11:31:28,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:31:28,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:30,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:31:31,395 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:32,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:32,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:33,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:31:33,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:31:33,329 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:31:33,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:31:33,450 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 11:31:33,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:31:33,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:33,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:31:33,563 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 11:31:33,565 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:31:33,566 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:31:33,658 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:31:33,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:31:33,986 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:31:33,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:31:34,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:31:34,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:31:34,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:31:34,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:31:34,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:31:34,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:31:34,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:31:34,168 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 11:31:34,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:31:34,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:37,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:31:38,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:38,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:38,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:31:38,908 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 11:31:38,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:31:38,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:38,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:31:38,984 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 11:31:38,984 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:31:38,986 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:31:39,474 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:31:39,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:31:39,917 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:31:39,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:31:39,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:31:39,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:31:39,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:31:39,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:31:40,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:31:40,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:31:40,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:31:40,059 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 11:31:40,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:31:40,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:41,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:31:42,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:42,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:43,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:31:43,115 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 11:31:43,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:31:43,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:43,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:31:43,203 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 11:31:43,203 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:31:43,207 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:31:43,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:31:43,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:31:43,644 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:31:43,645 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:31:43,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:31:43,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:31:43,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:31:43,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:31:43,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:31:43,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:31:43,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:31:43,792 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 11:31:43,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:31:43,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:45,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:31:46,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:47,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:47,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:47,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:31:48,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:31:48,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:31:48,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:31:48,398 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 11:31:48,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:31:48,510 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:48,696 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:31:48,845 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 11:31:48,845 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:31:48,848 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 11:31:48,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:31:49,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 11:31:49,249 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:31:49,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 11:31:49,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 11:31:49,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 11:31:49,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 11:31:49,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 11:31:49,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 11:31:49,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 11:31:49,362 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:31:49,380 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 11:31:49,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:31:49,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:49,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:31:49,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:49,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:31:49,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:31:49,503 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 11:31:49,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:31:49,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:31:49,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:31:49,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:49,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:31:49,591 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:49,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:49,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:49,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:49,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:49,729 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:49,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:49,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:31:49,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:49,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:31:49,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 5 queries submitted, 0 failed, 14614 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:31:49,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:49,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:49,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:49,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:49,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:49,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:49,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:31:49,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:50,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:31:50,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:50,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:50,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:50,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:50,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:50,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:50,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:50,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:31:50,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:50,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:31:50,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:50,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:50,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:50,295 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:50,313 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:50,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:50,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:50,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:31:50,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:50,410 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:31:50,428 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:50,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:50,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:50,487 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:50,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:50,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:50,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:50,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:31:50,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:50,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:31:50,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:50,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:50,660 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:50,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:50,696 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:50,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:50,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:50,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:31:50,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:50,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:31:50,808 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:50,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:50,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:50,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:50,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:50,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:50,913 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:50,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:31:50,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:50,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:31:50,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:51,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:51,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:51,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:51,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:51,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:51,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:51,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:31:51,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:51,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:31:51,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:51,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:51,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:51,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:51,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:51,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:51,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:51,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:31:51,322 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 11:31:51,323 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:31:51,325 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:31:51,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:31:51,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:31:51,832 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:31:51,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:31:51,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:31:51,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:31:51,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:31:51,918 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:31:51,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:31:51,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:31:51,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:31:52,436 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:31:52,437 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.1130001545. 2022-02-08 11:31:52,437 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:31:52,438 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:31:53,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:31:53,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:31:53,723 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:31:53,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:31:53,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:31:53,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:31:53,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:31:53,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:31:53,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:31:53,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:31:53,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:31:54,119 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:31:54,119 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:31:56,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:31:56,448 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:31:56,450 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:31:58,454 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:31:59,062 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:31:59,062 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:32:01,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:01,298 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:32:01,299 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:32:03,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:03,490 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:32:03,492 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:32:05,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:05,681 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:32:05,681 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:32:07,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:07,966 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:32:07,967 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:32:09,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:10,161 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:32:10,161 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:32:12,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:12,345 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:32:12,345 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:32:14,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:14,559 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:32:14,559 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:32:16,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:16,742 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:32:16,742 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:32:18,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:18,971 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:32:18,973 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 26.5360000134. 2022-02-08 11:32:18,973 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:32:18,974 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:32:19,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:32:20,221 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:32:20,572 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:32:20,573 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:32:20,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:32:20,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:32:20,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:32:20,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:32:20,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:32:20,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:32:20,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:21,138 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:32:21,138 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.16499996185. 2022-02-08 11:32:21,138 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:32:21,140 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:32:21,808 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:32:22,229 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:32:22,329 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:32:22,329 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:32:22,349 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:32:22,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:32:22,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:32:22,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:32:22,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:32:22,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:32:22,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:23,196 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:32:23,197 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.05900001526. 2022-02-08 11:32:23,197 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:32:23,200 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:32:23,923 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:32:24,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:32:24,312 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:32:24,313 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:32:24,331 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:32:24,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:32:24,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:32:24,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:32:24,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:32:24,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:32:24,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:25,144 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:32:25,144 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.9470000267. 2022-02-08 11:32:25,144 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:32:25,145 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:32:25,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:32:26,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:32:26,763 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:32:26,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:32:26,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:32:26,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:32:26,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:32:26,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:32:26,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:32:27,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:32:27,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:28,072 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:32:28,072 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 11:32:30,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:30,325 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:32:30,325 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 11:32:32,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:32,562 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:32:32,562 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 11:32:34,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:34,739 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:32:34,740 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 9.59599995613. 2022-02-08 11:32:34,740 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:32:34,743 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:32:35,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:32:35,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:32:35,753 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:32:35,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:32:35,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:32:35,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:32:35,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:32:35,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:32:35,836 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:32:35,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:32:35,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:36,726 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:32:36,726 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.98699998856. 2022-02-08 11:32:36,727 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:32:36,729 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:32:37,816 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:32:38,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:32:38,578 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:32:38,579 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:32:38,596 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:32:38,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:32:38,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:32:38,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:32:38,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:32:38,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:32:38,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:42,539 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:32:42,539 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.81200003624. 2022-02-08 11:32:42,540 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:32:42,542 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:32:43,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:32:43,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:32:44,112 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:32:44,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:32:44,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:32:44,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:32:44,198 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:32:44,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:32:44,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:32:44,270 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:32:44,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:32:45,953 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:32:45,953 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.41400003433. ############################################################ # # RAM FootPrint: 4894.90625 # ############################################################ 2022-02-08 11:32:45,987 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 11:32:45,989 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 119 seconds. Reason: Iteration:14 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14626 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 14653 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:34:44,999 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 11:34:44,999 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 5 queries submitted, 0 failed, 14677 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:34:50,003 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:34:50,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:34:50,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:34:50,943 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:34:50,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:34:50,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:34:50,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:34:50,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:34:51,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:34:51,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:34:51,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:34:51,114 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:34:51,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:34:51,174 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['60329 memcached\n'], []) 2022-02-08 11:34:51,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 11:34:51,201 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:34:51,224 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['60329 memcached\n'], []) 2022-02-08 11:34:51,226 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:34:51,227 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:34:51,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:34:51,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:34:52,045 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:34:52,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:34:52,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:34:52,086 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:34:52,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:34:52,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:34:52,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:34:52,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:34:52,168 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:34:52,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:34:52,223 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['115308 memcached\n'], []) 2022-02-08 11:34:52,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 11:34:52,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:34:52,266 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['115308 memcached\n'], []) 2022-02-08 11:34:52,267 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:34:52,269 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:34:52,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:34:52,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:34:52,665 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:34:52,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:34:52,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:34:52,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:34:52,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:34:52,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:34:52,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:34:52,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:34:52,778 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:34:52,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:34:52,831 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['98199 memcached\n'], []) 2022-02-08 11:34:52,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 11:34:52,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:34:52,884 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['98199 memcached\n'], []) 2022-02-08 11:34:52,885 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:34:52,888 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:34:52,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:34:53,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:34:53,253 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:34:53,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:34:53,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:34:53,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:34:53,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:34:53,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:34:53,323 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:34:53,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:34:53,351 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:34:53,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:34:53,424 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['124503 memcached\n'], []) 2022-02-08 11:34:53,426 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 11:34:53,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:34:53,473 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['124503 memcached\n'], []) 2022-02-08 11:34:53,474 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:34:53,477 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:34:53,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:34:53,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:34:53,819 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:34:53,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:34:53,834 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:34:53,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:34:53,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:34:53,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:34:53,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:34:53,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:34:53,920 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:34:53,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:34:53,983 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['118499 memcached\n'], []) 2022-02-08 11:34:53,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 11:34:54,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:34:54,039 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['118499 memcached\n'], []) 2022-02-08 11:34:54,039 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:34:54,042 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:34:54,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:34:54,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:34:55,032 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:34:55,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:34:55,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:34:55,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:34:55,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:34:55,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:34:55,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:34:55,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:34:55,206 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:34:55,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:34:55,265 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['22924 memcached\n'], []) 2022-02-08 11:34:55,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 11:34:55,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:34:55,316 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['22924 memcached\n'], []) 2022-02-08 11:34:55,316 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:34:55,319 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:34:55,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:34:55,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:34:55,700 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:34:55,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:34:55,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:34:55,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:34:55,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:34:55,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:34:55,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:34:55,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:34:55,795 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:34:55,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:34:55,845 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['86811 memcached\n'], []) 2022-02-08 11:34:55,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 11:34:55,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:34:55,894 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['86811 memcached\n'], []) 2022-02-08 11:34:55,894 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:34:55,898 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:34:56,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:34:56,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:34:57,062 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:34:57,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:34:57,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:34:57,102 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:34:57,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:34:57,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:34:57,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:34:57,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:34:57,190 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:34:57,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:34:57,246 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['102024 memcached\n'], []) 2022-02-08 11:34:57,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 11:34:57,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:34:57,293 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['102024 memcached\n'], []) 2022-02-08 11:34:57,293 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:34:57,296 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:34:57,374 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:34:57,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:34:57,641 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:34:57,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:34:57,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:34:57,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:34:57,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:34:57,696 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:34:57,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:34:57,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:34:57,740 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:34:57,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:34:57,789 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['38584 memcached\n'], []) 2022-02-08 11:34:57,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 11:34:57,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:34:57,842 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['38584 memcached\n'], []) 2022-02-08 11:34:57,844 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:34:57,845 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:34:57,918 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:34:58,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:34:58,194 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:34:58,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:34:58,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:34:58,221 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:34:58,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:34:58,260 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:34:58,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:34:58,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:34:58,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:34:58,315 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 11:34:58,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:34:58,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:00,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:35:01,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:02,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:02,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:03,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:35:03,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:35:03,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:35:03,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:35:03,437 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 11:35:03,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:35:03,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:03,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:35:03,529 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 11:35:03,529 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:35:03,530 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:35:03,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:35:03,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:35:03,901 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:35:03,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:35:03,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:35:03,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:35:03,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:35:03,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:35:03,971 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:35:03,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:35:04,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:35:04,019 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 11:35:04,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:35:04,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:08,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:35:09,303 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:09,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:10,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:10,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:35:10,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:35:11,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:35:11,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:35:11,108 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 11:35:11,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:35:11,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:11,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:35:11,209 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 11:35:11,210 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:35:11,211 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 11:35:11,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:35:11,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 11:35:11,624 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:35:11,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 11:35:11,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 11:35:11,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 11:35:11,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 11:35:11,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 11:35:11,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 11:35:11,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 11:35:11,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:35:11,743 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 11:35:11,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:35:11,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:11,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:35:11,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:11,834 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:11,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:35:11,862 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 11:35:11,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:35:11,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:11,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:35:11,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:11,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:35:11,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:11,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:11,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:12,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:12,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:12,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:12,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:12,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:12,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:12,127 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:35:12,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:12,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:12,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:12,196 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:12,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:12,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:12,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:12,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:12,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:12,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:35:12,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:12,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:12,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:12,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:12,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:12,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:12,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:12,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:12,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:12,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:35:12,505 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:12,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:12,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:12,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:12,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:12,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:12,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:12,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:12,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:12,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:35:12,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:12,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:12,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:12,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:12,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:12,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:12,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:12,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:12,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:12,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:35:12,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:12,910 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:12,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:12,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:12,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:12,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:12,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:13,012 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:13,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:13,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:35:13,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:13,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:13,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:13,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:13,127 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:13,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:13,160 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:13,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:13,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:13,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:35:13,227 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:13,244 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:13,260 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:13,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:13,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:13,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:13,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:13,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:13,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:13,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:35:13,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:13,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:13,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:13,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:13,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:13,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:13,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:13,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:13,523 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 11:35:13,525 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:35:13,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:35:14,178 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:35:14,582 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:35:14,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:35:14,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:35:14,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:35:14,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:35:15,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:35:15,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:35:15,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:35:15,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:35:15,680 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 11:35:15,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:35:15,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:17,647 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:35:18,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:18,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:18,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:35:19,009 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 11:35:19,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:35:19,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:19,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:35:19,108 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 11:35:19,108 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:35:19,111 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 11:35:19,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:35:19,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 11:35:19,516 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:35:19,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 11:35:19,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 11:35:19,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 11:35:19,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 11:35:19,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 11:35:19,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 11:35:19,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 11:35:19,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:35:19,634 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 11:35:19,660 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:35:19,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:19,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:35:19,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:19,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:19,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:35:19,740 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 11:35:19,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:35:19,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:19,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:35:19,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:19,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:35:19,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:19,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:19,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:19,874 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:19,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:19,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:19,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:19,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:19,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:19,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:35:19,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:20,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:20,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:20,043 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:20,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:20,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:20,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:20,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:20,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:20,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:35:20,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:20,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:20,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:20,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:20,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:20,232 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:20,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:20,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:20,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:20,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:35:20,313 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:20,329 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:20,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:20,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:20,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:20,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:20,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:20,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:20,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:20,482 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:35:20,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:20,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:20,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:20,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:20,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:20,579 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:20,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:20,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:20,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:20,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:35:20,660 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:20,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:20,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:20,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:20,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:20,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:20,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:20,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:20,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:20,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:35:20,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:20,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:20,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:20,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:20,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:20,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:20,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:20,923 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:20,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:20,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:35:20,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:20,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:20,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:21,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:21,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:21,038 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:21,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:21,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:21,086 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:21,102 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:35:21,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:21,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:21,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:21,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:21,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:21,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:21,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:21,229 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:21,243 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 11:35:21,246 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:35:21,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:35:21,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:35:21,911 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:35:21,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:35:21,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:35:21,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:35:21,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:35:21,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:35:21,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:35:22,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:35:22,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:35:22,042 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 11:35:22,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:35:22,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:25,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:35:27,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:27,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:28,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:30,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:35:30,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:35:30,290 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:35:30,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:35:30,398 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 11:35:30,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:35:30,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:30,474 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:35:30,492 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 11:35:30,493 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:35:30,496 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:35:30,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:35:31,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:35:31,621 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:35:31,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:35:31,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:35:31,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:35:31,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:35:31,887 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:35:31,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:35:31,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:35:31,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:35:31,963 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 11:35:31,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:35:32,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:38,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:35:39,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:40,022 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:40,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:41,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:35:41,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:35:41,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:35:41,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:35:42,006 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 11:35:42,007 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:35:42,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:42,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:35:42,312 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 11:35:42,312 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:35:42,315 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 11:35:42,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:35:42,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 11:35:42,724 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:35:42,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 11:35:42,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 11:35:42,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 11:35:42,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 11:35:42,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 11:35:42,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 11:35:42,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 11:35:42,836 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:35:42,854 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 11:35:42,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:35:42,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:42,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:35:42,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:42,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:42,961 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:35:42,977 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 11:35:42,979 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:35:42,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:43,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:35:43,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:43,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:35:43,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:43,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:43,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:43,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:43,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:43,178 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:43,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:43,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:43,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:43,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:35:43,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:43,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:43,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:43,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:43,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:43,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:43,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:43,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:43,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:43,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:35:43,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:43,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:43,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:43,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:43,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:43,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:43,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:43,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:43,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:43,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:35:43,676 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:43,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:43,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:43,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:43,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:43,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:43,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:43,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:43,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:43,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:35:43,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:43,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:43,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:43,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:43,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:43,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:43,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:44,002 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:44,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:44,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:35:44,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:44,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:44,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:44,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:44,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:44,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:44,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:44,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:44,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:44,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:35:44,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:44,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:44,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:44,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:44,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:44,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:44,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:44,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:44,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:44,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:35:44,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:44,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:44,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:44,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:44,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:44,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:44,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:44,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:44,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:44,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:35:44,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:44,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:44,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:44,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:44,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:44,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:44,729 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:44,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:44,767 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 11:35:44,770 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 11:35:44,854 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:35:45,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 11:35:45,142 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:35:45,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 11:35:45,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 11:35:45,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 11:35:45,186 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 11:35:45,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 11:35:45,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 11:35:45,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 11:35:45,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:35:45,259 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 11:35:45,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:35:45,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:45,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:35:45,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:45,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:45,362 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:35:45,384 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 11:35:45,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:35:45,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:45,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:35:45,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:45,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:35:45,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:45,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:45,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:45,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:45,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:45,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:45,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:45,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:35:45,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:45,645 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:35:45,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:45,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:45,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:45,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:45,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:45,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:45,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:45,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:35:45,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:45,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:35:45,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:45,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:45,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:45,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:45,915 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:45,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:45,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:45,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:35:45,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:46,007 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:35:46,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:46,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:46,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:46,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:46,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:46,119 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:46,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:46,160 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:35:46,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:46,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:35:46,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:46,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:46,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:46,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:46,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:46,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:46,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:46,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:35:46,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:46,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:35:46,395 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:46,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:46,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:46,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:46,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:46,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:46,494 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:46,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:35:46,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:46,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:35:46,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:46,573 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:46,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:46,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:46,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:46,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:46,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:46,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:35:46,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:46,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:35:46,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:46,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:46,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:46,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:46,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:46,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:46,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:46,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:35:46,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:46,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:35:46,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:46,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:46,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:46,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:46,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:46,989 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:47,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:47,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:35:47,042 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 11:35:47,045 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:35:47,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:35:47,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:35:47,453 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:35:47,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:35:47,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:35:47,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:35:47,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:35:47,520 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:35:47,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:35:47,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:35:47,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:35:47,582 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 11:35:47,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:35:47,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:48,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:35:49,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 14685 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:35:50,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:50,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:51,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:35:51,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:35:51,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:35:51,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:35:51,546 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 11:35:51,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:35:51,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:51,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:35:51,654 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 11:35:51,655 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:35:51,667 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:35:51,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:35:52,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:35:52,081 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:35:52,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:35:52,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:35:52,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:35:52,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:35:52,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:35:52,170 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:35:52,191 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:35:52,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:35:52,233 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 11:35:52,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:35:52,278 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:56,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:35:56,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:57,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:35:57,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:35:57,635 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 11:35:57,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:35:57,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:35:57,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:35:57,713 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 11:35:57,713 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:35:57,716 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:35:57,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:35:58,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:35:58,111 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:35:58,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:35:58,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:35:58,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:35:58,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:35:58,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:35:58,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:35:58,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:35:58,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:35:58,289 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 11:35:58,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:35:58,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:36:03,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:36:04,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:36:04,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:36:05,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:36:05,364 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 11:36:05,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:36:05,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:36:05,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:36:05,453 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 11:36:05,453 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:36:05,460 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:36:05,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:36:05,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:36:05,973 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:36:05,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:36:05,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:36:06,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:36:06,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:36:06,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:36:06,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:36:06,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:36:06,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:36:06,138 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 11:36:06,163 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:36:06,191 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:36:13,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:36:14,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:36:15,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:36:15,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:36:17,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:36:17,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:36:17,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:36:18,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:36:18,138 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 11:36:18,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:36:18,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:36:18,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:36:18,299 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 11:36:18,299 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:36:18,302 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 11:36:18,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:36:18,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 11:36:18,688 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:36:18,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 11:36:18,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 11:36:18,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 11:36:18,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 11:36:18,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 11:36:18,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 11:36:18,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 11:36:18,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:36:18,806 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 11:36:18,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:36:18,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:36:18,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:36:18,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:36:18,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:36:18,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:36:18,923 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 11:36:18,923 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:36:18,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:36:18,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:36:18,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:36:18,989 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:36:19,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:36:19,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:36:19,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:36:19,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:36:19,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:36:19,147 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:36:19,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:36:19,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:36:19,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:36:19,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:36:19,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:36:19,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:36:19,272 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:36:19,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:36:19,303 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:36:19,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:36:19,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:36:19,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:36:19,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:36:19,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:36:19,410 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:36:19,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:36:19,446 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:36:19,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:36:19,482 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:36:19,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:36:19,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:36:19,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:36:19,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:36:19,573 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:36:19,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:36:19,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:36:19,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:36:19,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:36:19,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:36:19,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:36:19,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:36:19,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:36:19,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:36:19,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:36:19,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:36:19,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:36:19,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:36:19,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:36:19,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:36:19,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:36:19,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:36:19,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:36:19,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:36:19,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:36:19,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:36:19,969 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:36:19,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:36:20,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:36:20,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:36:20,038 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:36:20,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:36:20,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:36:20,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:36:20,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:36:20,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:36:20,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:36:20,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:36:20,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:36:20,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:36:20,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:36:20,229 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:36:20,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:36:20,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:36:20,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:36:20,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:36:20,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:36:20,349 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:36:20,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:36:20,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:36:20,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:36:20,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:36:20,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:36:20,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:36:20,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:36:20,494 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:36:20,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:36:20,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:36:20,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:36:20,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:36:20,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:36:20,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:36:20,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:36:20,637 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 11:36:20,637 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:36:20,638 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:36:21,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:36:21,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:36:21,513 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:36:21,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:36:21,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:36:21,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:36:21,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:36:21,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:36:21,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:36:21,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:36:21,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:36:22,223 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:36:22,223 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.58599996567. 2022-02-08 11:36:22,223 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:36:22,226 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:36:22,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:36:22,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:36:22,663 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:36:22,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:36:22,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:36:22,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:36:22,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:36:22,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:36:22,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:36:22,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:36:22,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:36:23,049 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:36:23,051 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:36:25,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:36:25,239 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:36:25,240 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:36:27,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:36:27,436 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:36:27,436 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:36:29,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:36:29,638 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:36:29,638 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:36:31,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:36:31,805 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:36:31,805 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:36:33,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:36:34,040 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:36:34,042 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:36:36,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:36:36,466 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:36:36,467 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:36:38,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:36:38,773 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:36:38,773 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:36:40,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:36:41,117 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:36:41,117 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:36:43,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:36:43,562 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:36:43,562 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:36:45,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:36:45,723 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:36:45,724 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 23.501999855. 2022-02-08 11:36:45,724 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:36:45,726 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:36:46,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:36:46,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:36:46,842 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:36:46,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:36:46,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:36:46,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:36:46,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:36:46,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:36:46,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:36:47,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:36:47,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:36:47,806 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:36:47,809 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.08400011063. 2022-02-08 11:36:47,809 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:36:47,812 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:36:48,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:36:48,915 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:36:49,088 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:36:49,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:36:49,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:36:49,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:36:49,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:36:49,227 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:36:49,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:36:49,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:36:49,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 14690 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:36:50,171 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:36:50,171 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.36299991608. 2022-02-08 11:36:50,171 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:36:50,174 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:36:51,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:36:51,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:36:51,586 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:36:51,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:36:51,607 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:36:51,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:36:51,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:36:51,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:36:51,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:36:51,795 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:36:51,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:36:52,492 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:36:52,492 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.32000017166. 2022-02-08 11:36:52,492 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:36:52,493 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:36:53,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:36:53,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:36:53,834 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:36:53,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:36:53,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:36:53,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:36:53,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:36:53,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:36:53,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:36:53,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:36:54,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:36:56,140 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:36:56,141 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.64899992943. 2022-02-08 11:36:56,141 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:36:56,142 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:36:56,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:36:57,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:36:57,605 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:36:57,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:36:57,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:36:57,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:36:57,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:36:57,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:36:57,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:36:57,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:36:57,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:37:03,565 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:37:03,566 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.42499995232. 2022-02-08 11:37:03,566 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:37:03,569 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:37:04,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:37:04,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:37:04,891 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:37:04,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:37:04,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:37:04,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:37:04,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:37:04,969 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:37:04,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:37:05,007 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:37:05,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:37:05,647 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:37:05,647 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.08000016212. 2022-02-08 11:37:05,648 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:37:05,648 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:37:06,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:37:07,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:37:07,308 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:37:07,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:37:07,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:37:07,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:37:07,374 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:37:07,395 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:37:07,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:37:07,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:37:07,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:37:07,959 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:37:07,960 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.31200003624. ############################################################ # # RAM FootPrint: 4904.1640625 # ############################################################ 2022-02-08 11:37:07,990 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 11:37:07,990 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 74 seconds. Reason: Iteration:15 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14715 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:38:21,996 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 11:38:21,996 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 11:38:27,000 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:38:27,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:38:27,543 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:38:27,618 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:38:27,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:38:27,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:38:27,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:38:27,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:38:27,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:38:27,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:38:27,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:38:27,717 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:38:27,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:38:27,773 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['61070 memcached\n'], []) 2022-02-08 11:38:27,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 11:38:27,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:38:27,826 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['61070 memcached\n'], []) 2022-02-08 11:38:27,826 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:38:27,829 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:38:28,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:38:28,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:38:28,614 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:38:28,614 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:38:28,627 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:38:28,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:38:28,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:38:28,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:38:28,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:38:28,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:38:28,711 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:38:28,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:38:28,769 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['116211 memcached\n'], []) 2022-02-08 11:38:28,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 11:38:28,795 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:38:28,815 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['116211 memcached\n'], []) 2022-02-08 11:38:28,816 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:38:28,819 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:38:28,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:38:29,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:38:29,187 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:38:29,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:38:29,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:38:29,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:38:29,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:38:29,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:38:29,265 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:38:29,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:38:29,295 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:38:29,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:38:29,342 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['99072 memcached\n'], []) 2022-02-08 11:38:29,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 11:38:29,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:38:29,391 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['99072 memcached\n'], []) 2022-02-08 11:38:29,391 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:38:29,394 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:38:29,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:38:29,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:38:29,753 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:38:29,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:38:29,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:38:29,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:38:29,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:38:29,808 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:38:29,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:38:29,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:38:29,849 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:38:29,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:38:29,898 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['125402 memcached\n'], []) 2022-02-08 11:38:29,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 11:38:29,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:38:29,944 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['125402 memcached\n'], []) 2022-02-08 11:38:29,944 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:38:29,947 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:38:30,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:38:30,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:38:30,296 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:38:30,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:38:30,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:38:30,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:38:30,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:38:30,357 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:38:30,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:38:30,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:38:30,395 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:38:30,397 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:38:30,471 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['119269 memcached\n'], []) 2022-02-08 11:38:30,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 11:38:30,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:38:30,520 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['119269 memcached\n'], []) 2022-02-08 11:38:30,520 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:38:30,523 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:38:30,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:38:30,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:38:30,878 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:38:30,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:38:30,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:38:30,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:38:30,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:38:30,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:38:30,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:38:30,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:38:30,973 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:38:30,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:38:31,052 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['23733 memcached\n'], []) 2022-02-08 11:38:31,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 11:38:31,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:38:31,099 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['23733 memcached\n'], []) 2022-02-08 11:38:31,099 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:38:31,101 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:38:31,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:38:31,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:38:31,446 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:38:31,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:38:31,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:38:31,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:38:31,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:38:31,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:38:31,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:38:31,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:38:31,549 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:38:31,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:38:31,618 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['87600 memcached\n'], []) 2022-02-08 11:38:31,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 11:38:31,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:38:31,664 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['87600 memcached\n'], []) 2022-02-08 11:38:31,665 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:38:31,667 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:38:32,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:38:32,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:38:32,880 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:38:32,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:38:32,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:38:32,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:38:32,936 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:38:32,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:38:32,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:38:32,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:38:33,005 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:38:33,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:38:33,065 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['102768 memcached\n'], []) 2022-02-08 11:38:33,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 11:38:33,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:38:33,111 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['102768 memcached\n'], []) 2022-02-08 11:38:33,111 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:38:33,114 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:38:33,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:38:33,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:38:33,480 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:38:33,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:38:33,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:38:33,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:38:33,520 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:38:33,540 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:38:33,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:38:33,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:38:33,588 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:38:33,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:38:33,638 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['39357 memcached\n'], []) 2022-02-08 11:38:33,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 11:38:33,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:38:33,688 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['39357 memcached\n'], []) 2022-02-08 11:38:33,690 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:38:33,713 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:38:33,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:38:34,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:38:34,092 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:38:34,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:38:34,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:38:34,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:38:34,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:38:34,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:38:34,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:38:34,178 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:38:34,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:38:34,209 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 11:38:34,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:38:34,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:38:36,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:38:37,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:38:38,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:38:38,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:38:38,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:38:39,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:38:39,163 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:38:39,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:38:39,256 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 11:38:39,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:38:39,270 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:38:39,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:38:39,355 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 11:38:39,355 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:38:39,357 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:38:39,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:38:39,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:38:39,700 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:38:39,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:38:39,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:38:39,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:38:39,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:38:39,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:38:39,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:38:39,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:38:39,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:38:39,816 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 11:38:39,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:38:39,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:38:41,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:38:42,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:38:43,428 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:38:43,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:38:44,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:38:44,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:38:44,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:38:44,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:38:44,811 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 11:38:44,811 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:38:44,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:38:44,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:38:44,921 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 11:38:44,921 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:38:44,924 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 11:38:45,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:38:45,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 11:38:45,311 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:38:45,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 11:38:45,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 11:38:45,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 11:38:45,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 11:38:45,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 11:38:45,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 11:38:45,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 11:38:45,413 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:38:45,428 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 11:38:45,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:38:45,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:38:45,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:38:45,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:38:45,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:38:45,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:38:45,551 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 11:38:45,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:38:45,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:38:45,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:38:45,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:45,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:38:45,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:45,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:45,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:45,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:45,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:45,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:45,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:45,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:45,808 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:45,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:38:45,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:45,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:45,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:45,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:45,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:45,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:45,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:45,966 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:45,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:46,002 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:38:46,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:46,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:46,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:46,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:46,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:46,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:46,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:46,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:46,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:46,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:38:46,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:46,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:46,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:46,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:46,285 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:46,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:46,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:46,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:46,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:46,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:38:46,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:46,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:46,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:46,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:46,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:46,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:46,510 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:46,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:46,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:46,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:38:46,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:46,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:46,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:46,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:46,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:46,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:46,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:46,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:46,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:46,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:38:46,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:46,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:46,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:46,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:46,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:46,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:46,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:46,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:46,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:46,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:38:46,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:46,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:47,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:47,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:47,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:47,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:47,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:47,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:47,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:47,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:38:47,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:47,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:47,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:47,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:47,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:47,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:47,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:47,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:47,302 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 11:38:47,303 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:38:47,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:38:47,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:38:48,453 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:38:48,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:38:48,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:38:48,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:38:48,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:38:49,043 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:38:49,163 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:38:49,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:38:49,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:38:49,608 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 11:38:49,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:38:49,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 14735 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:38:55,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:38:56,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:38:56,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:38:56,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:38:56,680 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 11:38:56,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:38:56,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:38:56,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:38:56,782 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 11:38:56,783 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:38:56,796 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 11:38:56,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:38:57,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 11:38:57,164 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:38:57,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 11:38:57,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 11:38:57,196 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 11:38:57,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 11:38:57,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 11:38:57,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 11:38:57,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 11:38:57,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:38:57,280 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 11:38:57,306 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:38:57,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:38:57,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:38:57,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:38:57,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:38:57,374 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:38:57,388 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 11:38:57,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:38:57,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:38:57,415 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:38:57,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:57,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:38:57,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:57,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:57,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:57,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:57,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:57,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:57,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:57,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:38:57,640 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:57,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:38:57,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:57,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:57,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:57,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:57,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:57,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:57,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:57,778 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:38:57,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:57,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:38:57,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:57,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:57,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:57,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:57,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:57,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:57,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:57,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:38:57,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:57,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:38:57,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:58,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:58,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:58,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:58,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:58,071 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:58,086 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:58,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:38:58,119 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:58,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:38:58,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:58,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:58,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:58,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:58,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:58,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:58,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:58,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:38:58,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:58,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:38:58,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:58,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:58,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:58,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:58,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:58,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:58,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:58,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:38:58,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:58,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:38:58,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:58,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:58,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:58,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:58,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:58,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:58,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:58,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:38:58,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:58,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:38:58,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:58,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:58,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:58,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:58,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:58,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:58,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:58,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:38:58,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:58,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:38:58,808 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:58,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:58,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:58,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:58,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:58,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:58,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:58,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:38:58,934 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 11:38:58,937 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:38:59,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:38:59,214 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:38:59,273 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:38:59,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:38:59,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:38:59,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:38:59,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:38:59,326 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:38:59,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:38:59,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:38:59,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:38:59,384 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 11:38:59,413 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:38:59,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:03,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:39:04,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:04,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:05,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:05,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:39:05,854 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:39:05,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:39:06,038 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:39:06,052 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 11:39:06,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:39:06,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:06,119 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:39:06,134 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 11:39:06,134 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:39:06,137 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:39:06,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:39:06,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:39:06,509 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:39:06,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:39:06,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:39:06,540 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:39:06,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:39:06,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:39:06,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:39:06,604 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:39:06,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:39:06,637 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 11:39:06,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:39:06,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:09,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:39:10,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:11,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:11,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:12,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:39:12,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:39:12,668 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:39:12,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:39:12,750 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 11:39:12,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:39:12,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:12,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:39:12,871 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 11:39:12,871 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:39:12,874 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 11:39:12,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:39:13,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 11:39:13,315 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:39:13,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 11:39:13,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 11:39:13,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 11:39:13,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 11:39:13,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 11:39:13,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 11:39:13,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 11:39:13,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:39:13,457 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 11:39:13,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:39:13,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:13,520 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:39:13,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:13,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:13,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:39:13,595 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 11:39:13,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:39:13,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:13,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:39:13,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:13,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:39:13,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:13,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:13,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:13,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:13,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:13,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:13,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:13,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:13,918 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:13,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:39:13,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:13,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:13,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:14,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:14,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:14,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:14,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:14,086 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:14,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:14,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:39:14,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:14,160 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:14,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:14,196 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:14,214 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:14,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:14,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:14,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:14,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:14,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:39:14,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:14,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:14,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:14,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:14,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:14,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:14,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:14,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:14,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:14,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:39:14,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:14,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:14,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:14,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:14,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:14,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:14,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:14,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:14,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:14,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:39:14,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:14,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:14,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:14,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:14,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:14,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:14,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:14,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:14,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:14,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:39:14,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:14,971 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:14,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:15,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:15,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:15,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:15,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:15,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:15,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:15,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:39:15,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:15,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:15,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:15,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:15,260 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:15,280 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:15,306 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:15,329 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:15,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:15,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:39:15,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:15,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:15,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:15,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:15,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:15,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:15,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:15,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:15,565 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 11:39:15,566 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 11:39:15,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:39:15,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 11:39:15,920 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:39:15,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 11:39:15,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 11:39:15,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 11:39:15,961 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 11:39:15,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 11:39:15,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 11:39:16,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 11:39:16,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:39:16,039 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 11:39:16,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:39:16,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:16,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:39:16,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:16,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:16,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:39:16,158 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 11:39:16,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:39:16,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:16,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:39:16,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:16,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:39:16,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:16,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:16,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:16,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:16,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:16,362 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:16,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:16,413 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:16,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:16,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:39:16,469 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:16,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:16,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:16,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:16,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:16,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:16,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:16,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:16,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:16,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:39:16,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:16,655 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:16,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:16,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:16,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:16,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:16,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:16,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:16,778 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:16,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:39:16,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:16,834 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:16,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:16,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:16,885 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:16,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:16,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:16,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:16,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:16,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:39:16,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:17,012 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:17,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:17,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:17,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:17,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:17,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:17,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:17,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:17,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:39:17,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:17,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:17,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:17,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:17,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:17,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:17,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:17,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:17,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:17,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:39:17,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:17,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:17,362 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:17,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:17,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:17,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:17,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:17,443 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:17,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:17,474 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:39:17,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:17,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:17,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:17,540 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:17,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:17,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:17,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:17,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:17,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:17,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:39:17,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:17,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:17,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:17,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:17,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:17,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:17,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:17,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:17,802 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 11:39:17,805 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:39:17,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:39:18,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:39:18,926 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:39:18,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:39:19,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:39:19,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:39:19,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:39:19,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:39:19,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:39:19,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:39:19,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:39:19,839 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 11:39:19,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:39:19,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:23,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:39:24,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:24,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:25,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:25,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:39:25,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:39:26,056 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:39:26,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:39:26,131 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 11:39:26,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:39:26,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:26,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:39:26,247 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 11:39:26,247 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:39:26,250 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:39:26,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:39:26,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:39:26,681 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:39:26,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:39:26,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:39:26,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:39:26,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:39:26,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:39:26,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:39:26,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:39:26,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:39:26,819 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 11:39:26,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:39:26,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:30,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:39:30,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:31,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:31,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:39:31,556 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 11:39:31,556 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:39:31,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:31,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:39:31,634 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 11:39:31,634 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:39:31,638 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:39:31,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:39:31,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:39:32,029 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:39:32,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:39:32,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:39:32,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:39:32,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:39:32,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:39:32,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:39:32,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:39:32,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:39:32,193 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 11:39:32,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:39:32,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:36,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:39:36,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:37,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:37,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:39:37,762 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 11:39:37,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:39:37,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:37,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:39:37,888 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 11:39:37,888 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:39:37,891 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:39:38,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:39:38,255 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:39:38,362 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:39:38,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:39:38,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:39:38,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:39:38,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:39:38,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:39:38,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:39:38,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:39:38,540 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:39:38,559 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 11:39:38,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:39:38,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:39,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:39:40,255 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:40,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:41,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:41,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:39:41,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:39:41,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:39:41,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:39:41,921 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 11:39:41,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:39:41,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:42,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:39:42,091 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 11:39:42,092 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:39:42,094 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 11:39:42,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:39:42,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 11:39:42,479 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:39:42,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 11:39:42,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 11:39:42,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 11:39:42,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 11:39:42,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 11:39:42,556 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 11:39:42,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 11:39:42,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:39:42,612 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 11:39:42,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:39:42,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:42,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:39:42,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:42,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:39:42,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:39:42,734 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 11:39:42,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:39:42,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:39:42,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:39:42,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:42,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:39:42,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:42,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:42,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:42,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:42,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:42,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:42,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:42,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:39:43,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:43,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:39:43,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:43,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:43,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:43,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:43,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:43,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:43,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:43,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:39:43,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:43,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:39:43,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:43,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:43,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:43,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:43,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:43,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:43,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:43,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:39:43,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:43,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:39:43,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:43,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:43,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:43,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:43,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:43,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:43,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:43,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:39:43,543 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:43,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:39:43,581 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:43,596 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:43,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:43,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:43,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:43,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:43,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:43,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:39:43,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:43,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:39:43,778 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:43,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:43,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:43,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:43,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:43,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:43,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:43,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:39:43,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:43,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:39:43,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:43,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:43,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:44,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:44,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:44,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:44,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:44,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:39:44,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:44,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:39:44,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:44,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:44,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:44,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:44,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:44,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:44,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:44,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:39:44,260 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:44,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:39:44,298 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:44,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:44,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:44,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:44,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:44,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:44,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:44,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:39:44,440 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 11:39:44,441 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:39:44,444 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:39:44,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:39:44,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:39:44,880 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:39:44,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:39:44,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:39:44,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:39:44,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:39:44,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:39:45,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:39:45,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:39:45,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:39:45,522 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:39:45,522 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.08000016212. 2022-02-08 11:39:45,522 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:39:45,523 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:39:45,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:39:46,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:39:46,305 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:39:46,306 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:39:46,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:39:46,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:39:46,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:39:46,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:39:46,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:39:46,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:39:46,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:39:46,901 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:39:46,901 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:39:48,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:39:49,105 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:39:49,107 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 14736 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:39:51,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:39:51,335 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:39:51,335 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:39:53,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:39:53,519 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:39:53,520 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:39:55,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:39:55,701 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:39:55,701 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:39:57,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:39:57,973 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:39:57,973 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:39:59,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:00,473 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:40:00,473 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:40:02,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:02,638 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:40:02,638 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:40:04,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:04,871 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:40:04,872 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:40:06,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:07,101 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:40:07,101 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:40:09,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:09,263 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:40:09,265 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:40:11,265 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:11,437 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:40:11,437 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:40:13,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:13,594 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:40:13,594 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 28.0720000267. 2022-02-08 11:40:13,595 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:40:13,595 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:40:14,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:40:14,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:40:14,638 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:40:14,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:40:14,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:40:14,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:40:14,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:40:14,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:40:14,793 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:40:14,829 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:40:14,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:15,563 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:40:15,565 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 11:40:17,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:17,775 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:40:17,776 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 11:40:19,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:19,984 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:40:19,984 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 11:40:21,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:22,170 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:40:22,170 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 11:40:24,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:24,391 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:40:24,391 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 10.7969999313. 2022-02-08 11:40:24,392 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:40:24,394 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:40:25,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:40:26,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:40:26,371 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:40:26,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:40:26,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:40:26,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:40:26,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:40:26,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:40:26,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:40:26,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:40:26,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:27,309 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:40:27,309 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.91799998283. 2022-02-08 11:40:27,309 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:40:27,312 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:40:28,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:40:28,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:40:29,020 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:40:29,022 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:40:29,086 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:40:29,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:40:29,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:40:29,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:40:29,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:40:29,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:40:29,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:30,127 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:40:30,127 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.81700015068. 2022-02-08 11:40:30,127 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:40:30,128 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:40:30,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:40:31,306 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:40:31,398 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:40:31,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:40:31,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:40:31,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:40:31,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:40:31,494 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:40:31,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:40:31,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:40:31,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:32,002 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:40:32,003 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.87599992752. 2022-02-08 11:40:32,003 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:40:32,006 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:40:32,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:40:33,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:40:33,365 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:40:33,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:40:33,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:40:33,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:40:33,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:40:33,443 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:40:33,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:40:33,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:40:33,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:33,973 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:40:33,973 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.97000002861. 2022-02-08 11:40:33,973 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:40:33,974 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:40:34,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:40:34,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:40:35,250 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:40:35,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:40:35,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:40:35,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:40:35,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:40:35,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:40:35,556 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:40:35,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:40:35,591 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:36,677 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:40:36,677 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 11:40:38,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:38,904 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:40:38,904 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.115 2022-02-08 11:40:40,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:41,075 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:40:41,076 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.10399985313. 2022-02-08 11:40:41,076 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:40:41,078 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:40:41,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:40:42,257 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:40:42,381 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:40:42,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:40:42,451 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:40:42,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:40:42,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:40:42,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:40:42,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:40:42,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:40:42,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:40:43,717 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:40:43,717 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.64100003242. ############################################################ # # RAM FootPrint: 4919.43359375 # ############################################################ 2022-02-08 11:40:43,753 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 11:40:43,753 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 100 seconds. Reason: Iteration:16 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14741 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 14765 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:42:23,759 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 11:42:23,759 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 11:42:28,762 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:42:29,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:42:29,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:42:29,805 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:42:29,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:42:29,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:42:29,854 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:42:29,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:42:29,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:42:29,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:42:29,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:42:29,944 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:42:29,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:42:30,000 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['61730 memcached\n'], []) 2022-02-08 11:42:30,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 11:42:30,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:42:30,046 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['61730 memcached\n'], []) 2022-02-08 11:42:30,046 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:42:30,049 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:42:30,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:42:30,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:42:31,134 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:42:31,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:42:31,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:42:31,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:42:31,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:42:31,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:42:31,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:42:31,237 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:42:31,250 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:42:31,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:42:31,299 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['117066 memcached\n'], []) 2022-02-08 11:42:31,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 11:42:31,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:42:31,345 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['117066 memcached\n'], []) 2022-02-08 11:42:31,345 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:42:31,351 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:42:31,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:42:32,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:42:32,183 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:42:32,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:42:32,198 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:42:32,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:42:32,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:42:32,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:42:32,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:42:32,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:42:32,375 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:42:32,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:42:32,446 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['99904 memcached\n'], []) 2022-02-08 11:42:32,446 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 11:42:32,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:42:32,493 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['99904 memcached\n'], []) 2022-02-08 11:42:32,494 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:42:32,497 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:42:32,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:42:32,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:42:32,862 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:42:32,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:42:32,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:42:32,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:42:32,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:42:32,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:42:32,938 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:42:32,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:42:32,966 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:42:32,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:42:33,012 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['126102 memcached\n'], []) 2022-02-08 11:42:33,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 11:42:33,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:42:33,056 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['126102 memcached\n'], []) 2022-02-08 11:42:33,056 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:42:33,059 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:42:33,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:42:33,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:42:33,423 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:42:33,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:42:33,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:42:33,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:42:33,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:42:33,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:42:33,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:42:33,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:42:33,523 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:42:33,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:42:33,565 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['120001 memcached\n'], []) 2022-02-08 11:42:33,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 11:42:33,591 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:42:33,614 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['120001 memcached\n'], []) 2022-02-08 11:42:33,615 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:42:33,617 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:42:34,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:42:34,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:42:34,651 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:42:34,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:42:34,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:42:34,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:42:34,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:42:34,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:42:34,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:42:34,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:42:34,744 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:42:34,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:42:34,812 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['24463 memcached\n'], []) 2022-02-08 11:42:34,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 11:42:34,839 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:42:34,861 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['24463 memcached\n'], []) 2022-02-08 11:42:34,861 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:42:34,864 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:42:34,956 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:42:35,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:42:35,229 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:42:35,229 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:42:35,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:42:35,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:42:35,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:42:35,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:42:35,295 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:42:35,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:42:35,319 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:42:35,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:42:35,387 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['88261 memcached\n'], []) 2022-02-08 11:42:35,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 11:42:35,413 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:42:35,453 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['88261 memcached\n'], []) 2022-02-08 11:42:35,453 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:42:35,456 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:42:36,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:42:36,591 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:42:36,737 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:42:36,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:42:36,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:42:36,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:42:36,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:42:36,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:42:36,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:42:36,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:42:36,858 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:42:36,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:42:36,917 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['103550 memcached\n'], []) 2022-02-08 11:42:36,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 11:42:36,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:42:36,973 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['103550 memcached\n'], []) 2022-02-08 11:42:36,973 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:42:36,979 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:42:37,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:42:37,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:42:37,345 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:42:37,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:42:37,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:42:37,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:42:37,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:42:37,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:42:37,410 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:42:37,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:42:37,436 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:42:37,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:42:37,483 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['40091 memcached\n'], []) 2022-02-08 11:42:37,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 11:42:37,507 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:42:37,529 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['40091 memcached\n'], []) 2022-02-08 11:42:37,529 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:42:37,530 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:42:37,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:42:37,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:42:37,892 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:42:37,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:42:37,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:42:37,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:42:37,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:42:37,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:42:37,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:42:37,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:42:37,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:42:38,006 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 11:42:38,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:42:38,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:42:42,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:42:43,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:42:44,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:42:44,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:42:45,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:42:45,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:42:45,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:42:45,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:42:45,892 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 11:42:45,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:42:45,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:42:45,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:42:45,990 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 11:42:45,990 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:42:45,993 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:42:46,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:42:46,329 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:42:46,384 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:42:46,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:42:46,397 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:42:46,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:42:46,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:42:46,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:42:46,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:42:46,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:42:46,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:42:46,489 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 11:42:46,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:42:46,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 14787 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:42:52,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:42:53,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:42:53,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:42:54,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:42:54,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:42:54,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:42:54,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:42:54,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:42:54,834 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 11:42:54,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:42:54,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:42:54,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:42:54,941 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 11:42:54,941 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:42:54,951 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 11:42:55,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:42:55,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 11:42:55,326 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:42:55,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 11:42:55,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 11:42:55,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 11:42:55,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 11:42:55,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 11:42:55,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 11:42:55,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 11:42:55,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:42:55,443 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 11:42:55,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:42:55,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:42:55,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:42:55,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:42:55,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:42:55,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:42:55,551 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 11:42:55,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:42:55,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:42:55,581 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:42:55,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:42:55,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:42:55,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:42:55,655 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:42:55,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:42:55,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:42:55,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:42:55,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:42:55,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:42:55,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:42:55,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:42:55,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:42:55,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:42:55,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:42:55,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:42:55,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:42:55,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:42:55,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:42:55,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:42:55,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:42:55,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:42:55,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:42:56,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:42:56,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:42:56,043 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:42:56,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:42:56,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:42:56,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:42:56,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:42:56,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:42:56,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:42:56,160 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:42:56,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:42:56,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:42:56,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:42:56,227 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:42:56,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:42:56,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:42:56,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:42:56,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:42:56,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:42:56,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:42:56,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:42:56,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:42:56,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:42:56,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:42:56,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:42:56,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:42:56,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:42:56,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:42:56,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:42:56,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:42:56,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:42:56,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:42:56,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:42:56,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:42:56,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:42:56,604 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:42:56,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:42:56,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:42:56,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:42:56,668 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:42:56,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:42:56,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:42:56,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:42:56,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:42:56,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:42:56,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:42:56,778 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:42:56,793 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:42:56,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:42:56,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:42:56,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:42:56,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:42:56,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:42:56,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:42:56,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:42:56,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:42:56,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:42:56,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:42:56,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:42:56,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:42:56,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:42:57,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:42:57,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:42:57,043 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:42:57,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:42:57,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:42:57,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:42:57,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:42:57,124 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 11:42:57,125 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:42:57,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:42:57,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:42:57,500 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:42:57,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:42:57,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:42:57,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:42:57,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:42:57,556 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:42:57,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:42:57,584 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:42:57,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:42:57,611 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 11:42:57,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:42:57,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:00,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:43:00,987 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:01,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:01,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:43:01,756 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 11:43:01,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:43:01,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:01,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:43:01,857 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 11:43:01,858 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:43:01,859 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 11:43:01,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:43:02,160 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 11:43:02,219 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:43:02,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 11:43:02,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 11:43:02,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 11:43:02,260 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 11:43:02,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 11:43:02,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 11:43:02,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 11:43:02,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:43:02,331 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 11:43:02,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:43:02,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:02,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:43:02,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:02,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:02,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:43:02,444 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 11:43:02,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:43:02,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:02,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:43:02,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:02,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:43:02,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:02,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:02,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:02,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:02,604 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:02,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:02,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:02,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:02,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:02,696 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:43:02,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:02,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:02,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:02,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:02,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:02,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:02,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:02,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:02,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:02,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:43:02,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:02,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:02,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:02,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:02,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:02,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:02,982 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:02,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:03,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:03,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:43:03,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:03,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:03,079 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:03,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:03,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:03,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:03,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:03,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:03,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:03,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:43:03,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:03,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:03,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:03,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:03,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:03,303 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:03,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:03,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:03,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:03,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:43:03,397 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:03,413 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:03,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:03,443 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:03,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:03,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:03,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:03,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:03,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:03,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:43:03,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:03,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:03,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:03,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:03,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:03,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:03,655 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:03,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:03,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:03,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:43:03,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:03,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:03,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:03,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:03,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:03,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:03,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:03,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:03,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:03,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:43:03,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:03,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:03,897 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:03,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:03,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:03,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:03,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:03,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:03,989 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 11:43:03,990 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:43:04,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:43:04,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:43:04,361 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:43:04,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:43:04,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:43:04,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:43:04,403 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:43:04,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:43:04,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:43:04,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:43:04,459 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:43:04,473 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 11:43:04,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:43:04,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:06,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:43:06,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:07,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:07,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:09,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:43:09,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:43:09,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:43:09,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:43:09,563 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 11:43:09,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:43:09,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:09,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:43:09,644 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 11:43:09,644 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:43:09,645 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:43:09,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:43:09,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:43:10,036 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:43:10,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:43:10,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:43:10,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:43:10,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:43:10,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:43:10,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:43:10,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:43:10,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:43:10,157 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 11:43:10,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:43:10,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:14,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:43:15,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:15,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:16,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:17,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:43:18,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:43:18,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:43:18,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:43:18,391 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 11:43:18,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:43:18,410 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:18,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:43:18,720 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 11:43:18,721 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:43:18,724 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 11:43:18,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:43:19,071 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 11:43:19,148 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:43:19,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 11:43:19,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 11:43:19,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 11:43:19,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 11:43:19,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 11:43:19,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 11:43:19,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 11:43:19,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:43:19,313 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 11:43:19,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:43:19,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:19,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:43:19,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:19,426 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:19,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:43:19,461 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 11:43:19,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:43:19,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:19,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:43:19,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:19,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:43:19,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:19,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:19,627 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:19,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:19,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:19,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:19,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:19,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:19,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:19,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:43:19,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:19,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:19,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:19,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:19,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:19,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:19,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:19,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:20,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:20,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:43:20,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:20,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:20,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:20,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:20,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:20,163 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:20,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:20,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:20,229 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:20,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:43:20,272 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:20,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:20,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:20,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:20,362 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:20,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:20,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:20,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:20,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:20,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:43:20,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:20,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:20,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:20,573 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:20,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:20,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:20,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:20,660 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:20,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:20,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:43:20,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:20,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:20,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:20,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:20,803 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:20,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:20,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:20,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:20,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:20,910 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:43:20,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:20,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:20,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:20,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:21,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:21,022 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:21,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:21,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:21,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:21,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:43:21,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:21,132 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:21,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:21,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:21,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:21,214 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:21,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:21,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:21,272 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:21,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:43:21,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:21,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:21,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:21,374 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:21,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:21,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:21,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:21,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:21,473 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 11:43:21,476 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 11:43:21,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:43:21,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 11:43:21,852 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:43:21,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 11:43:21,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 11:43:21,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 11:43:21,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 11:43:21,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 11:43:21,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 11:43:21,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 11:43:21,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:43:21,980 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 11:43:22,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:43:22,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:22,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:43:22,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:22,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:22,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:43:22,102 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 11:43:22,102 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:43:22,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:22,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:43:22,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:22,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:43:22,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:22,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:22,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:22,255 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:22,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:22,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:22,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:22,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:22,367 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:22,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:43:22,403 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:22,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:22,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:22,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:22,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:22,482 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:22,497 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:22,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:22,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:22,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:43:22,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:22,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:22,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:22,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:22,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:22,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:22,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:22,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:22,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:22,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:43:22,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:22,778 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:22,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:22,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:22,834 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:22,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:22,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:22,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:22,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:22,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:43:22,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:22,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:22,982 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:23,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:23,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:23,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:23,056 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:23,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:23,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:23,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:43:23,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:23,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:23,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:23,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:23,198 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:23,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:23,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:23,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:23,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:23,285 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:43:23,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:23,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:23,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:23,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:23,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:23,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:23,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:23,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:23,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:23,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:43:23,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:23,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:23,507 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:23,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:23,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:23,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:23,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:23,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:23,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:23,627 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:43:23,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:23,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:23,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:23,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:23,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:23,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:23,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:23,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:23,788 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 11:43:23,789 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:43:23,969 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:43:24,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:43:24,316 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:43:24,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:43:24,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:43:24,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:43:24,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:43:24,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:43:24,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:43:24,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:43:24,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:43:24,474 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 11:43:24,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:43:24,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:26,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:43:27,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:28,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:28,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:29,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:43:29,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:43:29,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:43:29,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:43:29,836 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 11:43:29,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:43:29,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:29,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:43:29,971 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 11:43:29,973 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:43:29,976 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:43:30,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:43:30,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:43:30,421 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:43:30,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:43:30,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:43:30,454 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:43:30,469 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:43:30,487 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:43:30,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:43:30,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:43:30,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:43:30,561 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 11:43:30,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:43:30,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:32,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:43:32,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:33,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:33,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:43:33,821 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 11:43:33,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:43:33,836 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:33,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:43:33,901 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 11:43:33,901 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:43:33,903 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:43:33,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:43:34,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:43:34,328 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:43:34,329 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:43:34,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:43:34,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:43:34,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:43:34,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:43:34,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:43:34,443 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:43:34,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:43:34,480 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 11:43:34,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:43:34,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:40,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:43:40,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:41,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:41,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:43:41,598 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 11:43:41,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:43:41,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:41,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:43:41,688 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 11:43:41,688 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:43:41,690 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:43:41,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:43:42,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:43:42,178 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:43:42,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:43:42,198 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:43:42,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:43:42,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:43:42,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:43:42,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:43:42,278 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:43:42,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:43:42,308 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 11:43:42,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:43:42,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:46,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:43:47,979 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:48,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:49,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 14799 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:43:51,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:43:51,535 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:43:51,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:43:51,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:43:51,914 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 11:43:51,915 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:43:51,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:51,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:43:51,976 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 11:43:51,977 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:43:51,982 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 11:43:52,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:43:52,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 11:43:52,391 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:43:52,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 11:43:52,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 11:43:52,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 11:43:52,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 11:43:52,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 11:43:52,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 11:43:52,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 11:43:52,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:43:52,506 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 11:43:52,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:43:52,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:52,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:43:52,579 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:52,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:43:52,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:43:52,621 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 11:43:52,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:43:52,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:43:52,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:43:52,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:52,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:43:52,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:52,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:52,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:52,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:52,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:52,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:52,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:52,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:43:52,885 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:52,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:43:52,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:52,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:52,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:52,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:52,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:53,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:53,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:53,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:43:53,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:53,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:43:53,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:53,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:53,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:53,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:53,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:53,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:53,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:53,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:43:53,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:53,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:43:53,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:53,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:53,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:53,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:53,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:53,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:53,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:53,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:43:53,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:53,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:43:53,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:53,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:53,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:53,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:53,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:53,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:53,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:53,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:43:53,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:53,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:43:53,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:53,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:53,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:53,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:53,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:53,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:53,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:53,767 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:43:53,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:53,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:43:53,816 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:53,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:53,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:53,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:53,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:53,900 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:53,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:53,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:43:53,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:53,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:43:53,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:54,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:54,017 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:54,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:54,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:54,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:54,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:54,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:43:54,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:54,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:43:54,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:54,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:54,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:54,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:54,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:54,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:54,290 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:54,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:43:54,325 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 11:43:54,325 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:43:54,328 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:43:54,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:43:54,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:43:54,816 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:43:54,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:43:54,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:43:54,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:43:54,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:43:54,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:43:54,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:43:54,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:43:54,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:43:55,424 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:43:55,426 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.10000014305. 2022-02-08 11:43:55,426 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:43:55,427 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:43:55,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:43:55,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:43:55,930 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:43:55,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:43:55,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:43:55,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:43:55,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:43:55,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:43:56,007 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:43:56,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:43:56,043 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:43:56,207 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:43:56,207 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:43:58,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:43:58,395 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:43:58,397 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:44:00,397 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:00,553 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:44:00,559 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:44:02,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:02,724 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:44:02,724 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:44:04,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:04,894 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:44:04,894 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:44:06,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:07,069 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:44:07,069 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:44:09,071 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:09,326 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:44:09,328 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:44:11,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:11,657 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:44:11,657 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:44:13,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:13,871 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:44:13,871 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:44:15,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:16,058 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:44:16,059 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:44:18,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:18,253 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:44:18,253 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:44:20,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:20,520 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:44:20,520 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:44:22,520 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:22,765 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:44:22,765 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:44:24,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:24,924 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:44:24,924 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 29.4989998341. 2022-02-08 11:44:24,924 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:44:24,927 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:44:25,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:44:26,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:44:26,627 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:44:26,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:44:26,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:44:26,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:44:26,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:44:26,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:44:26,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:44:26,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:44:26,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:27,578 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:44:27,578 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.65300011635. 2022-02-08 11:44:27,578 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:44:27,581 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:44:28,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:44:29,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:44:29,545 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:44:29,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:44:29,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:44:29,594 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:44:29,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:44:29,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:44:29,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:44:29,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:44:29,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:30,579 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:44:30,581 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.00200009346. 2022-02-08 11:44:30,581 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:44:30,582 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:44:31,640 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:44:32,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:44:32,464 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:44:32,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:44:32,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:44:32,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:44:32,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:44:32,556 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:44:32,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:44:32,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:44:32,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:33,296 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:44:33,298 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.71700000763. 2022-02-08 11:44:33,298 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:44:33,299 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:44:34,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:44:35,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:44:35,365 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:44:35,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:44:35,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:44:35,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:44:35,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:44:35,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:44:35,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:44:35,535 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:44:35,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:36,295 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:44:36,296 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.99799990654. 2022-02-08 11:44:36,296 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:44:36,298 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:44:37,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:44:37,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:44:37,671 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:44:37,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:44:37,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:44:37,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:44:37,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:44:37,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:44:37,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:44:37,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:44:37,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:38,260 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:44:38,260 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.96499991417. 2022-02-08 11:44:38,262 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:44:38,263 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:44:38,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:44:39,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:44:39,200 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:44:39,201 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:44:39,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:44:39,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:44:39,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:44:39,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:44:39,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:44:39,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:44:39,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:39,959 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:44:39,960 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.6970000267. 2022-02-08 11:44:39,960 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:44:39,961 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:44:40,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:44:41,591 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:44:41,730 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:44:41,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:44:41,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:44:41,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:44:41,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:44:41,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:44:41,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:44:41,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:44:42,017 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:44:43,069 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:44:43,069 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.10899996758. ############################################################ # # RAM FootPrint: 4922.8515625 # ############################################################ 2022-02-08 11:44:43,108 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 11:44:43,108 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 91 seconds. Reason: Iteration:17 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14804 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 14829 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:46:14,115 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 11:46:14,115 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 11:46:19,118 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:46:19,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:46:19,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:46:19,947 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:46:19,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:46:19,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:46:19,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:46:19,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:46:20,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:46:20,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:46:20,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:46:20,081 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:46:20,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:46:20,134 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['62456 memcached\n'], []) 2022-02-08 11:46:20,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 11:46:20,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:46:20,183 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['62456 memcached\n'], []) 2022-02-08 11:46:20,183 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:46:20,186 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:46:20,367 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:46:20,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:46:20,676 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:46:20,676 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:46:20,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:46:20,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:46:20,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:46:20,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:46:20,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:46:20,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:46:20,769 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:46:20,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:46:20,831 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['117967 memcached\n'], []) 2022-02-08 11:46:20,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 11:46:20,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:46:20,878 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['117967 memcached\n'], []) 2022-02-08 11:46:20,878 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:46:20,881 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:46:20,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:46:21,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:46:21,279 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:46:21,280 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:46:21,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:46:21,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:46:21,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:46:21,346 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:46:21,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:46:21,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:46:21,394 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:46:21,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:46:21,460 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['100828 memcached\n'], []) 2022-02-08 11:46:21,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 11:46:21,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:46:21,509 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['100828 memcached\n'], []) 2022-02-08 11:46:21,509 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:46:21,512 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:46:21,591 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:46:21,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:46:21,871 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:46:21,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:46:21,887 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:46:21,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:46:21,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:46:21,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:46:21,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:46:21,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:46:21,973 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:46:21,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:46:22,033 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['126890 memcached\n'], []) 2022-02-08 11:46:22,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 11:46:22,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:46:22,079 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['126890 memcached\n'], []) 2022-02-08 11:46:22,079 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:46:22,082 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:46:22,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:46:22,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:46:22,430 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:46:22,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:46:22,446 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:46:22,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:46:22,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:46:22,487 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:46:22,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:46:22,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:46:22,526 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:46:22,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:46:22,573 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['120760 memcached\n'], []) 2022-02-08 11:46:22,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 11:46:22,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:46:22,632 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['120760 memcached\n'], []) 2022-02-08 11:46:22,632 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:46:22,637 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:46:22,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:46:22,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:46:22,997 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:46:22,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:46:23,012 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:46:23,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:46:23,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:46:23,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:46:23,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:46:23,079 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:46:23,095 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:46:23,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:46:23,138 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['25222 memcached\n'], []) 2022-02-08 11:46:23,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 11:46:23,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:46:23,184 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['25222 memcached\n'], []) 2022-02-08 11:46:23,184 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:46:23,187 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:46:23,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:46:23,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:46:23,539 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:46:23,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:46:23,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:46:23,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:46:23,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:46:23,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:46:23,604 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:46:23,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:46:23,630 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:46:23,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:46:23,678 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['88998 memcached\n'], []) 2022-02-08 11:46:23,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 11:46:23,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:46:23,720 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['88998 memcached\n'], []) 2022-02-08 11:46:23,721 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:46:23,723 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:46:24,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:46:24,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:46:24,990 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:46:24,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:46:25,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:46:25,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:46:25,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:46:25,056 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:46:25,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:46:25,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:46:25,121 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:46:25,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:46:25,200 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['104334 memcached\n'], []) 2022-02-08 11:46:25,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 11:46:25,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:46:25,246 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['104334 memcached\n'], []) 2022-02-08 11:46:25,247 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:46:25,250 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:46:25,331 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:46:25,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:46:25,618 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:46:25,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:46:25,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:46:25,645 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:46:25,660 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:46:25,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:46:25,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:46:25,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:46:25,730 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:46:25,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:46:25,789 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['40854 memcached\n'], []) 2022-02-08 11:46:25,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 11:46:25,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:46:25,835 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['40854 memcached\n'], []) 2022-02-08 11:46:25,835 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:46:25,838 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:46:25,910 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:46:26,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:46:26,210 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:46:26,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:46:26,224 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:46:26,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:46:26,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:46:26,272 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:46:26,285 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:46:26,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:46:26,313 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:46:26,328 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 11:46:26,357 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:46:26,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:46:29,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:46:30,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:46:31,012 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:46:31,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:46:31,660 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:46:31,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:46:31,923 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:46:32,007 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:46:32,022 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 11:46:32,022 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:46:32,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:46:32,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:46:32,119 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 11:46:32,121 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:46:32,122 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:46:32,196 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:46:32,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:46:32,463 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:46:32,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:46:32,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:46:32,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:46:32,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:46:32,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:46:32,538 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:46:32,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:46:32,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:46:32,586 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 11:46:32,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:46:32,627 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:46:36,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:46:36,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:46:37,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:46:37,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:46:38,260 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:46:38,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:46:38,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:46:38,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:46:38,483 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 11:46:38,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:46:38,497 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:46:38,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:46:38,576 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 11:46:38,578 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:46:38,581 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 11:46:38,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:46:38,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 11:46:38,983 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:46:38,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 11:46:39,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 11:46:39,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 11:46:39,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 11:46:39,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 11:46:39,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 11:46:39,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 11:46:39,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:46:39,111 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 11:46:39,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:46:39,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:46:39,173 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:46:39,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:46:39,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:46:39,221 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:46:39,236 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 11:46:39,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:46:39,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:46:39,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:46:39,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:39,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:46:39,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:39,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:39,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:39,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:39,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:39,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:39,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:39,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:39,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:46:39,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:46:39,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:46:39,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:46:39,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:46:39,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:46:39,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:46:39,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:46:39,655 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:46:39,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:46:39,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:39,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:46:39,729 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:39,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:39,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:39,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:39,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:39,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:39,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:39,854 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:39,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:39,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:46:39,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:39,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:39,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:39,966 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:39,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:40,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:40,022 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:40,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:40,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:40,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:46:40,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:40,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:40,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:40,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:40,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:40,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:40,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:40,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:40,244 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:40,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:46:40,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:40,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:40,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:40,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:40,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:40,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:40,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:40,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:40,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:40,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:46:40,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:40,474 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:40,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:40,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:40,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:40,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:40,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:40,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:40,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:40,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:46:40,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:40,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:40,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:40,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:40,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:40,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:40,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:40,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:40,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:40,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:46:40,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:40,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:40,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:40,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:40,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:40,885 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:40,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:40,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:40,937 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 11:46:40,940 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:46:41,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:46:41,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:46:42,240 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:46:42,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:46:42,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:46:42,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:46:42,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:46:42,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:46:42,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:46:42,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:46:43,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:46:43,145 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 11:46:43,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:46:43,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:46:47,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:46:48,163 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:46:48,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:46:48,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:46:48,890 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 11:46:48,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:46:48,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:46:48,966 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:46:48,980 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 11:46:48,980 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:46:48,982 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 11:46:49,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:46:49,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 11:46:49,372 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:46:49,374 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 11:46:49,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 11:46:49,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 11:46:49,410 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 11:46:49,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 11:46:49,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 11:46:49,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 11:46:49,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:46:49,476 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 11:46:49,507 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:46:49,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:46:49,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:46:49,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:46:49,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:46:49,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:46:49,589 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 11:46:49,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:46:49,604 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:46:49,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:46:49,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:49,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:46:49,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:49,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:49,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:49,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:49,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:49,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:49,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:49,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:46:49,839 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:46:49,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:46:49,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 5 queries submitted, 0 failed, 14848 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:46:49,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:46:49,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:46:49,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:46:49,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:46:49,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:46:49,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:46:49,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:46:49,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:50,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:46:50,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:50,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:50,061 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:50,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:50,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:50,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:50,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:50,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:46:50,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:50,173 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:46:50,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:50,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:50,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:50,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:50,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:50,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:50,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:50,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:46:50,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:50,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:46:50,357 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:50,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:50,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:50,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:50,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:50,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:50,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:50,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:46:50,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:50,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:46:50,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:50,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:50,579 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:50,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:50,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:50,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:50,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:50,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:46:50,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:50,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:46:50,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:50,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:50,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:50,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:50,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:50,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:50,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:50,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:46:50,816 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:50,834 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:46:50,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:50,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:50,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:50,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:50,913 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:50,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:50,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:50,956 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:46:50,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:50,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:46:51,002 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:51,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:51,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:51,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:51,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:51,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:51,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:51,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:46:51,128 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 11:46:51,130 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:46:51,214 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:46:51,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:46:51,502 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:46:51,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:46:51,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:46:51,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:46:51,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:46:51,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:46:51,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:46:51,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:46:51,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:46:51,627 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 11:46:51,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:46:51,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:46:54,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:46:55,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:46:55,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:46:56,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:46:56,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:46:56,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:46:57,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:46:57,127 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:46:57,142 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 11:46:57,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:46:57,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:46:57,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:46:57,256 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 11:46:57,256 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:46:57,259 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:46:57,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:46:57,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:46:57,654 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:46:57,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:46:57,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:46:57,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:46:57,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:46:57,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:46:57,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:46:57,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:46:57,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:46:57,782 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 11:46:57,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:46:57,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:00,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:47:01,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:47:01,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:47:02,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:02,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:47:02,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:47:02,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:47:02,811 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:47:02,828 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 11:47:02,829 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:47:02,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:02,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:47:02,934 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 11:47:02,934 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:47:02,937 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 11:47:03,017 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:47:03,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 11:47:03,325 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:47:03,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 11:47:03,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 11:47:03,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 11:47:03,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 11:47:03,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 11:47:03,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 11:47:03,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 11:47:03,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:47:03,453 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 11:47:03,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:47:03,497 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:03,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:47:03,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:47:03,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:47:03,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:47:03,584 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 11:47:03,584 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:47:03,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:03,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:47:03,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:03,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:47:03,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:03,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:03,729 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:03,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:03,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:03,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:03,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:03,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:03,867 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:03,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:47:03,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:03,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:03,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:03,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:03,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:03,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:04,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:04,017 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:04,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:04,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:47:04,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:04,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:04,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:04,132 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:04,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:04,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:04,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:04,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:04,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:04,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:47:04,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:04,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:04,367 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:04,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:04,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:04,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:04,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:04,459 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:04,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:04,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:47:04,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:04,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:04,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:04,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:04,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:04,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:04,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:04,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:04,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:04,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:47:04,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:04,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:04,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:04,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:04,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:04,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:04,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:04,836 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:04,854 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:04,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:47:04,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:04,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:04,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:04,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:04,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:04,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:04,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:05,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:05,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:05,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:47:05,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:05,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:05,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:05,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:05,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:05,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:05,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:05,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:05,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:05,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:47:05,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:05,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:05,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:05,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:05,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:05,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:05,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:05,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:05,424 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 11:47:05,427 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 11:47:05,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:47:05,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 11:47:05,792 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:47:05,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 11:47:05,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 11:47:05,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 11:47:05,834 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 11:47:05,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 11:47:05,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 11:47:05,887 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 11:47:05,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:47:05,920 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 11:47:05,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:47:05,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:06,002 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:47:06,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:47:06,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:47:06,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:47:06,065 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 11:47:06,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:47:06,079 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:06,096 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:47:06,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:06,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:47:06,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:06,183 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:06,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:06,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:06,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:06,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:06,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:06,326 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:06,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:06,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:47:06,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:06,397 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:06,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:06,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:06,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:06,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:06,487 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:06,505 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:06,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:06,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:47:06,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:06,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:06,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:06,607 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:06,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:06,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:06,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:06,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:06,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:06,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:47:06,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:06,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:06,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:06,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:06,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:06,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:06,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:06,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:06,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:06,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:47:06,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:06,956 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:06,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:06,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:07,012 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:07,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:07,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:07,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:07,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:07,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:47:07,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:07,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:07,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:07,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:07,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:07,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:07,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:07,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:07,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:07,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:47:07,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:07,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:07,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:07,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:07,349 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:07,367 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:07,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:07,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:07,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:07,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:47:07,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:07,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:07,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:07,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:07,512 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:07,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:07,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:07,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:07,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:07,594 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:47:07,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:07,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:07,647 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:07,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:07,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:07,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:07,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:07,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:07,755 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 11:47:07,756 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:47:07,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:47:08,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:47:08,321 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:47:08,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:47:08,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:47:08,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:47:08,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:47:08,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:47:08,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:47:08,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:47:08,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:47:08,746 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 11:47:08,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:47:08,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:11,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:47:12,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:47:13,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:47:14,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:14,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:47:14,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:47:14,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:47:15,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:47:15,081 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 11:47:15,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:47:15,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:15,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:47:15,255 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 11:47:15,255 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:47:15,256 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:47:15,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:47:15,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:47:15,701 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:47:15,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:47:15,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:47:15,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:47:15,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:47:15,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:47:15,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:47:15,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:47:15,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:47:15,849 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 11:47:15,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:47:15,897 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:19,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:47:20,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:47:21,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:47:21,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:47:21,641 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 11:47:21,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:47:21,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:21,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:47:21,730 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 11:47:21,730 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:47:21,733 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:47:21,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:47:22,079 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:47:22,151 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:47:22,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:47:22,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:47:22,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:47:22,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:47:22,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:47:22,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:47:22,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:47:22,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:47:22,296 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 11:47:22,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:47:22,336 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:28,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:47:29,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:47:29,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:47:29,918 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:47:29,934 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 11:47:29,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:47:29,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:30,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:47:30,032 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 11:47:30,032 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:47:30,033 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:47:30,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:47:30,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:47:30,460 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:47:30,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:47:30,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:47:30,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:47:30,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:47:30,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:47:30,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:47:30,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:47:30,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:47:30,614 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 11:47:30,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:47:30,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:33,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:47:34,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:47:35,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:47:35,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:35,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:47:36,221 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:47:36,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:47:36,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:47:36,509 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 11:47:36,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:47:36,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:36,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:47:36,565 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 11:47:36,565 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:47:36,568 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 11:47:36,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:47:36,915 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 11:47:36,990 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:47:36,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 11:47:37,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 11:47:37,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 11:47:37,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 11:47:37,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 11:47:37,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 11:47:37,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 11:47:37,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:47:37,121 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 11:47:37,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:47:37,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:37,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:47:37,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:47:37,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:47:37,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:47:37,246 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 11:47:37,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:47:37,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:47:37,278 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:47:37,295 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:37,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:47:37,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:37,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:37,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:37,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:37,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:37,454 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:37,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:37,505 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:47:37,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:37,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:47:37,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:37,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:37,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:37,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:37,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:37,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:37,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:37,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:47:37,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:37,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:47:37,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:37,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:37,795 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:37,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:37,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:37,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:37,864 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:37,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:47:37,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:37,923 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:47:37,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:37,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:37,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:37,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:38,017 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:38,038 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:38,056 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:38,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:47:38,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:38,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:47:38,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:38,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:38,173 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:38,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:38,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:38,227 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:38,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:38,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:47:38,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:38,303 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:47:38,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:38,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:38,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:38,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:38,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:38,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:38,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:38,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:47:38,459 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:38,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:47:38,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:38,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:38,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:38,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:38,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:38,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:38,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:38,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:47:38,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:38,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:47:38,660 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:38,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:38,696 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:38,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:38,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:38,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:38,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:38,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:47:38,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:38,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:47:38,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:38,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:38,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:38,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:38,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:38,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:38,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:38,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:47:38,999 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 11:47:39,000 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:47:39,002 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:47:39,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:47:39,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:47:39,447 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:47:39,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:47:39,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:47:39,482 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:47:39,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:47:39,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:47:39,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:47:39,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:47:39,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:47:40,092 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:47:40,094 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.09400010109. 2022-02-08 11:47:40,094 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:47:40,095 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:47:40,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:47:41,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:47:41,313 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:47:41,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:47:41,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:47:41,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:47:41,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:47:41,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:47:41,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:47:41,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:47:41,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:47:41,967 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:47:41,967 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:47:43,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:47:44,164 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:47:44,164 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:47:46,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:47:46,375 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:47:46,377 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:47:48,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:47:48,599 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:47:48,599 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 14849 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:47:50,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:47:50,825 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:47:50,825 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:47:52,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:47:53,043 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:47:53,043 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:47:55,043 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:47:55,318 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:47:55,319 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:47:57,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:47:57,732 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:47:57,732 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:47:59,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:47:59,970 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:47:59,970 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:48:01,971 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:48:02,165 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:48:02,165 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:48:04,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:48:04,357 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:48:04,358 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:48:06,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:48:06,529 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:48:06,529 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:48:08,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:48:08,763 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:48:08,763 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:48:10,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:48:10,940 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:48:10,940 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 30.8469998837. 2022-02-08 11:48:10,941 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:48:10,943 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:48:12,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:48:12,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:48:12,894 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:48:12,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:48:12,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:48:12,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:48:13,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:48:13,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:48:13,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:48:13,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:48:13,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:48:13,928 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:48:13,928 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 11:48:15,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:48:16,147 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:48:16,148 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 11:48:18,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:48:18,319 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:48:18,319 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 11:48:20,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:48:20,539 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:48:20,539 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.124 2022-02-08 11:48:22,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:48:22,726 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:48:22,727 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 11.7860000134. 2022-02-08 11:48:22,727 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:48:22,730 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:48:23,793 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:48:24,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:48:24,650 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:48:24,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:48:24,668 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:48:24,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:48:24,716 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:48:24,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:48:24,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:48:24,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:48:24,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:48:25,536 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:48:25,536 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.80900001526. 2022-02-08 11:48:25,538 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:48:25,539 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:48:26,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:48:27,346 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:48:27,592 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:48:27,594 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:48:27,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:48:27,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:48:27,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:48:27,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:48:27,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:48:27,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:48:27,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:48:28,536 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:48:28,536 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.99799990654. 2022-02-08 11:48:28,536 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:48:28,538 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:48:29,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:48:30,086 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:48:30,209 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:48:30,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:48:30,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:48:30,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:48:30,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:48:30,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:48:30,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:48:30,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:48:30,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:48:31,371 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:48:31,371 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.129 2022-02-08 11:48:33,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:48:33,645 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:48:33,645 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.1099998951. 2022-02-08 11:48:33,645 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:48:33,648 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:48:34,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:48:35,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:48:35,325 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:48:35,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:48:35,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:48:35,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:48:35,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:48:35,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:48:35,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:48:35,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:48:35,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:48:36,059 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:48:36,061 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.41500020027. 2022-02-08 11:48:36,061 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:48:36,062 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:48:37,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:48:37,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:48:37,895 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:48:37,897 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:48:37,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:48:38,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:48:38,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:48:38,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:48:38,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:48:38,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:48:38,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:48:40,187 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:48:40,187 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.12599992752. 2022-02-08 11:48:40,187 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:48:40,190 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:48:41,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:48:41,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:48:41,802 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:48:41,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:48:41,816 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:48:41,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:48:41,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:48:41,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:48:41,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:48:41,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:48:41,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:48:42,559 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:48:42,561 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.37300014496. ############################################################ # # RAM FootPrint: 4942.85546875 # ############################################################ 2022-02-08 11:48:42,591 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 11:48:42,591 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 61 seconds. Reason: Iteration:18 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14859 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:49:43,598 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 11:49:43,598 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 11:49:48,601 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:49:49,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:49:49,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:49:49,415 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:49:49,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:49:49,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:49:49,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:49:49,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:49:49,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:49:49,497 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:49:49,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:49:49,539 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:49:49,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:49:49,611 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['63177 memcached\n'], []) 2022-02-08 11:49:49,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 11:49:49,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:49:49,680 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['63177 memcached\n'], []) 2022-02-08 11:49:49,680 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:49:49,683 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 5 queries submitted, 0 failed, 14884 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:49:50,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:49:50,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:49:50,490 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:49:50,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:49:50,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:49:50,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:49:50,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:49:50,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:49:50,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:49:50,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:49:50,634 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:49:50,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:49:50,701 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['118871 memcached\n'], []) 2022-02-08 11:49:50,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 11:49:50,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:49:50,750 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['118871 memcached\n'], []) 2022-02-08 11:49:50,752 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:49:50,753 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:49:50,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:49:51,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:49:51,184 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:49:51,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:49:51,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:49:51,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:49:51,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:49:51,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:49:51,265 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:49:51,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:49:51,305 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:49:51,306 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:49:51,358 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['101675 memcached\n'], []) 2022-02-08 11:49:51,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 11:49:51,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:49:51,414 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['101675 memcached\n'], []) 2022-02-08 11:49:51,414 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:49:51,415 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:49:51,507 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:49:51,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:49:51,812 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:49:51,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:49:51,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:49:51,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:49:51,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:49:51,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:49:51,885 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:49:51,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:49:51,913 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:49:51,913 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:49:51,957 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['127605 memcached\n'], []) 2022-02-08 11:49:51,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 11:49:51,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:49:52,013 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['127605 memcached\n'], []) 2022-02-08 11:49:52,013 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:49:52,015 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:49:52,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:49:52,306 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:49:52,367 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:49:52,367 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:49:52,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:49:52,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:49:52,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:49:52,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:49:52,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:49:52,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:49:52,469 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:49:52,469 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:49:52,519 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['121505 memcached\n'], []) 2022-02-08 11:49:52,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 11:49:52,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:49:52,568 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['121505 memcached\n'], []) 2022-02-08 11:49:52,569 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:49:52,571 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:49:53,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:49:53,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:49:53,762 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:49:53,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:49:53,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:49:53,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:49:53,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:49:53,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:49:53,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:49:53,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:49:53,884 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:49:53,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:49:53,951 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['25968 memcached\n'], []) 2022-02-08 11:49:53,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 11:49:53,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:49:54,000 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['25968 memcached\n'], []) 2022-02-08 11:49:54,000 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:49:54,003 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:49:54,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:49:54,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:49:54,381 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:49:54,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:49:54,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:49:54,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:49:54,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:49:54,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:49:54,454 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:49:54,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:49:54,483 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:49:54,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:49:54,532 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['89715 memcached\n'], []) 2022-02-08 11:49:54,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 11:49:54,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:49:54,579 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['89715 memcached\n'], []) 2022-02-08 11:49:54,581 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:49:54,585 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:49:55,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:49:55,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:49:55,871 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:49:55,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:49:55,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:49:55,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:49:55,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:49:55,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:49:55,961 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:49:55,989 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:49:56,019 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:49:56,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:49:56,081 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['105022 memcached\n'], []) 2022-02-08 11:49:56,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 11:49:56,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:49:56,130 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['105022 memcached\n'], []) 2022-02-08 11:49:56,131 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:49:56,132 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:49:56,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:49:56,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:49:56,490 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:49:56,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:49:56,505 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:49:56,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:49:56,535 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:49:56,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:49:56,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:49:56,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:49:56,599 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:49:56,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:49:56,661 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['41600 memcached\n'], []) 2022-02-08 11:49:56,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 11:49:56,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:49:56,710 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['41600 memcached\n'], []) 2022-02-08 11:49:56,710 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:49:56,713 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:49:56,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:49:57,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:49:57,118 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:49:57,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:49:57,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:49:57,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:49:57,160 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:49:57,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:49:57,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:49:57,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:49:57,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:49:57,233 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 11:49:57,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:49:57,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:03,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:50:04,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:05,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:05,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:06,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:50:07,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:50:07,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:50:07,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:50:07,210 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 11:50:07,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:50:07,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:07,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:50:07,312 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 11:50:07,312 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:50:07,315 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:50:07,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:50:07,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:50:07,665 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:50:07,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:50:07,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:50:07,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:50:07,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:50:07,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:50:07,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:50:07,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:50:07,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:50:07,779 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 11:50:07,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:50:07,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:14,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:50:15,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:16,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:16,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:17,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:50:17,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:50:17,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:50:17,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:50:17,448 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 11:50:17,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:50:17,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:17,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:50:17,549 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 11:50:17,549 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:50:17,552 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 11:50:17,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:50:17,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 11:50:17,927 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:50:17,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 11:50:17,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 11:50:17,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 11:50:17,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 11:50:17,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 11:50:18,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 11:50:18,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 11:50:18,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:50:18,066 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 11:50:18,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:50:18,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:18,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:50:18,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:18,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:18,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:50:18,210 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 11:50:18,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:50:18,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:18,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:50:18,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:18,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:50:18,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:18,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:18,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:18,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:18,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:18,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:18,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:18,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:18,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:18,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:50:18,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:18,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:18,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:18,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:18,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:18,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:18,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:18,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:18,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:18,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:50:18,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:18,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:18,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:18,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:18,778 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:18,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:18,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:18,829 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:18,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:18,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:50:18,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:18,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:18,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:18,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:18,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:18,969 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:18,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:19,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:19,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:19,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:50:19,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:19,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:19,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:19,127 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:19,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:19,163 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:19,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:19,198 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:19,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:19,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:50:19,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:19,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:19,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:19,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:19,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:19,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:19,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:19,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:19,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:19,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:50:19,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:19,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:19,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:19,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:19,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:19,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:19,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:19,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:19,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:19,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:50:19,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:19,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:19,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:19,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:19,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:19,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:19,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:19,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:19,721 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:19,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:50:19,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:19,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:19,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:19,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:19,829 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:19,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:19,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:19,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:19,898 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 11:50:19,900 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:50:20,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:50:20,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:50:20,596 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:50:20,596 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:50:20,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:50:20,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:50:20,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:50:20,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:50:20,667 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:50:20,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:50:20,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:50:20,707 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 11:50:20,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:50:20,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:22,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:50:22,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:23,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:23,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:50:23,744 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 11:50:23,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:50:23,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:23,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:50:23,839 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 11:50:23,841 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:50:23,842 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 11:50:23,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:50:24,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 11:50:24,206 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:50:24,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 11:50:24,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 11:50:24,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 11:50:24,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 11:50:24,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 11:50:24,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 11:50:24,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 11:50:24,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:50:24,329 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 11:50:24,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:50:24,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:24,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:50:24,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:24,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:24,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:50:24,447 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 11:50:24,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:50:24,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:24,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:50:24,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:24,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:50:24,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:24,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:24,584 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:24,604 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:24,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:24,647 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:24,668 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:24,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:24,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:24,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:50:24,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:24,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:24,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:24,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:24,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:24,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:24,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:24,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:24,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:24,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:50:24,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:24,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:24,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:24,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:24,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:24,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:24,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:25,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:25,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:25,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:50:25,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:25,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:25,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:25,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:25,132 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:25,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:25,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:25,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:25,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:25,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:50:25,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:25,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:25,260 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:25,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:25,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:25,308 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:25,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:25,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:25,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:25,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:50:25,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:25,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:25,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:25,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:25,443 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:25,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:25,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:25,487 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:25,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:25,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:50:25,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:25,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:25,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:25,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:25,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:25,607 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:25,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:25,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:25,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:25,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:50:25,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:25,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:25,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:25,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:25,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:25,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:25,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:25,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:25,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:25,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:50:25,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:25,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:25,872 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:25,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:25,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:25,923 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:25,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:25,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:25,973 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 11:50:25,976 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:50:26,061 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:50:26,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:50:26,364 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:50:26,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:50:26,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:50:26,390 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:50:26,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:50:26,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:50:26,428 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:50:26,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:50:26,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:50:26,469 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 11:50:26,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:50:26,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:32,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:50:33,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:34,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:34,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:35,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:50:35,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:50:35,295 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:50:35,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:50:35,411 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 11:50:35,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:50:35,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:35,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:50:35,502 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 11:50:35,502 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:50:35,503 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:50:35,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:50:36,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:50:37,003 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:50:37,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:50:37,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:50:37,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:50:37,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:50:37,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:50:38,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:50:38,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:50:38,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:50:39,072 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 11:50:39,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:50:39,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:44,012 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:50:44,936 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:45,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:46,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:47,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:50:47,244 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:50:47,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:50:47,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:50:47,523 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 11:50:47,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:50:47,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:47,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:50:47,664 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 11:50:47,664 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:50:47,667 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 11:50:47,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:50:48,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 11:50:48,135 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:50:48,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 11:50:48,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 11:50:48,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 11:50:48,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 11:50:48,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 11:50:48,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 11:50:48,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 11:50:48,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:50:48,303 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 11:50:48,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:50:48,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:48,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:50:48,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:48,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:48,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:50:48,503 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 11:50:48,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:50:48,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:48,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:50:48,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:48,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:50:48,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:48,658 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:48,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:48,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:48,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:48,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:48,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:48,839 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:48,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:48,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:50:48,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:48,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:48,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:48,971 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:49,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:49,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:49,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:49,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:49,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:49,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:50:49,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:49,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:49,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:49,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:49,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:49,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:49,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:49,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:49,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:49,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:50:49,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:49,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:49,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:49,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:49,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:49,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:49,525 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:49,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:49,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:49,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:50:49,614 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:49,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:49,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:49,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:49,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:49,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:49,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:49,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:49,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:49,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:50:49,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:49,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:49,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 5 queries submitted, 0 failed, 14895 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:50:49,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:49,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:49,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:49,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:49,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:50,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:50,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:50:50,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:50,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:50,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:50,119 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:50,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:50,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:50,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:50,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:50,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:50,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:50:50,278 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:50,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:50,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:50,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:50,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:50,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:50,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:50,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:50,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:50,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:50:50,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:50,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:50,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:50,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:50,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:50,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:50,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:50,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:50,698 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 11:50:50,701 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 11:50:50,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:50:51,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 11:50:51,078 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:50:51,079 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 11:50:51,096 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 11:50:51,109 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 11:50:51,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 11:50:51,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 11:50:51,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 11:50:51,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 11:50:51,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:50:51,207 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 11:50:51,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:50:51,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:51,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:50:51,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:51,295 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:51,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:50:51,323 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 11:50:51,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:50:51,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:51,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:50:51,367 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:51,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:50:51,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:51,443 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:51,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:51,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:51,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:51,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:51,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:51,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:50:51,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:51,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:50:51,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:51,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:51,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:51,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:51,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:51,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:51,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:51,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:50:51,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:51,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:50:51,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:51,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:51,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:51,897 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:51,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:51,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:51,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:51,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:50:51,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:52,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:50:52,020 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:52,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:52,056 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:52,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:52,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:52,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:52,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:52,147 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:50:52,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:52,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:50:52,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:52,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:52,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:52,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:52,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:52,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:52,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:52,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:50:52,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:52,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:50:52,403 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:52,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:52,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:52,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:52,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:52,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:52,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:52,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:50:52,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:52,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:50:52,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:52,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:52,604 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:52,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:52,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:52,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:52,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:52,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:50:52,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:52,721 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:50:52,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:52,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:52,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:52,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:52,808 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:52,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:52,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:52,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:50:52,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:52,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:50:52,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:52,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:52,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:52,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:52,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:52,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:53,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:53,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:50:53,048 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 11:50:53,051 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:50:53,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:50:53,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:50:53,563 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:50:53,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:50:53,640 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:50:53,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:50:53,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:50:53,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:50:53,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:50:53,734 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:50:53,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:50:53,773 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 11:50:53,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:50:53,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:56,614 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:50:57,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:58,224 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:50:58,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:59,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:50:59,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:50:59,677 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:50:59,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:50:59,746 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 11:50:59,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:50:59,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:50:59,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:50:59,842 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 11:50:59,844 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:50:59,845 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:50:59,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:51:00,178 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:51:00,246 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:51:00,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:51:00,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:51:00,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:51:00,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:51:00,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:51:00,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:51:00,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:51:00,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:51:00,368 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 11:51:00,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:51:00,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:51:04,510 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:51:05,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:51:05,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:51:06,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:51:06,197 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 11:51:06,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:51:06,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:51:06,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:51:06,292 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 11:51:06,293 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:51:06,296 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:51:06,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:51:06,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:51:06,696 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:51:06,696 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:51:06,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:51:06,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:51:06,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:51:06,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:51:06,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:51:06,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:51:06,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:51:06,813 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 11:51:06,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:51:06,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:51:12,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:51:13,397 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:51:13,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:51:14,191 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:51:14,207 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 11:51:14,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:51:14,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:51:14,285 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:51:14,303 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 11:51:14,303 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:51:14,305 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:51:14,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:51:14,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:51:14,750 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:51:14,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:51:14,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:51:14,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:51:14,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:51:14,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:51:14,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:51:14,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:51:14,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:51:14,920 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 11:51:14,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:51:14,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:51:19,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:51:20,002 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:51:20,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:51:21,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:51:23,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:51:23,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:51:23,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:51:23,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:51:23,700 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 11:51:23,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:51:23,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:51:23,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:51:23,799 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 11:51:23,801 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:51:23,802 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 11:51:23,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:51:24,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 11:51:24,168 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:51:24,170 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 11:51:24,186 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 11:51:24,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 11:51:24,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 11:51:24,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 11:51:24,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 11:51:24,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 11:51:24,278 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:51:24,303 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 11:51:24,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:51:24,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:51:24,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:51:24,400 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:51:24,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:51:24,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:51:24,443 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 11:51:24,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:51:24,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:51:24,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:51:24,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:51:24,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:51:24,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:51:24,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:51:24,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:51:24,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:51:24,647 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:51:24,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:51:24,696 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:51:24,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:51:24,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:51:24,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:51:24,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:51:24,793 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:51:24,811 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:51:24,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:51:24,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:51:24,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:51:24,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:51:24,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:51:24,910 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:51:24,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:51:24,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:51:24,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:51:24,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:51:25,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:51:25,025 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:51:25,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:51:25,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:51:25,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:51:25,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:51:25,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:51:25,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:51:25,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:51:25,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:51:25,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:51:25,207 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:51:25,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:51:25,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:51:25,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:51:25,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:51:25,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:51:25,323 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:51:25,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:51:25,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:51:25,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:51:25,397 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:51:25,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:51:25,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:51:25,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:51:25,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:51:25,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:51:25,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:51:25,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:51:25,548 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:51:25,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:51:25,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:51:25,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:51:25,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:51:25,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:51:25,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:51:25,670 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:51:25,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:51:25,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:51:25,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:51:25,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:51:25,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:51:25,772 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:51:25,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:51:25,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:51:25,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:51:25,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:51:25,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:51:25,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:51:25,887 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:51:25,913 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:51:25,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:51:25,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:51:25,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:51:25,982 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:51:25,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:51:26,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:51:26,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:51:26,056 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:51:26,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:51:26,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:51:26,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:51:26,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:51:26,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:51:26,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:51:26,190 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 11:51:26,191 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:51:26,194 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:51:26,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:51:26,923 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:51:26,990 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:51:26,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:51:27,012 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:51:27,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:51:27,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:51:27,127 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:51:27,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:51:27,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:51:27,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:51:27,759 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:51:27,759 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.5680000782. 2022-02-08 11:51:27,759 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:51:27,762 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:51:27,938 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:51:28,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:51:28,364 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:51:28,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:51:28,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:51:28,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:51:28,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:51:28,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:51:28,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:51:28,561 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:51:28,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:51:28,859 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:51:28,859 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:51:30,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:51:31,104 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:51:31,104 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:51:33,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:51:33,299 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:51:33,299 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:51:35,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:51:35,480 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:51:35,480 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:51:37,482 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:51:37,753 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:51:37,753 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:51:39,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:51:39,967 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:51:39,969 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:51:41,970 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:51:42,161 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:51:42,161 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:51:44,163 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:51:44,651 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:51:44,651 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:51:46,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:51:46,867 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:51:46,868 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:51:48,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:51:49,256 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:51:49,257 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 14896 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:51:51,257 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:51:51,608 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:51:51,608 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 23.8489999771. 2022-02-08 11:51:51,609 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:51:51,611 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:51:52,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:51:53,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:51:53,490 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:51:53,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:51:53,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:51:53,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:51:53,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:51:53,596 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:51:53,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:51:53,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:51:53,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:51:54,522 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:51:54,522 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.91200017929. 2022-02-08 11:51:54,522 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:51:54,523 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:51:55,417 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:51:55,900 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:51:56,071 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:51:56,071 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:51:56,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:51:56,114 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:51:56,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:51:56,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:51:56,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:51:56,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:51:56,232 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:51:56,914 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:51:56,915 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.39299988747. 2022-02-08 11:51:56,915 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:51:56,918 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:51:58,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:51:58,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:51:58,867 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:51:58,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:51:58,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:51:58,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:51:58,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:51:58,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:51:58,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:51:59,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:51:59,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:51:59,693 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:51:59,694 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.77800011635. 2022-02-08 11:51:59,694 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:51:59,696 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:52:00,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:52:01,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:52:01,589 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:52:01,589 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:52:01,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:52:01,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:52:01,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:52:01,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:52:01,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:52:01,724 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:52:01,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:52:02,315 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:52:02,315 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.62199997902. 2022-02-08 11:52:02,315 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:52:02,318 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:52:03,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:52:03,719 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:52:03,884 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:52:03,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:52:03,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:52:03,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:52:03,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:52:03,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:52:03,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:52:03,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:52:04,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:52:04,667 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:52:04,667 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.35199999809. 2022-02-08 11:52:04,668 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:52:04,671 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:52:05,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:52:05,783 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:52:05,990 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:52:05,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:52:06,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:52:06,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:52:06,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:52:06,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:52:06,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:52:06,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:52:06,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:52:06,815 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:52:06,816 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.14800000191. 2022-02-08 11:52:06,816 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:52:06,819 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:52:07,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:52:08,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:52:08,681 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:52:08,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:52:08,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:52:08,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:52:08,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:52:08,769 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:52:08,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:52:08,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:52:08,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:52:09,690 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:52:09,690 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.87400007248. ############################################################ # # RAM FootPrint: 4911.375 # ############################################################ 2022-02-08 11:52:09,719 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 11:52:09,720 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 89 seconds. Reason: Iteration:19 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14918 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:53:38,724 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 11:53:38,724 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 11:53:43,734 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:53:44,071 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:53:44,336 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:53:44,448 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:53:44,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:53:44,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:53:44,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:53:44,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:53:44,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:53:44,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:53:44,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:53:44,619 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:53:44,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:53:44,678 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['63881 memcached\n'], []) 2022-02-08 11:53:44,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 11:53:44,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:53:44,729 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['63881 memcached\n'], []) 2022-02-08 11:53:44,729 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:53:44,730 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:53:45,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:53:45,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:53:45,569 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:53:45,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:53:45,584 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:53:45,612 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:53:45,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:53:45,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:53:45,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:53:45,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:53:45,733 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:53:45,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:53:45,796 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['119714 memcached\n'], []) 2022-02-08 11:53:45,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 11:53:45,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:53:45,842 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['119714 memcached\n'], []) 2022-02-08 11:53:45,844 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:53:45,846 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:53:45,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:53:46,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:53:46,232 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:53:46,232 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:53:46,247 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:53:46,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:53:46,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:53:46,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:53:46,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:53:46,323 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:53:46,339 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:53:46,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:53:46,390 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['102519 memcached\n'], []) 2022-02-08 11:53:46,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 11:53:46,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:53:46,443 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['102519 memcached\n'], []) 2022-02-08 11:53:46,444 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:53:46,446 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:53:46,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:53:46,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:53:46,811 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:53:46,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:53:46,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:53:46,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:53:46,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:53:46,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:53:46,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:53:46,897 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:53:46,913 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:53:46,913 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:53:46,956 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['128336 memcached\n'], []) 2022-02-08 11:53:46,956 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 11:53:46,979 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:53:47,000 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['128336 memcached\n'], []) 2022-02-08 11:53:47,002 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:53:47,006 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:53:47,117 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:53:47,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:53:47,405 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:53:47,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:53:47,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:53:47,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:53:47,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:53:47,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:53:47,474 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:53:47,487 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:53:47,500 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:53:47,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:53:47,549 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['122193 memcached\n'], []) 2022-02-08 11:53:47,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 11:53:47,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:53:47,601 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['122193 memcached\n'], []) 2022-02-08 11:53:47,601 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:53:47,602 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:53:47,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:53:47,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:53:47,966 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:53:47,966 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:53:47,982 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:53:47,997 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:53:48,012 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:53:48,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:53:48,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:53:48,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:53:48,076 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:53:48,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:53:48,155 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['26667 memcached\n'], []) 2022-02-08 11:53:48,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 11:53:48,183 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:53:48,206 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['26667 memcached\n'], []) 2022-02-08 11:53:48,206 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:53:48,209 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:53:48,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:53:48,510 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:53:48,569 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:53:48,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:53:48,584 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:53:48,596 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:53:48,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:53:48,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:53:48,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:53:48,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:53:48,663 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:53:48,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:53:48,711 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['90419 memcached\n'], []) 2022-02-08 11:53:48,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 11:53:48,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:53:48,769 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['90419 memcached\n'], []) 2022-02-08 11:53:48,770 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:53:48,772 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:53:48,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:53:49,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:53:49,326 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:53:49,326 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:53:49,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:53:49,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:53:49,367 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:53:49,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:53:49,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:53:49,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:53:49,427 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:53:49,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:53:49,493 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['105727 memcached\n'], []) 2022-02-08 11:53:49,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 11:53:49,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:53:49,539 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['105727 memcached\n'], []) 2022-02-08 11:53:49,539 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:53:49,540 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:53:49,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:53:49,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 5 queries submitted, 0 failed, 14942 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:53:49,914 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:53:49,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:53:49,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:53:49,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:53:49,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:53:49,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:53:49,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:53:50,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:53:50,028 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:53:50,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:53:50,076 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['42303 memcached\n'], []) 2022-02-08 11:53:50,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 11:53:50,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:53:50,121 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['42303 memcached\n'], []) 2022-02-08 11:53:50,121 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:53:50,124 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:53:50,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:53:50,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:53:50,457 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:53:50,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:53:50,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:53:50,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:53:50,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:53:50,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:53:50,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:53:50,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:53:50,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:53:50,575 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 11:53:50,602 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:53:50,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:53:57,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:53:58,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:53:58,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:53:59,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:53:59,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:53:59,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:53:59,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:53:59,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:53:59,904 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 11:53:59,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:53:59,918 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:53:59,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:54:00,006 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 11:54:00,006 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:54:00,009 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:54:00,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:54:00,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:54:00,372 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:54:00,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:54:00,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:54:00,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:54:00,410 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:54:00,426 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:54:00,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:54:00,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:54:00,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:54:00,479 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 11:54:00,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:54:00,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:06,301 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:54:07,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:07,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:08,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:08,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:54:08,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:54:08,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:54:08,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:54:08,515 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 11:54:08,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:54:08,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:08,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:54:08,612 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 11:54:08,612 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:54:08,615 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 11:54:08,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:54:08,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 11:54:09,005 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:54:09,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 11:54:09,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 11:54:09,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 11:54:09,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 11:54:09,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 11:54:09,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 11:54:09,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 11:54:09,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:54:09,119 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 11:54:09,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:54:09,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:09,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:54:09,196 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:09,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:09,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:54:09,236 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 11:54:09,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:54:09,252 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:09,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:54:09,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:09,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:54:09,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:09,357 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:09,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:09,408 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:09,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:09,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:09,494 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:09,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:09,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:09,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:54:09,584 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:09,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:09,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:09,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:09,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:09,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:09,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:09,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:09,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:09,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:54:09,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:09,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:09,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:09,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:09,905 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:09,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:09,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:09,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:09,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:09,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:54:10,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:10,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:10,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:10,071 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:10,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:10,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:10,124 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:10,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:10,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:10,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:54:10,196 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:10,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:10,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:10,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:10,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:10,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:10,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:10,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:10,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:10,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:54:10,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:10,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:10,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:10,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:10,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:10,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:10,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:10,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:10,507 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:10,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:54:10,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:10,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:10,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:10,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:10,596 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:10,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:10,627 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:10,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:10,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:10,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:54:10,688 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:10,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:10,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:10,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:10,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:10,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:10,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:10,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:10,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:10,826 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:54:10,844 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:10,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:10,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:10,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:10,910 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:10,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:10,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:10,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:10,980 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 11:54:10,983 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:54:11,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:54:11,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:54:11,367 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:54:11,367 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:54:11,380 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:54:11,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:54:11,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:54:11,426 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:54:11,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:54:11,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:54:11,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:54:11,480 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 11:54:11,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:54:11,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:16,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:54:17,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:17,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:17,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:54:17,926 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 11:54:17,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:54:17,938 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:18,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:54:18,017 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 11:54:18,017 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:54:18,019 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 11:54:18,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:54:18,308 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 11:54:18,362 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:54:18,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 11:54:18,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 11:54:18,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 11:54:18,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 11:54:18,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 11:54:18,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 11:54:18,454 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 11:54:18,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:54:18,492 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 11:54:18,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:54:18,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:18,543 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:54:18,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:18,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:18,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:54:18,604 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 11:54:18,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:54:18,619 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:18,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:54:18,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:18,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:54:18,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:18,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:18,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:18,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:18,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:18,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:18,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:18,849 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:18,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:18,885 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:54:18,903 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:18,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:18,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:18,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:18,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:18,974 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:18,989 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:19,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:19,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:19,038 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:54:19,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:19,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:19,088 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:19,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:19,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:19,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:19,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:19,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:19,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:19,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:54:19,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:19,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:19,244 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:19,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:19,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:19,290 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:19,306 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:19,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:19,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:19,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:54:19,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:19,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:19,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:19,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:19,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:19,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:19,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:19,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:19,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:19,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:54:19,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:19,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:19,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:19,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:19,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:19,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:19,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:19,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:19,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:19,645 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:54:19,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:19,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:19,706 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:19,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:19,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:19,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:19,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:19,785 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:19,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:19,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:54:19,829 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:19,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:19,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:19,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:19,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:19,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:19,920 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:19,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:19,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:19,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:54:19,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:19,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:20,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:20,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:20,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:20,061 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:20,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:20,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:20,108 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 11:54:20,111 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:54:20,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:54:20,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:54:20,487 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:54:20,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:54:20,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:54:20,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:54:20,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:54:20,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:54:20,556 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:54:20,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:54:20,584 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:54:20,598 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 11:54:20,624 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:54:20,637 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:25,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:54:26,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:26,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:27,428 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:27,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:54:27,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:54:28,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:54:28,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:54:28,144 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 11:54:28,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:54:28,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:28,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:54:28,226 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 11:54:28,226 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:54:28,227 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:54:28,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:54:28,540 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:54:28,608 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:54:28,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:54:28,627 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:54:28,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:54:28,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:54:28,686 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:54:28,704 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:54:28,720 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:54:28,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:54:28,759 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 11:54:28,782 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:54:28,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:33,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:54:34,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:35,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:35,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:36,012 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:54:36,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:54:36,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:54:36,290 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:54:36,312 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 11:54:36,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:54:36,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:36,413 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:54:36,434 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 11:54:36,434 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:54:36,437 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 11:54:36,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:54:36,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 11:54:36,947 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:54:36,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 11:54:36,966 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 11:54:36,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 11:54:37,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 11:54:37,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 11:54:37,045 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 11:54:37,063 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 11:54:37,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:54:37,104 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 11:54:37,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:54:37,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:37,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:54:37,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:37,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:37,224 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:54:37,243 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 11:54:37,244 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:54:37,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:37,280 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:54:37,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:37,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:54:37,346 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:37,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:37,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:37,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:37,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:37,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:37,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:37,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:37,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:37,573 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:54:37,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:37,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:37,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:37,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:37,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:37,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:37,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:37,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:37,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:37,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:54:37,802 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:37,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:37,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:37,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:37,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:37,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:37,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:37,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:37,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:37,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:54:38,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:38,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:38,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:38,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:38,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:38,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:38,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:38,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:38,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:38,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:54:38,216 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:38,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:38,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:38,278 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:38,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:38,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:38,339 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:38,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:38,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:38,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:54:38,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:38,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:38,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:38,483 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:38,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:38,523 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:38,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:38,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:38,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:38,607 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:54:38,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:38,647 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:38,668 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:38,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:38,709 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:38,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:38,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:38,766 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:38,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:38,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:54:38,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:38,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:38,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:38,894 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:38,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:38,931 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:38,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:38,971 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:38,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:39,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:54:39,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:39,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:39,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:39,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:39,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:39,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:39,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:39,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:39,200 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 11:54:39,203 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 11:54:39,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:54:39,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 11:54:39,535 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:54:39,535 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 11:54:39,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 11:54:39,559 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 11:54:39,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 11:54:39,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 11:54:39,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 11:54:39,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 11:54:39,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:54:39,635 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 11:54:39,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:54:39,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:39,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:54:39,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:39,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:39,737 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:54:39,752 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 11:54:39,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:54:39,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:39,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:54:39,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:39,818 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:54:39,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:39,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:39,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:39,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:39,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:39,950 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:39,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:39,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:54:40,017 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:40,035 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:54:40,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:40,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:40,081 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:40,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:40,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:40,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:40,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:40,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:54:40,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:40,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:54:40,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:40,221 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:40,237 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:40,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:40,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:40,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:40,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:40,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:54:40,335 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:40,352 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:54:40,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:40,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:40,405 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:40,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:40,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:40,461 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:40,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:40,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:54:40,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:40,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:54:40,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:40,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:40,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:40,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:40,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:40,645 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:40,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:40,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:54:40,696 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:40,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:54:40,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:40,743 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:40,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:40,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:40,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:40,806 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:40,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:40,836 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:54:40,854 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:40,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:54:40,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:40,914 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:40,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:40,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:40,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:40,969 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:40,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:40,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:54:41,010 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:41,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:54:41,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:41,055 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:41,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:41,084 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:41,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:41,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:41,127 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:41,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:54:41,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:41,170 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:54:41,186 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:41,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:41,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:41,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:41,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:41,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:41,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:41,298 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:54:41,312 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 11:54:41,315 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:54:41,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:54:41,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:54:41,727 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:54:41,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:54:41,744 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:54:41,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:54:41,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:54:41,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:54:41,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:54:41,835 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:54:41,852 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:54:41,871 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 11:54:41,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:54:41,915 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:43,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:54:44,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:45,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:45,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:46,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:54:46,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:54:46,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:54:46,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:54:46,619 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 11:54:46,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:54:46,634 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:46,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:54:46,711 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 11:54:46,711 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:54:46,714 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:54:46,821 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:54:47,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:54:47,124 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:54:47,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:54:47,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:54:47,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:54:47,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:54:47,183 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:54:47,196 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:54:47,209 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:54:47,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:54:47,240 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 11:54:47,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:54:47,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 14950 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:54:50,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:54:51,286 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:51,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:54:52,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:54:52,128 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 11:54:52,130 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:54:52,142 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:52,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:54:52,204 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 11:54:52,204 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:54:52,207 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:54:52,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:54:52,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:54:52,861 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:54:52,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:54:52,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:54:52,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:54:52,908 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:54:52,926 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:54:52,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:54:52,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:54:52,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:54:52,993 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 11:54:53,022 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:54:53,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:54:58,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:54:59,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:55:00,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:55:00,614 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:55:00,635 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 11:55:00,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:55:00,660 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:55:00,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:55:00,938 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 11:55:00,938 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:55:00,941 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:55:01,022 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:55:01,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:55:01,311 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:55:01,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:55:01,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:55:01,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:55:01,349 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:55:01,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:55:01,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:55:01,395 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:55:01,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:55:01,428 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 11:55:01,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:55:01,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:55:06,028 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:55:06,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:55:07,413 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:55:07,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:55:08,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:55:08,313 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:55:08,454 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:55:08,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:55:08,591 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 11:55:08,591 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:55:08,608 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:55:08,663 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:55:08,677 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 11:55:08,677 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:55:08,680 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 11:55:08,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:55:09,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 11:55:09,073 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:55:09,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 11:55:09,089 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 11:55:09,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 11:55:09,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 11:55:09,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 11:55:09,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 11:55:09,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 11:55:09,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:55:09,197 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 11:55:09,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:55:09,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:55:09,280 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:55:09,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:55:09,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:55:09,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:55:09,339 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 11:55:09,341 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:55:09,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:55:09,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:55:09,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:55:09,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:55:09,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:55:09,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:55:09,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:55:09,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:55:09,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:55:09,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:55:09,591 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:55:09,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:55:09,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:55:09,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:55:09,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:55:09,696 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:55:09,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:55:09,729 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:55:09,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:55:09,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:55:09,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:55:09,798 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:55:09,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:55:09,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:55:09,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:55:09,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:55:09,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:55:09,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:55:09,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:55:09,960 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:55:09,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:55:09,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:55:10,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:55:10,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:55:10,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:55:10,068 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:55:10,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:55:10,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:55:10,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:55:10,140 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:55:10,157 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:55:10,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:55:10,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:55:10,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:55:10,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:55:10,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:55:10,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:55:10,285 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:55:10,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:55:10,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:55:10,336 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:55:10,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:55:10,372 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:55:10,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:55:10,410 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:55:10,426 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:55:10,443 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:55:10,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:55:10,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:55:10,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:55:10,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:55:10,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:55:10,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:55:10,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:55:10,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:55:10,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:55:10,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:55:10,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:55:10,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:55:10,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:55:10,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:55:10,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:55:10,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:55:10,736 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:55:10,755 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:55:10,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:55:10,788 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:55:10,805 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:55:10,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:55:10,836 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:55:10,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:55:10,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:55:10,884 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:55:10,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:55:10,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:55:10,936 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:55:10,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:55:10,969 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:55:10,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:55:11,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:55:11,022 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:55:11,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:55:11,058 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 11:55:11,059 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:55:11,061 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:55:11,576 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:55:11,987 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:55:12,096 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:55:12,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:55:12,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:55:12,132 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:55:12,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:55:12,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:55:12,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:55:12,221 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:55:12,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:12,739 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:55:12,739 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.68099999428. 2022-02-08 11:55:12,739 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:55:12,740 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:55:12,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:55:13,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:55:13,151 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:55:13,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:55:13,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:55:13,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:55:13,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:55:13,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:55:13,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:55:13,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:55:13,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:13,454 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:13,454 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:15,456 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:15,628 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:15,630 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:17,630 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:17,786 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:17,786 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:19,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:19,950 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:19,951 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:21,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:22,098 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:22,098 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:24,098 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:24,266 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:24,266 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:26,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:26,469 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:26,469 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:28,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:28,631 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:28,631 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:30,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:30,855 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:30,855 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:32,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:33,102 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:33,104 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:35,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:35,283 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:35,283 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:37,283 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:37,460 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:37,460 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:39,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:39,681 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:39,683 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:41,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:41,874 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:41,875 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:43,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:44,052 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:44,053 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:46,053 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:46,244 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:46,244 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:48,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:48,470 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:48,470 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 5 queries submitted, 0 failed, 14950 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:55:50,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:50,746 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:50,746 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:52,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:52,982 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:52,983 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:54,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:55,204 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:55,204 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:57,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:57,710 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:55:57,710 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.121.123 2022-02-08 11:55:59,711 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:55:59,908 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:55:59,908 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 47.1699998379. 2022-02-08 11:55:59,908 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:55:59,911 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:56:01,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:56:01,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:56:02,029 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:56:02,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:56:02,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:56:02,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:56:02,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:56:02,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:56:02,160 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:56:02,198 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:56:02,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:56:02,982 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:56:02,983 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.07400012016. 2022-02-08 11:56:02,983 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:56:02,984 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:56:04,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:56:04,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:56:05,059 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:56:05,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:56:05,079 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:56:05,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:56:05,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:56:05,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:56:05,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:56:05,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:56:05,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:56:05,878 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:56:05,878 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.89499998093. 2022-02-08 11:56:05,878 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:56:05,880 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:56:06,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:56:07,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:56:07,490 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:56:07,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:56:07,510 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:56:07,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:56:07,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:56:07,584 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:56:07,599 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:56:07,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:56:07,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:56:09,088 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:56:09,088 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.21000003815. 2022-02-08 11:56:09,089 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:56:09,091 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:56:10,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:56:10,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:56:11,016 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:56:11,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:56:11,043 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:56:11,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:56:11,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:56:11,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:56:11,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:56:11,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:56:11,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:56:11,970 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:56:11,970 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.88000011444. 2022-02-08 11:56:11,970 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:56:11,971 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:56:12,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:56:13,318 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:56:13,519 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:56:13,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:56:13,533 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:56:13,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:56:13,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:56:13,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:56:13,617 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:56:13,650 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:56:13,668 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:56:14,267 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:56:14,267 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.29699993134. 2022-02-08 11:56:14,267 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:56:14,269 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:56:15,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:56:15,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:56:15,654 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:56:15,655 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:56:15,676 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:56:15,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:56:15,723 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:56:15,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:56:15,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:56:15,795 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:56:15,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:56:17,197 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:56:17,197 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.93000006676. 2022-02-08 11:56:17,197 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:56:17,200 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:56:18,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:56:18,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:56:19,102 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:56:19,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:56:19,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:56:19,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:56:19,186 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:56:19,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:56:19,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:56:19,272 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:56:19,299 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:56:20,960 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:56:20,960 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.76300001144. ############################################################ # # RAM FootPrint: 4929.9609375 # ############################################################ 2022-02-08 11:56:21,005 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 11:56:21,005 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 61 seconds. Reason: Iteration:20 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 14972 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:57:22,012 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 11:57:22,012 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 11:57:27,015 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:57:27,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:57:28,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:57:28,693 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:57:28,694 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:57:28,753 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:57:28,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:57:28,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:57:28,983 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:57:29,038 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:57:29,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:57:29,138 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:57:29,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:57:29,211 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['64602 memcached\n'], []) 2022-02-08 11:57:29,211 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 11:57:29,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 11:57:29,341 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['64602 memcached\n'], []) 2022-02-08 11:57:29,341 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:57:29,342 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:57:30,242 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:57:30,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:57:30,882 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:57:30,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:57:30,900 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:57:30,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:57:30,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:57:30,979 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:57:30,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:57:31,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:57:31,062 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:57:31,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:57:31,121 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['120586 memcached\n'], []) 2022-02-08 11:57:31,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 11:57:31,171 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 11:57:31,207 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['120586 memcached\n'], []) 2022-02-08 11:57:31,207 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:57:31,209 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:57:31,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:57:32,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:57:32,476 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:57:32,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:57:32,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:57:32,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:57:32,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:57:32,566 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:57:32,586 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:57:32,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:57:32,664 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:57:32,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:57:32,740 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['103378 memcached\n'], []) 2022-02-08 11:57:32,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 11:57:32,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 11:57:32,825 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['103378 memcached\n'], []) 2022-02-08 11:57:32,825 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:57:32,828 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:57:33,607 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:57:34,111 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:57:34,240 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:57:34,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:57:34,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:57:34,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:57:34,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:57:34,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:57:34,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:57:34,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:57:34,411 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:57:34,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:57:34,565 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['129088 memcached\n'], []) 2022-02-08 11:57:34,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 11:57:34,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 11:57:34,621 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['129088 memcached\n'], []) 2022-02-08 11:57:34,621 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:57:34,624 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:57:35,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:57:35,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:57:35,967 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:57:35,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:57:35,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:57:36,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:57:36,038 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:57:36,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:57:36,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:57:36,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:57:36,151 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:57:36,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:57:36,216 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['122942 memcached\n'], []) 2022-02-08 11:57:36,217 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 11:57:36,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 11:57:36,279 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['122942 memcached\n'], []) 2022-02-08 11:57:36,279 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:57:36,282 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:57:37,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:57:37,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:57:37,749 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:57:37,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:57:37,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:57:37,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:57:37,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:57:37,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:57:37,878 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:57:37,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:57:37,936 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:57:37,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:57:38,107 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['27416 memcached\n'], []) 2022-02-08 11:57:38,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 11:57:38,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 11:57:38,160 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['27416 memcached\n'], []) 2022-02-08 11:57:38,160 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:57:38,161 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:57:38,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:57:39,257 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:57:39,426 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:57:39,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:57:39,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:57:39,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:57:39,505 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:57:39,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:57:39,539 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:57:39,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:57:39,582 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:57:39,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:57:39,635 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['91114 memcached\n'], []) 2022-02-08 11:57:39,635 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 11:57:39,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 11:57:39,683 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['91114 memcached\n'], []) 2022-02-08 11:57:39,684 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:57:39,686 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:57:40,471 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:57:40,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:57:40,986 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:57:40,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:57:41,005 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:57:41,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:57:41,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:57:41,061 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:57:41,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:57:41,161 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:57:41,177 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:57:41,177 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:57:41,232 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['106446 memcached\n'], []) 2022-02-08 11:57:41,232 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 11:57:41,257 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 11:57:41,282 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['106446 memcached\n'], []) 2022-02-08 11:57:41,282 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:57:41,286 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:57:42,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:57:42,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:57:42,691 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:57:42,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:57:42,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:57:42,727 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:57:42,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:57:42,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:57:42,811 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:57:42,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:57:42,858 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 11:57:42,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:57:42,921 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['43067 memcached\n'], []) 2022-02-08 11:57:42,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 11:57:42,956 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 11:57:42,979 | infra | DEBUG | Thread-17565 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['43067 memcached\n'], []) 2022-02-08 11:57:42,980 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:57:42,983 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:57:43,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:57:43,290 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:57:43,352 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:57:43,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:57:43,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:57:43,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:57:43,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:57:43,414 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:57:43,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:57:43,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:57:43,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:57:43,469 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 11:57:43,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:57:43,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 14999 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:57:50,433 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:57:51,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:57:51,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:57:52,357 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:57:53,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:57:53,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:57:53,762 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:57:53,855 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:57:53,869 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 11:57:53,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:57:53,882 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:57:53,956 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:57:53,970 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 11:57:53,970 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 11:57:53,971 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:57:54,061 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:57:54,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:57:54,342 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:57:54,342 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:57:54,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:57:54,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:57:54,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:57:54,398 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:57:54,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:57:54,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:57:54,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:57:54,450 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 11:57:54,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:57:54,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:01,815 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:58:02,740 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:58:03,233 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:58:03,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:04,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:58:04,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:58:04,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:58:04,822 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:58:04,835 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 11:58:04,836 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:58:04,848 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:04,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:58:04,937 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 11:58:04,937 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:58:04,938 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 11:58:05,017 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:58:05,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 11:58:05,345 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:58:05,346 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 11:58:05,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 11:58:05,375 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 11:58:05,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 11:58:05,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 11:58:05,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 11:58:05,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 11:58:05,477 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:58:05,494 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 11:58:05,522 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:58:05,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:05,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:58:05,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:58:05,581 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:58:05,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:58:05,609 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 11:58:05,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:58:05,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:05,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:58:05,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:05,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:58:05,707 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:05,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:05,760 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:05,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:05,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:05,838 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:05,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:05,891 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:05,917 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:05,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:58:05,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:05,969 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:05,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:06,000 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:06,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:06,033 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:06,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:06,065 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:06,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:06,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:58:06,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:06,135 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:06,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:06,170 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:06,187 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:06,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:06,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:06,240 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:06,256 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:06,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:58:06,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:06,313 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:06,331 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:06,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:06,381 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:06,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:06,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:06,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:06,474 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:06,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:58:06,513 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:06,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:06,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:06,565 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:06,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:06,604 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:06,622 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:06,641 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:06,658 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:06,674 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:58:06,693 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:06,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:06,726 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:06,742 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:06,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:06,775 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:06,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:06,808 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:06,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:06,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:58:06,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:06,877 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:06,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:06,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:06,937 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:06,956 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:06,973 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:06,990 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:07,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:07,023 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:58:07,042 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:07,061 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:07,076 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:07,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:07,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:07,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:07,141 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:07,158 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:07,174 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:07,190 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:58:07,210 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:07,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:07,243 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:07,260 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:07,279 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:07,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:07,313 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:07,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:07,348 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 11:58:07,351 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:58:07,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:58:07,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:58:07,950 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:58:07,951 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:58:08,007 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:58:08,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:58:08,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:58:08,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:58:08,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:58:08,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:58:08,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:58:08,196 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 11:58:08,220 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:58:08,266 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:14,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:58:14,869 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:58:15,236 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:58:15,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:58:15,582 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 11:58:15,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:58:15,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:15,673 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:58:15,687 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 11:58:15,687 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:58:15,688 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 11:58:15,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:58:16,009 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 11:58:16,086 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:58:16,086 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 11:58:16,104 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 11:58:16,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 11:58:16,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 11:58:16,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 11:58:16,168 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 11:58:16,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 11:58:16,194 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:58:16,210 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 11:58:16,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:58:16,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:16,265 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:58:16,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:58:16,296 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:58:16,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:58:16,325 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 11:58:16,325 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:58:16,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:16,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:58:16,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:16,403 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:58:16,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:16,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:16,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:16,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:16,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:16,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:16,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:16,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:16,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:16,680 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:58:16,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:16,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:16,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:16,746 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:16,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:16,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:16,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:16,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:16,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:16,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:58:16,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:16,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:16,895 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:16,913 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:16,928 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:16,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:16,963 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:16,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:16,996 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:17,015 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:58:17,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:17,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:17,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:17,085 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:17,102 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:17,127 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:17,145 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:17,164 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:17,181 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:17,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:58:17,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:17,234 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:17,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:17,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:17,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:17,315 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:17,332 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:17,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:17,368 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:17,387 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:58:17,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:17,423 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:17,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:17,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:17,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:17,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:17,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:17,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:17,543 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:17,562 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:58:17,579 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:17,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:17,615 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:17,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:17,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:17,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:17,681 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:17,698 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:17,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:17,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:58:17,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:17,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:17,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:17,793 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:17,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:17,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:17,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:17,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:17,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:17,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:58:17,907 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:17,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:17,941 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:17,959 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:17,982 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:18,002 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:18,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:18,036 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:18,052 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 11:58:18,055 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:58:18,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:58:18,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:58:18,436 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:58:18,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:58:18,450 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:58:18,464 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:58:18,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:58:18,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:58:18,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:58:18,530 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:58:18,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:58:18,573 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 11:58:18,601 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:58:18,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:25,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:58:26,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:58:27,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:58:28,361 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:29,992 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:58:30,377 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:58:30,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:58:30,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:58:30,779 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 11:58:30,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:58:30,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:30,865 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:58:30,905 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 11:58:30,905 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:58:30,907 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:58:31,039 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:58:31,303 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:58:31,418 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:58:31,420 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:58:31,440 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:58:31,457 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:58:31,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:58:31,502 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:58:31,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:58:31,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:58:31,558 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:58:31,576 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 11:58:31,611 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:58:31,632 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:39,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:58:40,578 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:58:41,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:58:41,759 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:42,890 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:58:43,058 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:58:43,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:58:43,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:58:43,250 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 11:58:43,250 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:58:43,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:43,378 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:58:43,404 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 11:58:43,405 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:58:43,411 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 11:58:43,517 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:58:43,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 11:58:43,842 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:58:43,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 11:58:43,862 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 11:58:43,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 11:58:43,897 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 11:58:43,918 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 11:58:43,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 11:58:43,966 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 11:58:43,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:58:44,003 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 11:58:44,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:58:44,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:44,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:58:44,094 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:58:44,112 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:58:44,131 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:58:44,150 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 11:58:44,151 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:58:44,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:44,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:58:44,206 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:44,239 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:58:44,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:44,305 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:44,336 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:44,365 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:44,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:44,434 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:44,467 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:44,499 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:44,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:44,549 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:58:44,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:44,592 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:44,614 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:44,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:44,653 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:44,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:44,691 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:44,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:44,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:44,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:58:44,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:44,793 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:44,816 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:44,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:44,861 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:44,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:44,901 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:44,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:44,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:44,964 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:58:44,987 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:45,012 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:45,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:45,052 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:45,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:45,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:45,121 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:45,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:45,163 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:45,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:58:45,204 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:45,227 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:45,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:45,269 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:45,289 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:45,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:45,331 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:45,358 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:45,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:45,418 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:58:45,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:45,463 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:45,489 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:45,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:45,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:45,552 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:45,572 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:45,594 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:45,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:45,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:58:45,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:45,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:45,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:45,730 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:45,752 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:45,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:45,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:45,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:45,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:45,854 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:58:45,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:45,897 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:45,915 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:45,936 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:45,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:45,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:46,003 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:46,026 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:46,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:46,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:58:46,105 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:46,148 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:46,180 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:46,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:46,226 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:46,249 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:46,270 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:46,292 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:46,315 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 11:58:46,316 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 11:58:46,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:58:46,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 11:58:46,713 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:58:46,714 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 11:58:46,733 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 11:58:46,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 11:58:46,763 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 11:58:46,780 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 11:58:46,795 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 11:58:46,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 11:58:46,823 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:58:46,842 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 11:58:46,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:58:46,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:46,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:58:46,924 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:58:46,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:58:46,954 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:58:46,969 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 11:58:46,969 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:58:46,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:47,002 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:58:47,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:47,046 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:58:47,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:47,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:47,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:47,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:47,196 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:47,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:47,253 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:47,282 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:58:47,309 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:47,328 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:58:47,354 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:47,369 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:47,385 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:47,401 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:47,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:47,438 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:47,459 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:47,476 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:58:47,493 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:47,510 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:58:47,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:47,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:47,563 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:47,582 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:47,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:47,618 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:47,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:47,664 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:58:47,684 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:47,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:58:47,721 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:47,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:47,756 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:47,773 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:47,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:47,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:47,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:47,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:58:47,858 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:47,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:58:47,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:47,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:47,927 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:47,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:47,961 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:47,979 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:47,999 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:48,016 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:58:48,032 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:48,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:58:48,066 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:48,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:48,099 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:48,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:48,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:48,150 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:48,167 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:48,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:58:48,198 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:48,214 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:58:48,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:48,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:48,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:48,278 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:48,295 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:48,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:48,326 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:48,344 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:58:48,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:48,384 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:58:48,404 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:48,431 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:48,448 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:48,469 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:48,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:48,503 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:48,519 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:48,536 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:58:48,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:48,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:58:48,594 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:48,609 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:48,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:48,644 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:48,661 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:48,678 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:48,697 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:48,713 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:58:48,730 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 11:58:48,733 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:58:49,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 15025 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:58:50,165 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:58:50,362 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:58:50,364 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:58:50,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:58:50,407 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:58:50,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:58:50,486 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:58:50,505 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:58:50,543 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:58:50,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:58:50,598 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 11:58:50,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:58:50,651 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:58:58,308 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:58:59,260 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:58:59,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:59:00,388 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:59:01,128 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:59:01,285 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:59:01,359 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:59:01,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:59:01,447 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 11:59:01,447 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:59:01,466 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:59:01,542 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:59:01,562 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 11:59:01,563 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:59:01,565 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:59:01,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:59:02,184 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:59:02,315 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:59:02,316 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:59:02,345 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:59:02,374 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:59:02,403 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:59:02,424 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:59:02,441 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:59:02,459 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:59:02,484 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:59:02,505 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 11:59:02,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:59:02,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:59:09,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:59:09,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:59:10,436 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:59:10,881 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:59:10,903 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 11:59:10,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:59:10,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:59:10,976 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:59:10,996 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 11:59:10,996 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:59:10,999 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:59:11,394 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:59:11,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:59:11,770 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:59:11,770 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:59:11,786 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:59:11,799 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:59:11,812 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:59:11,831 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:59:11,845 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:59:11,859 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:59:11,880 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:59:11,895 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 11:59:11,923 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:59:11,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:59:17,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:59:18,701 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:59:19,144 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:59:19,545 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:59:19,581 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 11:59:19,581 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:59:19,595 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:59:19,647 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:59:19,664 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 11:59:19,664 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:59:19,667 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:59:19,801 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:59:20,263 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:59:20,528 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:59:20,529 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:59:20,553 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:59:20,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:59:20,628 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:59:20,647 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:59:20,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:59:20,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:59:20,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:59:20,874 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 11:59:20,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:59:20,947 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:59:29,828 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:59:31,351 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:59:31,944 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:59:32,645 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:59:34,569 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:59:34,946 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:59:35,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 11:59:35,312 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:59:35,490 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 11:59:35,490 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:59:35,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:59:35,627 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:59:35,655 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 11:59:35,657 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:59:35,657 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 11:59:35,750 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:59:35,980 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 11:59:36,046 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:59:36,048 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 11:59:36,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 11:59:36,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 11:59:36,091 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 11:59:36,108 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 11:59:36,122 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 11:59:36,137 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 11:59:36,153 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 11:59:36,168 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 11:59:36,197 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 11:59:36,213 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:59:36,230 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 11:59:36,244 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:59:36,259 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 11:59:36,273 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 11:59:36,286 | test | DEBUG | Thread-17565 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 11:59:36,288 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 11:59:36,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 11:59:36,319 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 11:59:36,334 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:59:36,362 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 11:59:36,391 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:59:36,421 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:59:36,453 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:59:36,479 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:59:36,506 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:59:36,535 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:59:36,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:59:36,604 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 11:59:36,631 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:59:36,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 11:59:36,665 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:59:36,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:59:36,700 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:59:36,717 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:59:36,732 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:59:36,747 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:59:36,765 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:59:36,779 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 11:59:36,796 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:59:36,813 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 11:59:36,832 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:59:36,851 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:59:36,875 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:59:36,892 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:59:36,911 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:59:36,930 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:59:36,948 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:59:36,967 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 11:59:36,993 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:59:37,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 11:59:37,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:59:37,059 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:59:37,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:59:37,101 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:59:37,118 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:59:37,138 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:59:37,155 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:59:37,176 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 11:59:37,193 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:59:37,219 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 11:59:37,246 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:59:37,265 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:59:37,285 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:59:37,303 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:59:37,321 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:59:37,338 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:59:37,355 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:59:37,374 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 11:59:37,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:59:37,411 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 11:59:37,427 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:59:37,444 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:59:37,460 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:59:37,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:59:37,496 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:59:37,515 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:59:37,532 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:59:37,551 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 11:59:37,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:59:37,588 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 11:59:37,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:59:37,621 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:59:37,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:59:37,654 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:59:37,671 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:59:37,687 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:59:37,703 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:59:37,721 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 11:59:37,739 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:59:37,757 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 11:59:37,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:59:37,792 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:59:37,808 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:59:37,825 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:59:37,841 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:59:37,857 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:59:37,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:59:37,888 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 11:59:37,904 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:59:37,921 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 11:59:37,940 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:59:37,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:59:37,977 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:59:37,994 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:59:38,012 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:59:38,029 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:59:38,049 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:59:38,069 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 11:59:38,085 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 11:59:38,086 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:59:38,089 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 11:59:38,568 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:59:38,842 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 11:59:38,918 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:59:38,918 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 11:59:38,934 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 11:59:38,957 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 11:59:39,013 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 11:59:39,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 11:59:39,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 11:59:39,075 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 11:59:39,092 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:59:39,638 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 11:59:39,638 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.55200004578. 2022-02-08 11:59:39,638 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:59:39,641 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 11:59:40,987 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:59:41,555 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 11:59:41,690 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:59:41,690 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 11:59:41,710 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 11:59:41,729 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 11:59:41,749 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 11:59:41,789 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 11:59:41,809 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 11:59:41,846 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 11:59:41,868 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:59:42,549 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 11:59:42,549 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.91100001335. 2022-02-08 11:59:42,551 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:59:42,552 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 11:59:43,470 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:59:44,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 11:59:44,436 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:59:44,437 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 11:59:44,480 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 11:59:44,516 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 11:59:44,546 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 11:59:44,575 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 11:59:44,598 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 11:59:44,638 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 11:59:44,660 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:59:45,342 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 11:59:45,342 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.79099988937. 2022-02-08 11:59:45,342 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:59:45,345 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 11:59:46,267 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:59:46,776 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 11:59:46,931 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:59:46,933 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 11:59:46,953 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 11:59:47,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 11:59:47,038 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 11:59:47,062 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 11:59:47,082 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 11:59:47,115 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 11:59:47,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:59:47,782 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 11:59:47,782 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.44099998474. 2022-02-08 11:59:47,782 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:59:47,785 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 11:59:48,648 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:59:49,107 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 11:59:49,273 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:59:49,275 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 11:59:49,293 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 11:59:49,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 11:59:49,348 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 11:59:49,371 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 11:59:49,392 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 11:59:49,430 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 11:59:49,454 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 15049 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 11:59:50,012 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 11:59:50,013 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.23000001907. 2022-02-08 11:59:50,013 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:59:50,015 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 11:59:50,571 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:59:50,871 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 11:59:50,986 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:59:50,986 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 11:59:51,006 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 11:59:51,030 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 11:59:51,051 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 11:59:51,073 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 11:59:51,095 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 11:59:51,134 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 11:59:51,154 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:59:51,755 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 11:59:51,755 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.742000103. 2022-02-08 11:59:51,756 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:59:51,757 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 11:59:52,657 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:59:53,078 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 11:59:53,187 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:59:53,188 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 11:59:53,203 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 11:59:53,223 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 11:59:53,244 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 11:59:53,262 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 11:59:53,276 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 11:59:53,302 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 11:59:53,322 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:59:53,831 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 11:59:53,832 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.0759999752. 2022-02-08 11:59:53,832 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:59:53,834 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 11:59:54,040 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:59:54,382 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 11:59:54,473 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:59:54,473 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 11:59:54,492 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 11:59:54,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 11:59:54,528 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 11:59:54,556 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 11:59:54,585 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 11:59:54,605 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 11:59:54,625 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:59:55,098 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 11:59:55,098 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.26699995995. 2022-02-08 11:59:55,098 | test | DEBUG | Thread-17565 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 11:59:55,101 | infra | DEBUG | Thread-17565 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 11:59:55,819 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 11:59:56,311 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 11:59:56,509 | infra | DEBUG | Thread-17565 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 11:59:56,509 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 11:59:56,642 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 11:59:56,683 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 11:59:56,790 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 11:59:56,898 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 11:59:56,984 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 11:59:57,125 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 11:59:57,200 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 11:59:58,500 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 11:59:58,500 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:00:00,500 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:00:01,071 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 12:00:01,071 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:00:03,072 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:00:03,525 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 12:00:03,525 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:00:05,526 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:00:06,019 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 12:00:06,019 | test | INFO | Thread-17565 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:00:08,019 | infra | DEBUG | Thread-17565 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:00:08,187 | infra | DEBUG | Thread-17565 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:00:08,187 | test | INFO | Thread-17565 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 13.0880000591. 2022-02-08 12:00:08,190 | test | DEBUG | Thread-17565 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L836 2022-02-08 12:00:08,190 | test | DEBUG | Thread-17565 | [common_lib:sleep:24] Sleep 300 seconds. Reason: None 5 queries submitted, 0 failed, 15073 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 15098 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 15120 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 15144 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 15169 passed, 0 rejected, 0 cancelled, 0 timeout ############################################################ # # Step 13: Failover a node and DeltaRecovery that node with loading in parallel # ############################################################ 2022-02-08 12:05:09,694 | test | DEBUG | MainThread | [rest_client:fail_over:1389] ns_1@172.23.121.115 - Failover successful 2022-02-08 12:05:09,703 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/Hospital.py -> SystemTestMagma():L675 2022-02-08 12:05:09,703 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 30 seconds. Reason: None 2022-02-08 12:05:39,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 48.06 % 2022-02-08 12:05:39,749 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:05:39,749 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:05:42,785 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 50.68 % 2022-02-08 12:05:42,792 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:05:42,792 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:05:45,818 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 55.08 % 2022-02-08 12:05:45,825 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:05:45,826 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:05:48,862 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 55.97 % 2022-02-08 12:05:48,872 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:05:48,874 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 15186 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:05:51,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 57.76 % 2022-02-08 12:05:51,901 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:05:51,901 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:05:54,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 60.43 % 2022-02-08 12:05:54,947 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:05:54,947 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:05:57,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 66.61 % 2022-02-08 12:05:57,970 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:05:57,970 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:06:01,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 67.50 % 2022-02-08 12:06:01,042 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:06:01,042 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:06:04,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 70.18 % 2022-02-08 12:06:04,078 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:06:04,078 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:06:07,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 73.69 % 2022-02-08 12:06:07,105 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:06:07,107 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:06:10,124 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 80.77 % 2022-02-08 12:06:10,131 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:06:10,131 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:06:13,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 84.28 % 2022-02-08 12:06:13,197 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:06:13,197 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:06:16,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 86.95 % 2022-02-08 12:06:16,220 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:06:16,220 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:06:19,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 90.41 % 2022-02-08 12:06:19,244 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:06:19,244 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:06:22,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 95.65 % 2022-02-08 12:06:22,273 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:06:22,273 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:06:25,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 98.32 % 2022-02-08 12:06:25,303 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:06:25,303 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:06:28,326 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 99.16 % 2022-02-08 12:06:28,335 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:06:28,335 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:06:31,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 100.00 % 2022-02-08 12:06:31,357 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:06:31,357 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:06:34,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 100.00 % 2022-02-08 12:06:34,426 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:06:34,426 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:06:37,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 100.00 % 2022-02-08 12:06:37,464 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:06:37,464 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:06:40,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 92.59 % 2022-02-08 12:06:40,503 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:06:40,505 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2022-02-08 12:06:43,538 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-08 12:06:43,539 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 12:06:48,558 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1590 2022-02-08 12:06:48,559 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 5 queries submitted, 0 failed, 15213 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:06:51,575 | test | INFO | MainThread | [rest_client:monitorRebalance:1599] Rebalance done. Taken 71.8710000515 seconds to complete 2022-02-08 12:06:51,586 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1602 2022-02-08 12:06:51,586 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-08 12:06:56,586 | test | DEBUG | MainThread | [rest_client:set_recovery_type:1397] ns_1@172.23.121.115 - Setting recoveryType=delta 2022-02-08 12:06:56,625 | test | DEBUG | MainThread | [rest_client:set_recovery_type:1412] ns_1@172.23.121.115 - RecoveryType set successful 2022-02-08 12:06:56,677 | infra | DEBUG | MainThread | [task:__init__:249] Last known rebalance status_id: None 2022-02-08 12:06:56,678 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[]_1644350816.62 2022-02-08 12:06:56,680 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[]_1644350816.62 2022-02-08 12:06:56,700 | infra | INFO | pool-3-thread-16 | [task:start_task:91] Thread 'Rebalance_task_IN=[]_OUT=[]_1644350816.62' started 2022-02-08 12:06:56,825 | test | DEBUG | pool-3-thread-16 | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.115,ns_1@172.23.121.132,ns_1@172.23.121.135,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.134,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123,ns_1@172.23.121.131,ns_1@172.23.121.130,ns_1@172.23.121.133', 'ejectedNodes': '', 'user': 'Administrator'} 2022-02-08 12:06:56,911 | test | DEBUG | pool-3-thread-16 | [rest_client:rebalance:1459] Rebalance operation started 2022-02-08 12:06:56,913 | test | INFO | pool-3-thread-16 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.121.115 | kv | 7.1.0-2255-enterprise | 13.288259004 | Cluster node | | 172.23.121.132 | index, n1ql | 7.1.0-2255-enterprise | 5.11941056911 | Cluster node | | 172.23.121.135 | kv | 7.1.0-2255-enterprise | 86.8461147791 | Cluster node | | 172.23.120.170 | kv | 7.1.0-2255-enterprise | 22.4543771489 | Cluster node | | 172.23.121.129 | kv | 7.1.0-2255-enterprise | 22.2016161081 | Cluster node | | 172.23.121.124 | kv | 7.1.0-2255-enterprise | 22.37503306 | Cluster node | | 172.23.121.126 | kv | 7.1.0-2255-enterprise | 22.3795659079 | Cluster node | | 172.23.121.134 | index, n1ql | 7.1.0-2255-enterprise | 7.53293260008 | Cluster node | | 172.23.121.116 | kv | 7.1.0-2255-enterprise | 87.4191502854 | Cluster node | | 172.23.121.127 | kv | 7.1.0-2255-enterprise | 21.7350993377 | Cluster node | | 172.23.121.123 | kv | 7.1.0-2255-enterprise | 22.4476266242 | Cluster node | | 172.23.121.131 | index, n1ql | 7.1.0-2255-enterprise | 5.35326434138 | Cluster node | | 172.23.121.130 | index, n1ql | 7.1.0-2255-enterprise | 7.00815494393 | Cluster node | | 172.23.121.133 | index, n1ql | 7.1.0-2255-enterprise | 4.5871559633 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2022-02-08 12:06:56,924 | infra | DEBUG | pool-3-thread-16 | [task:call:316] New rebalance status_id: a5af329d122a8f360078f8feab31ee99 2022-02-08 12:06:56,941 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:06:56,941 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:06:56,971 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:06:56,974 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:06:56,974 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:07:06,990 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:07:06,990 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:07:07,020 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:07:07,023 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:07:07,023 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:07:17,039 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:07:17,040 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:07:17,065 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:07:17,068 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:07:17,068 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:07:27,082 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:07:27,084 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:07:27,112 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:07:27,115 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:07:27,115 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:07:37,135 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:07:37,135 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:07:37,164 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:07:37,167 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:07:37,167 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:07:47,181 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:07:47,183 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:07:47,270 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:07:47,273 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:07:47,273 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 15238 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:07:57,289 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:07:57,289 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:07:57,318 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:07:57,321 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:07:57,321 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:08:07,372 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:08:07,372 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:08:07,407 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:08:07,411 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:08:07,411 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:08:17,421 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:08:17,423 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:08:17,453 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:08:17,457 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:08:17,457 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:08:27,471 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:08:27,473 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:08:27,497 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:08:27,500 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:08:27,502 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:08:37,520 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:08:37,522 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:08:37,563 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:08:37,566 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:08:37,566 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:08:47,588 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:08:47,588 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:08:47,621 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:08:47,625 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:08:47,625 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 15260 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:08:57,641 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:08:57,641 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:08:57,674 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:08:57,677 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:08:57,678 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:09:07,701 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:09:07,701 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:09:07,733 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:09:07,736 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:09:07,736 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:09:17,755 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:09:17,755 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:09:17,792 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:09:17,796 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:09:17,796 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:09:27,829 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:09:27,831 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:09:27,897 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:09:27,901 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:09:27,903 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:09:37,921 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:09:37,921 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:09:37,967 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:09:37,970 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:09:37,971 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:09:48,017 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:09:48,019 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:09:48,052 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:09:48,055 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:09:48,055 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 15279 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:09:58,071 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:09:58,072 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:09:58,115 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:09:58,119 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:09:58,121 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:10:08,141 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:10:08,142 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:10:08,180 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:10:08,186 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:10:08,186 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:10:18,200 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:10:18,200 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:10:18,232 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:10:18,234 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:10:18,234 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:10:28,256 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:10:28,256 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:10:28,293 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:10:28,298 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:10:28,298 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:10:38,315 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:10:38,315 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:10:38,349 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:10:38,352 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:10:38,354 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:10:48,377 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:10:48,377 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:10:48,414 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:10:48,417 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:10:48,417 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 15301 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:10:58,436 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:10:58,437 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:10:58,473 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:10:58,476 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:10:58,476 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:11:08,492 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:11:08,492 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:11:08,523 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:11:08,526 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:11:08,526 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:11:18,539 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:11:18,540 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:11:18,569 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:11:18,571 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:11:18,571 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:11:28,601 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:11:28,601 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:11:28,644 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:11:28,648 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:11:28,648 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:11:38,661 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:11:38,661 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:11:38,709 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:11:38,713 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:11:38,714 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:11:48,736 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:11:48,736 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:11:48,782 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:11:48,785 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:11:48,786 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 15322 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:11:58,803 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:11:58,803 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:11:58,884 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:11:58,887 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:11:58,888 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:12:08,903 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:12:08,903 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:12:08,953 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:12:08,957 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:12:08,957 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:12:18,971 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:12:18,971 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:12:19,017 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:12:19,020 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:12:19,020 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:12:29,072 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:12:29,072 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:12:29,141 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:12:29,144 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:12:29,144 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:12:39,158 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:12:39,160 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:12:39,194 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:12:39,198 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:12:39,198 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:12:49,219 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:12:49,220 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:12:49,262 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:12:49,267 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:12:49,267 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 15348 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:12:59,283 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:12:59,285 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:12:59,328 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:12:59,331 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:12:59,331 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:13:09,344 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 12:13:09,345 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 12:13:09,391 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:13:09,394 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:13:09,394 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:13:19,407 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.54 % 2022-02-08 12:13:19,407 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 0.54 2022-02-08 12:13:19,437 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:13:19,438 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:13:19,440 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:13:29,451 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.08 % 2022-02-08 12:13:29,453 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 1.08 2022-02-08 12:13:29,480 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:13:29,483 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:13:29,483 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:13:39,497 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.08 % 2022-02-08 12:13:39,497 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 1.08 2022-02-08 12:13:39,526 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:13:39,529 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:13:39,529 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:13:49,545 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.12 % 2022-02-08 12:13:49,546 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 8.12 2022-02-08 12:13:49,584 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:13:49,586 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:13:49,586 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 15371 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:13:59,601 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 11.37 % 2022-02-08 12:13:59,601 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 11.37 2022-02-08 12:13:59,644 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:13:59,647 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:13:59,647 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:14:09,660 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 16.25 % 2022-02-08 12:14:09,660 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 16.25 2022-02-08 12:14:09,693 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:14:09,694 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:14:09,694 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:14:19,709 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 27.08 % 2022-02-08 12:14:19,709 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 27.08 2022-02-08 12:14:19,743 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:14:19,746 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:14:19,746 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:14:29,778 | test | DEBUG | pool-3-thread-16 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 55.13 % 2022-02-08 12:14:29,779 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: running, progress: 55.13 2022-02-08 12:14:29,818 | infra | DEBUG | pool-3-thread-16 | [task:check:441] Wait before next rebalance progress check 2022-02-08 12:14:29,821 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 12:14:29,821 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 12:14:39,842 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1686 2022-02-08 12:14:39,842 | infra | DEBUG | pool-3-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2022-02-08 12:14:44,854 | test | INFO | pool-3-thread-16 | [task:check:414] Rebalance - status: none, progress: 100 2022-02-08 12:14:44,880 | test | INFO | pool-3-thread-16 | [task:check:473] Rebalance completed with progress: 100% in 467.967000008 sec 2022-02-08 12:14:44,880 | infra | INFO | pool-3-thread-16 | [task:complete_task:106] Thread 'Rebalance_task_IN=[]_OUT=[]_1644350816.62' completed 2022-02-08 12:14:44,927 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3970] Stats: {u'opsPerSec': 16772.5, u'diskFetches': 0, u'dataUsed': 193859314178L, u'itemCount': 186020795, u'quotaPercentUsed': 71.22383534287839, u'storageTotals': {u'hdd': {u'usedByData': 335708811238L, u'used': 444119721364L, u'quotaTotal': 1168799956992L, u'total': 1168799956992L, u'free': 593458839063L}, u'ram': {u'usedByData': 55459297768L, u'used': 109031018496L, u'quotaTotal': 78809923584L, u'total': 110647066624L, u'quotaUsed': 77866205184L, u'quotaUsedPerNode': 8651800576L, u'quotaTotalPerNode': 8756658176L}}, u'diskUsed': 335708811238L, u'vbActiveNumNonResident': 164790461, u'memUsed': 55459297768L} 2022-02-08 12:14:46,262 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+-----------+-----------+-----------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+-----------+-----------+-----------+------------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 186020795 | 72.52 GiB | 51.65 GiB | 312.65 GiB | 11.4529204269 | +-----------------+-----------+-----------------+----------+------------+-----+-----------+-----------+-----------+------------+---------------+ 2022-02-08 12:14:46,262 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+------------------+------------------+-----------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+------------------+------------------+-----------------+---------------+ | 40000000 | 380000000 | 2000000-20000000 | 2000000-20000000 | 1000000-2000000 | 0-0 | +---------------+---------------+------------------+------------------+-----------------+---------------+ 2022-02-08 12:14:46,757 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 11.4529204269 2022-02-08 12:14:46,760 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:14:47,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:14:48,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:14:48,395 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:14:48,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:14:48,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:14:48,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:14:48,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:14:48,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:14:48,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:14:48,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:14:48,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 12:14:48,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 12:14:48,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 12:14:48,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 12:14:48,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 12:14:49,599 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:14:49,601 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 5 queries submitted, 0 failed, 15395 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:14:50,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:14:51,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:14:51,269 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:14:51,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:14:51,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:14:51,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:14:51,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:14:51,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:14:51,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:14:51,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:14:51,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 12:14:51,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 12:14:51,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 12:14:51,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 12:14:51,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 12:14:52,417 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:14:52,420 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 12:14:52,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:14:52,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 12:14:52,786 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:14:52,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 12:14:52,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 12:14:52,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 12:14:52,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 12:14:52,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 12:14:52,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 12:14:52,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 12:14:52,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 12:14:52,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 12:14:52,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 12:14:52,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 12:14:52,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 12:14:53,015 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 12:14:53,016 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:14:54,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:14:54,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:14:55,075 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:14:55,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:14:55,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:14:55,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:14:55,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:14:55,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:14:55,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:14:55,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:14:55,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 12:14:55,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 12:14:55,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 12:14:55,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 12:14:55,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 12:14:56,573 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:14:56,575 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 12:14:56,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:14:56,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 12:14:56,950 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:14:56,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 12:14:56,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 12:14:56,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 12:14:56,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 12:14:57,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 12:14:57,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 12:14:57,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 12:14:57,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 12:14:57,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 12:14:57,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 12:14:57,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 12:14:57,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 12:14:57,151 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 12:14:57,154 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:14:57,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:14:58,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:14:58,667 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:14:58,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:14:58,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:14:58,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:14:58,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:14:58,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:14:58,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:14:58,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:14:58,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 12:14:59,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 12:14:59,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 12:14:59,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 12:14:59,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 12:14:59,717 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:14:59,720 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:15:00,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:15:01,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:15:01,336 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:15:01,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:15:01,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:15:01,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:15:01,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:15:01,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:15:01,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:15:01,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:15:01,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 12:15:01,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 12:15:01,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 12:15:01,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 12:15:01,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 12:15:02,535 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:15:02,538 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 12:15:02,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:15:02,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 12:15:02,927 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:15:02,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 12:15:02,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 12:15:02,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 12:15:02,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 12:15:02,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 12:15:03,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 12:15:03,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 12:15:03,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 12:15:03,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 12:15:03,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 12:15:03,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 12:15:03,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 12:15:03,142 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 12:15:03,145 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 12:15:03,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:15:03,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 12:15:03,519 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:15:03,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 12:15:03,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 12:15:03,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 12:15:03,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 12:15:03,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 12:15:03,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 12:15:03,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 12:15:03,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 12:15:03,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 12:15:03,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 12:15:03,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 12:15:03,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 12:15:03,736 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 12:15:03,737 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:15:04,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:15:05,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:15:05,285 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:15:05,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:15:05,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:15:05,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:15:05,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:15:05,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:15:05,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:15:05,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:15:05,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 12:15:05,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 12:15:05,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 12:15:05,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 12:15:05,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 12:15:06,381 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:15:06,382 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:15:07,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:15:07,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:15:07,892 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:15:07,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:15:07,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:15:07,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:15:07,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:15:07,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:15:07,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:15:07,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:15:08,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 12:15:08,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 12:15:08,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 12:15:08,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 12:15:08,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 12:15:08,767 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:15:08,770 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:15:09,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:15:10,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:15:10,364 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:15:10,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:15:10,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:15:10,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:15:10,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:15:10,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:15:10,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:15:10,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:15:10,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 12:15:10,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 12:15:11,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 12:15:11,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 12:15:11,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 12:15:12,200 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:15:12,203 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:15:13,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:15:13,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:15:13,753 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:15:13,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:15:13,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:15:13,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:15:13,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:15:13,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:15:13,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:15:13,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:15:13,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 12:15:14,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 12:15:14,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 12:15:14,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 12:15:14,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 12:15:14,913 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:15:14,914 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 12:15:15,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:15:15,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 12:15:15,302 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:15:15,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 12:15:15,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 12:15:15,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 12:15:15,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 12:15:15,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 12:15:15,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 12:15:15,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 12:15:15,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2022-02-08 12:15:15,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2022-02-08 12:15:15,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2022-02-08 12:15:15,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2022-02-08 12:15:15,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2022-02-08 12:15:15,540 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 12:15:15,540 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for kvstore is 317644MB 2022-02-08 12:15:15,542 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for wal is 3043MB 2022-02-08 12:15:15,542 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:642] Total Disk usage for keyTree is 19857MB 2022-02-08 12:15:15,542 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:643] Total Disk usage for seqTree is 297484MB 2022-02-08 12:15:15,543 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:15:15,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:15:15,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:15:15,951 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:15:15,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:15:15,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:15:15,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:15:15,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:15:16,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:15:16,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:15:16,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:15:16,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 12:15:16,203 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.116 - core(s): 8 2022-02-08 12:15:16,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 12:15:17,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 12:15:17,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 12:15:17,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 12:15:17,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 12:15:17,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 12:15:18,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 12:15:18,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 12:15:18,750 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:15:18,753 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:15:18,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:15:19,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:15:19,163 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:15:19,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:15:19,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:15:19,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:15:19,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:15:19,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:15:19,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:15:19,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:15:19,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 12:15:19,331 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.123 - core(s): 8 2022-02-08 12:15:19,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 12:15:19,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 12:15:20,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 12:15:20,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 12:15:20,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 12:15:20,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 12:15:20,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 12:15:20,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 12:15:21,033 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:15:21,039 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:15:21,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:15:21,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:15:21,467 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:15:21,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:15:21,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:15:21,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:15:21,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:15:21,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:15:21,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:15:21,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:15:21,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 12:15:21,651 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.124 - core(s): 8 2022-02-08 12:15:21,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 12:15:22,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 12:15:22,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 12:15:22,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 12:15:22,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 12:15:22,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 12:15:23,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 12:15:23,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 12:15:23,394 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:15:23,395 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:15:23,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:15:23,711 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:15:23,772 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:15:23,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:15:23,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:15:23,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:15:23,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:15:23,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:15:23,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:15:23,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:15:23,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 12:15:23,943 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.126 - core(s): 8 2022-02-08 12:15:23,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 12:15:24,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 12:15:24,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 12:15:24,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 12:15:25,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 12:15:25,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 12:15:25,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 12:15:25,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 12:15:25,691 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:15:25,694 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:15:25,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:15:26,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:15:26,137 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:15:26,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:15:26,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:15:26,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:15:26,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:15:26,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:15:26,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:15:26,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:15:26,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 12:15:26,322 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.127 - core(s): 8 2022-02-08 12:15:26,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 12:15:26,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 12:15:27,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 12:15:27,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 12:15:27,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 12:15:27,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 12:15:27,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 12:15:27,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 12:15:28,095 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:15:28,098 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:15:28,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:15:28,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:15:28,526 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:15:28,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:15:28,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:15:28,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:15:28,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:15:28,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:15:28,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:15:28,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:15:28,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 12:15:28,686 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.129 - core(s): 8 2022-02-08 12:15:28,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 12:15:29,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 12:15:29,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 12:15:29,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 12:15:29,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 12:15:29,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 12:15:30,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 12:15:30,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 12:15:30,536 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:15:30,542 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:15:30,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:15:30,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:15:31,193 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:15:31,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:15:31,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:15:31,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:15:31,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:15:31,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:15:31,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:15:31,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:15:31,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 12:15:31,710 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.135 - core(s): 8 2022-02-08 12:15:31,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 12:15:32,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 12:15:32,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 12:15:33,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 12:15:33,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 12:15:33,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 12:15:33,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 12:15:34,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 12:15:34,273 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:15:34,275 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:15:34,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:15:34,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:15:34,717 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:15:34,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:15:34,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:15:34,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:15:34,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:15:34,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:15:34,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:15:34,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:15:34,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 12:15:34,892 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.121.115 - core(s): 8 2022-02-08 12:15:34,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 12:15:35,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 12:15:35,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 12:15:35,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 12:15:35,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 12:15:35,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 12:15:36,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 12:15:36,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 12:15:36,500 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:15:36,503 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:15:36,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:15:36,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:15:36,915 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:15:36,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:15:36,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:15:36,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:15:36,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:15:36,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:15:36,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:15:37,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:15:37,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2022-02-08 12:15:37,062 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:712] machine: 172.23.120.170 - core(s): 8 2022-02-08 12:15:37,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2022-02-08 12:15:37,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2022-02-08 12:15:37,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2022-02-08 12:15:37,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2022-02-08 12:15:38,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2022-02-08 12:15:38,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2022-02-08 12:15:38,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2022-02-08 12:15:38,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2022-02-08 12:15:38,717 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:15:38,717 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:723] {'172.23.121.116': {'rw_0:magma': {u'NReadBytesGetPerSec': 1240578.25, u'ReplayWALOffset': u'2534:2535424', u'NTablesCreated': 10256, u'RecentReadIOAmpSet': 0.02941080927848816, u'ReadIOAmp': 5.251567363739014, u'BlockCacheMisses': 247679, u'NFlushes': 2562, u'RecentWriteAmp': 2.1583831310272217, u'NumSyncFlushes': 292, u'TotalMemUsed': 65873985, u'NBloomFilterHits': 4802139, u'Fragmentation': 0.38907405734062195, u'NumNormalFlushes': 1227, u'RecentBlockCacheHitRatio': 55.90657043457031, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 5199, u'ReadIOAmp': 92.78125, u'MemtablesMemUsed': 4924856, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 42296417, u'CheckpointOverhead': 4988928, u'NItemsGet': 64, u'NWriteIOs': 6711, u'NReadBytes': 164357539, u'LevelStats': [{u'TargetSize': 130023424, u'LogicalSize': 21675498, u'SizeRatio': 0.16670456240254064, u'NWriteBytes': 23007232, u'BloomFilterMemUsed': 0, u'NumTables': 245, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2412544}, {u'TargetSize': 130023424, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 130023424, u'LogicalSize': 98242511, u'SizeRatio': 0.7555754799996653, u'NWriteBytes': 11018240, u'BloomFilterMemUsed': 0, u'NumTables': 62, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 1667072}, {u'TargetSize': 2395753810L, u'LogicalSize': 1858259019, u'SizeRatio': 0.7756469012982599, u'NWriteBytes': 9244672, u'BloomFilterMemUsed': 0, u'NumTables': 846, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 28348416}, {u'TargetSize': 9330753374L, u'LogicalSize': 4791507634L, u'SizeRatio': 0.5135177666737459, u'NWriteBytes': 340856832, u'BloomFilterMemUsed': 0, u'NumTables': 1406, u'WastedSpace': 126130947, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 4791390208L}], u'NItemsGarbageCollected': 437682, u'NInternalFragmentationCompacts': 6, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 33012989, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 301440, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 124432, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 885480, u'CompactWriter': {u'NDataBlockWrites': 35050, u'NWriteBytes': 105820160, u'NIndexBlockWrites': 798, u'NBytesIncoming': 159320043, u'NSyncs': 0, u'NInserts': 1288242, u'NWriteIOs': 758}, u'NonResidentBloomFilterSize': 0, u'IteratorMemUsed': 243504, u'NWriteBytes': 389952080, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 5938, u'NReadBytes': 913181, u'NDataBlockReads': 64, u'NBytesOutgoing': 0, u'NIndexBlockReads': 128}, u'NBloomFilterMisses': 0, u'NItemsMerged': 101792, u'LogicalDiskSize': 4791507634L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.523172577794321, u'BufferMemUsed': 1476608, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 2933248615L, u'NTableFiles': 2968, u'NFileCountCompacts': 133, u'TableObjectMemUsed': 1248720, u'TreeSnapshotMemoryUsed': 389392, u'TotalDiskUsage': 6128975872L, u'BytesIncoming': 745360320, u'NItemsDel': 0, u'PendingCheckpointBytes': 507904, u'NCompacts': 267, u'NItemsPut': 1171298, u'NSyncs': 1344, u'CompactReader': {u'NReadIOs': 61524, u'NReadBytes': 146763288, u'NDataBlockReads': 58336, u'NBytesOutgoing': 0, u'NIndexBlockReads': 3188}, u'ActiveDiskUsage': 4823818240L, u'NTablesDeleted': 2827, u'IterateReader': {u'NReadIOs': 56970, u'NReadBytes': 16681070, u'NDataBlockReads': 28491, u'NBytesOutgoing': 0, u'NIndexBlockReads': 28479}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 94759, u'NWriteBytes': 284131920, u'NIndexBlockWrites': 7404, u'NBytesIncoming': 432260528, u'NSyncs': 1344, u'NInserts': 722481, u'NWriteIOs': 5953}}, u'NReadBytes': 1888920923, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 294.5, u'NInternalFragmentationCompacts': 6, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 36512215, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 41164, u'NTxns': 61638, u'NWriteBytes': 940220416, u'NSegments': 75, u'NSyncs': 61637, u'NLoggedBytes': 940220416, u'NActiveSegments': 4, u'NWrites': 61860, u'PreallocateSupported': True, u'NInserts': 1232716, u'DiskUsed': 55984128}, u'NGets': 90755, u'NReadBytesSet': 67576509, u'NReadBytesCompact': 485531247, u'IndexResidentRatio': 0.30688998103141785, u'CompactQueueSize': 0, u'NSyncsPerSec': 4.5, u'BlockCacheHitRatio': 50.86650085449219, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 1386708606, u'BloomFilterMemUsedRatio': 0.06398668140172958, u'NonResidentBloomFilterSize': 886393, u'NWriteBytes': 1770660383, u'BytesOutgoing': 57365, u'RecentReadIOAmp': 3.5640883445739746, u'WriteAmp': 2.1452367305755615, u'NBlocksCached': 14690, u'LogicalDataSize': 3040027843L, u'ActiveBloomFilterMemUsed': 6239552, u'NFileCountCompacts': 400, u'NTableFiles': 4355, u'FlushQueueSize': 8, u'TableObjectMemUsed': 1581600, u'BlockCacheQuota': 78817662, u'TotalDiskUsage': 6457483264L, u'NPurgerRuns': 0, u'ReadIOAmpSet': 0.0, u'NSyncs': 4032, u'NReadIOsPerSec': 662.0999755859375, u'NReadIOsSetPerSec': 30.200000762939453, u'NTablesDeleted': 7779, u'BloomFilterFPR': 0.016079524531960487, u'NBloomFilterHitsPerSec': 6734.2998046875, u'WriteCacheMemUsed': 6664488, u'RecentBloomFilterFPR': 0.01627298817038536, u'BytesPerRead': 14535.0908203125, u'BytesIncomingPerSec': 1061879.875, u'NReadIOsGetPerSec': 430.0666809082031, u'NumKVStores': 31, u'NGetStats': 934, u'TotalIndexBlocksSize': 46499475, u'NGetStatsPerSec': 0.8333333134651184, u'NWriteBytesCompact': 479989760, u'NReadIOsSet': 23900, u'ActiveDataSize': 3040027843L, u'NWriteBytesCompactPerSec': 541081.625, u'NBloomFilterMissesPerSec': 111.4000015258789, u'NCommitBatches': 65077, u'NGetStatsComputed': 934, u'NWriteIOs': 76944, u'NReadIOsGet': 500506, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 19704416, u'LSMTreeObjectMemUsed': 534688, u'NSetsPerSec': 1026.8333740234375, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 3.4666666984558105, u'ManagedFileObjectMemUsed': 434592, u'ReadAmpGet': 22995.41796875, u'BlockCacheMemUsed': 45648880, u'NWriterCompacts': 0, u'NReadIOs': 746155, u'WALBufferMemUsed': 81920, u'ReadAmp': 31750.09765625, u'keyStats': {u'NItemsTransformed': 7717426, u'NTablesCreated': 2580, u'ReadIOAmp': 0.569770069520334, u'MemtablesMemUsed': 779704, u'NBloomFilterHits': 4801312, u'TotalIndexBlocksSize': 4171641, u'CheckpointOverhead': 43958272, u'NItemsGet': 862913, u'NWriteIOs': 4552, u'NReadBytes': 1719804769, u'LevelStats': [{u'TargetSize': 130023424, u'LogicalSize': 5131226, u'SizeRatio': 0.03946385845061271, u'NWriteBytes': 44503040, u'BloomFilterMemUsed': 61112, u'NumTables': 237, u'WastedSpace': 69, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 4661248}, {u'TargetSize': 130023424, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 130023424, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 0}, {u'TargetSize': 130023424, u'LogicalSize': 77811924, u'SizeRatio': 0.59844543087867, u'NWriteBytes': 330678272, u'BloomFilterMemUsed': 2607488, u'NumTables': 54, u'WastedSpace': 171823, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 77971456}, {u'TargetSize': 184291355, u'LogicalSize': 184262234, u'SizeRatio': 0.9998419839064073, u'NWriteBytes': 34992128, u'BloomFilterMemUsed': 3565568, u'NumTables': 98, u'WastedSpace': 478204, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 184733696}], u'NItemsGarbageCollected': 590592, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 3483970, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 71232, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 614671, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 234536, u'NFailedCompacts': 157, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 226299, u'CompactWriter': {u'NDataBlockWrites': 130259, u'NWriteBytes': 373071872, u'NIndexBlockWrites': 1820, u'NBytesIncoming': 492723898, u'NSyncs': 0, u'NInserts': 9262770, u'NWriteIOs': 880}, u'NonResidentBloomFilterSize': 882682, u'IteratorMemUsed': 0, u'NWriteBytes': 419512269, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 491662, u'NReadBytes': 1384750933, u'NDataBlockReads': 1611092, u'NBytesOutgoing': 0, u'NIndexBlockReads': 2132291}, u'NBloomFilterMisses': 78123, u'NItemsMerged': 0, u'LogicalDiskSize': 184262234, u'TotalBloomFilterMemUsed': 6913416, u'WriteAmp': 10.590587048482138, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 6234168, u'LogicalDataSize': 106450310, u'NTableFiles': 742, u'NFileCountCompacts': 133, u'TableObjectMemUsed': 178080, u'TreeSnapshotMemoryUsed': 185248, u'TotalDiskUsage': 323223552, u'BytesIncoming': 39611805, u'NItemsDel': 0, u'PendingCheckpointBytes': 22966272, u'NCompacts': 139, u'NItemsPut': 792235, u'NSyncs': 1344, u'CompactReader': {u'NReadIOs': 123009, u'NReadBytes': 335053836, u'NDataBlockReads': 135667, u'NBytesOutgoing': 0, u'NIndexBlockReads': 4065}, u'ActiveDiskUsage': 267366400, u'NTablesDeleted': 2540, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.016010665169225537, u'Writer': {u'NDataBlockWrites': 8279, u'NWriteBytes': 46440397, u'NIndexBlockWrites': 2328, u'NBytesIncoming': 26657174, u'NSyncs': 1344, u'NInserts': 451819, u'NWriteIOs': 3672}}, u'NReadBytesCompactPerSec': 471042.9375, u'TableMetaMemUsed': 1296849, u'NDeletes': 0, u'BlockCacheHitsPerSec': 373.3999938964844, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 78478, u'LogicalDiskSize': 4976098786L, u'NSets': 792235, u'TotalBloomFilterMemUsed': 6920000, u'BufferMemUsed': 1476608, u'RecentReadAmp': 0.0, u'NGetsPerSec': 120.66666412353516, u'NInserts': 0, u'NCommitBatchesPerSec': 78.9000015258789, u'FragmentationWithOverhead': 0.5292240381240845, u'NCompactsPerSec': 0.6666666865348816, u'WALMemUsed': 147536, u'MemoryQuota': 108147512, u'NReadBytesPerSec': 1731035.75, u'TreeSnapshotMemoryUsed': 702336, u'BytesIncoming': 825391645, u'NCompacts': 540, u'NumPersistentFlushes': 1043, u'NWriteIOsPerSec': 97.36666870117188, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 10281.03515625, u'ActiveDiskUsage': 5093740544L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 0.8333333134651184, u'NReadBytesSetPerSec': 87334.4296875, u'BlockCacheHits': 256415, u'NWriteBytesPerSec': 2291943.75, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 2477, u'ReadIOAmp': 1.4287118977384463, u'MemtablesMemUsed': 959928, u'NBloomFilterHits': 827, u'TotalIndexBlocksSize': 31417, u'CheckpointOverhead': 2678784, u'NItemsGet': 2034, u'NWriteIOs': 3821, u'NReadBytes': 4758615, u'LevelStats': [{u'TargetSize': 130023424, u'LogicalSize': 2867449, u'SizeRatio': 0.0220533263298773, u'NWriteBytes': 18685952, u'BloomFilterMemUsed': 4392, u'NumTables': 281, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2301952}, {u'TargetSize': 130023424, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 130023424, u'LogicalSize': 328918, u'SizeRatio': 0.00252968265164283, u'NWriteBytes': 1081344, u'BloomFilterMemUsed': 992, u'NumTables': 31, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 253952}], u'NItemsGarbageCollected': 313685, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 15256, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 61920, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 7052, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 185070, u'CompactWriter': {u'NDataBlockWrites': 134, u'NWriteBytes': 1097728, u'NIndexBlockWrites': 134, u'NBytesIncoming': 415120, u'NSyncs': 0, u'NInserts': 3082, u'NWriteIOs': 134}, u'NonResidentBloomFilterSize': 3711, u'IteratorMemUsed': 0, u'NWriteBytes': 20975618, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 2906, u'NReadBytes': 1044492, u'NDataBlockReads': 1619, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1619}, u'NBloomFilterMisses': 355, u'NItemsMerged': 0, u'LogicalDiskSize': 328918, u'TotalBloomFilterMemUsed': 6584, u'WriteAmp': 0.2220441851416453, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 5384, u'LogicalDataSize': 328918, u'NTableFiles': 645, u'NFileCountCompacts': 134, u'TableObjectMemUsed': 154800, u'TreeSnapshotMemoryUsed': 127696, u'TotalDiskUsage': 5283840, u'BytesIncoming': 94465964, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 134, u'NItemsPut': 316227, u'NSyncs': 1344, u'CompactReader': {u'NReadIOs': 4146, u'NReadBytes': 3714123, u'NDataBlockReads': 2412, u'NBytesOutgoing': 0, u'NIndexBlockReads': 2412}, u'ActiveDiskUsage': 2555904, u'NTablesDeleted': 2412, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.3003384094754653, u'Writer': {u'NDataBlockWrites': 2343, u'NWriteBytes': 19877890, u'NIndexBlockWrites': 2343, u'NBytesIncoming': 3835174, u'NSyncs': 1344, u'NInserts': 24759, u'NWriteIOs': 3687}}}}} 2022-02-08 12:15:38,720 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:728] magma stats fragmentation result {'172.23.121.123': [0.39152637124061584, 0.42699670791625977, 0.4094174802303314, 0.3695693910121918, 0.39145946502685547, 0.42069536447525024, 0.4211365282535553, 0.4074237644672394], '172.23.121.124': [0.39925387501716614, 0.42565497756004333, 0.41825392842292786, 0.39815571904182434, 0.37718677520751953, 0.4190085828304291, 0.4194212555885315, 0.41195276379585266], '172.23.121.135': [0.42720088362693787, 0.41817453503608704, 0.381883442401886, 0.38714519143104553, 0.387759804725647, 0.43082374334335327, 0.39179450273513794, 0.40132108330726624], '172.23.121.126': [0.4076201617717743, 0.43990063667297363, 0.4092015027999878, 0.40795114636421204, 0.4295206367969513, 0.4185371398925781, 0.3972418010234833, 0.4032629728317261], '172.23.121.115': [0.20536428689956665, 0.195445716381073, 0.18512968719005585, 0.19749411940574646, 0.1856013983488083, 0.16666735708713531, 0.17521490156650543, 0.19102254509925842], '172.23.121.116': [0.38907405734062195, 0.425810843706131, 0.34276914596557617, 0.3902292847633362, 0.41146737337112427, 0.4195329248905182, 0.38473331928253174, 0.42140188813209534], '172.23.121.127': [0.40230700373649597, 0.4303233027458191, 0.38998129963874817, 0.39481690526008606, 0.4213199317455292, 0.4140775203704834, 0.42332926392555237, 0.3872431218624115], '172.23.121.129': [0.41222652792930603, 0.4197700321674347, 0.40695568919181824, 0.3834541440010071, 0.406955361366272, 0.4200989305973053, 0.4023829996585846, 0.4165036082267761], '172.23.120.170': [0.41009801626205444, 0.4181998074054718, 0.3816041648387909, 0.3966940641403198, 0.39473968744277954, 0.40718477964401245, 0.41080424189567566, 0.4151187241077423]} 2022-02-08 12:15:39,207 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.116 is 40.0 2022-02-08 12:15:39,667 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.123 is 40.0 2022-02-08 12:15:40,042 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.124 is 40.0 2022-02-08 12:15:40,417 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.126 is 40.0 2022-02-08 12:15:40,811 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.127 is 40.0 2022-02-08 12:15:41,233 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.129 is 40.0 2022-02-08 12:15:41,707 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.135 is 40.0 2022-02-08 12:15:42,099 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.121.115 is 40.0 2022-02-08 12:15:42,539 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:674] Current Fragmentation for node 172.23.120.170 is 40.0 2022-02-08 12:15:42,539 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:677] KV stats fragmentation values {'172.23.121.123': 40.0, '172.23.121.124': 40.0, '172.23.121.135': 40.0, '172.23.121.126': 40.0, '172.23.121.115': 40.0, '172.23.121.116': 40.0, '172.23.121.127': 40.0, '172.23.121.129': 40.0, '172.23.120.170': 40.0} ############################################################ # # RAM FootPrint: 4734.1171875 # ############################################################ 2022-02-08 12:15:42,598 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 12:15:42,598 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 73 seconds. Reason: Iteration:0 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 15417 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 15444 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:16:55,602 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 12:16:55,602 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 12:17:00,605 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:17:01,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:17:01,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:17:01,888 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:17:01,890 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:17:01,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:17:01,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:17:01,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:17:01,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:17:02,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:17:02,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:17:02,173 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:17:02,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:17:02,309 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['65280 memcached\n'], []) 2022-02-08 12:17:02,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 12:17:02,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:17:02,378 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['65280 memcached\n'], []) 2022-02-08 12:17:02,380 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:17:02,381 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:17:02,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:17:02,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:17:02,930 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:17:02,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:17:02,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:17:02,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:17:02,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:17:03,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:17:03,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:17:03,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:17:03,144 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:17:03,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:17:03,223 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['121583 memcached\n'], []) 2022-02-08 12:17:03,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 12:17:03,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:17:03,286 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['121583 memcached\n'], []) 2022-02-08 12:17:03,286 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:17:03,289 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:17:03,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:17:03,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:17:03,792 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:17:03,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:17:03,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:17:03,841 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:17:03,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:17:03,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:17:03,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:17:03,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:17:03,956 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:17:03,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:17:04,032 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['104224 memcached\n'], []) 2022-02-08 12:17:04,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 12:17:04,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:17:04,109 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['104224 memcached\n'], []) 2022-02-08 12:17:04,109 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:17:04,114 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:17:04,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:17:04,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:17:04,637 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:17:04,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:17:04,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:17:04,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:17:04,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:17:04,742 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:17:04,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:17:04,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:17:04,792 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:17:04,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:17:04,857 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['129789 memcached\n'], []) 2022-02-08 12:17:04,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 12:17:04,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:17:04,914 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['129789 memcached\n'], []) 2022-02-08 12:17:04,914 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:17:04,915 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:17:05,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:17:05,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:17:05,414 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:17:05,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:17:05,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:17:05,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:17:05,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:17:05,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:17:05,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:17:05,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:17:05,555 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:17:05,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:17:05,628 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['123690 memcached\n'], []) 2022-02-08 12:17:05,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 12:17:05,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:17:05,698 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['123690 memcached\n'], []) 2022-02-08 12:17:05,700 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:17:05,701 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:17:05,897 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:17:06,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:17:06,250 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:17:06,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:17:06,270 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:17:06,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:17:06,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:17:06,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:17:06,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:17:06,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:17:06,562 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:17:06,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:17:06,674 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['28163 memcached\n'], []) 2022-02-08 12:17:06,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 12:17:06,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:17:06,736 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['28163 memcached\n'], []) 2022-02-08 12:17:06,736 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:17:06,739 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:17:07,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:17:07,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:17:08,137 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:17:08,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:17:08,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:17:08,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:17:08,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:17:08,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:17:08,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:17:08,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:17:08,282 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:17:08,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:17:08,354 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['91834 memcached\n'], []) 2022-02-08 12:17:08,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 12:17:08,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:17:08,403 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['91834 memcached\n'], []) 2022-02-08 12:17:08,403 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:17:08,405 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:17:08,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:17:09,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:17:09,357 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:17:09,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:17:09,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:17:09,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:17:09,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:17:09,431 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:17:09,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:17:09,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:17:09,496 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:17:09,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:17:09,565 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['107163 memcached\n'], []) 2022-02-08 12:17:09,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 12:17:09,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:17:09,614 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['107163 memcached\n'], []) 2022-02-08 12:17:09,614 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:17:09,615 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:17:09,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:17:10,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:17:10,115 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:17:10,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:17:10,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:17:10,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:17:10,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:17:10,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:17:10,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:17:10,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:17:10,243 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:17:10,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:17:10,292 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['43813 memcached\n'], []) 2022-02-08 12:17:10,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 12:17:10,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:17:10,345 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['43813 memcached\n'], []) 2022-02-08 12:17:10,345 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:17:10,348 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:17:10,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:17:10,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:17:10,753 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:17:10,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:17:10,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:17:10,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:17:10,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:17:10,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:17:10,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:17:10,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:17:10,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:17:10,890 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 12:17:10,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:17:10,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:17:17,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:17:18,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:17:19,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:17:19,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:17:21,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:17:21,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:17:21,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:17:21,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:17:21,584 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 12:17:21,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:17:21,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:17:21,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:17:21,684 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 12:17:21,684 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:17:21,687 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:17:21,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:17:22,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:17:22,065 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:17:22,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:17:22,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:17:22,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:17:22,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:17:22,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:17:22,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:17:22,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:17:22,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:17:22,190 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 12:17:22,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:17:22,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:17:30,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:17:31,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:17:31,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:17:32,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:17:32,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:17:33,122 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:17:33,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:17:33,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:17:33,296 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 12:17:33,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:17:33,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:17:33,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:17:33,404 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 12:17:33,405 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:17:33,407 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 12:17:33,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:17:33,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 12:17:33,802 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:17:33,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 12:17:33,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 12:17:33,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 12:17:33,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 12:17:33,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 12:17:33,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 12:17:33,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 12:17:33,905 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:17:33,924 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 12:17:33,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:17:33,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:17:33,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:17:34,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:17:34,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:17:34,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:17:34,046 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 12:17:34,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:17:34,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:17:34,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:17:34,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:34,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:17:34,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:34,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:34,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:34,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:34,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:34,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:34,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:34,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:34,464 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:34,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:17:34,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:34,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:34,538 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:34,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:34,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:34,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:34,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:34,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:34,640 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:34,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:17:34,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:34,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:34,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:34,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:34,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:34,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:34,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:34,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:34,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:34,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:17:34,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:34,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:34,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:34,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:34,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:34,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:34,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:34,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:34,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:35,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:17:35,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:35,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:35,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:35,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:35,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:35,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:35,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:35,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:35,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:35,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:17:35,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:35,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:35,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:35,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:35,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:35,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:35,288 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:35,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:35,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:35,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:17:35,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:35,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:35,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:35,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:35,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:35,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:35,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:35,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:35,515 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:35,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:17:35,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:35,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:35,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:35,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:35,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:35,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:35,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:35,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:35,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:35,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:17:35,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:35,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:35,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:35,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:35,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:35,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:35,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:35,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:35,842 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 12:17:35,845 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:17:35,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:17:36,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:17:36,243 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:17:36,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:17:36,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:17:36,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:17:36,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:17:36,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:17:36,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:17:36,364 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:17:36,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:17:36,408 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 12:17:36,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:17:36,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:17:42,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:17:43,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:17:44,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:17:44,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:17:44,684 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 12:17:44,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:17:44,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:17:44,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:17:44,795 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 12:17:44,795 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:17:44,796 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 12:17:44,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:17:45,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 12:17:45,171 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:17:45,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 12:17:45,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 12:17:45,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 12:17:45,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 12:17:45,239 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 12:17:45,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 12:17:45,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 12:17:45,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:17:45,301 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 12:17:45,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:17:45,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:17:45,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:17:45,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:17:45,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:17:45,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:17:45,430 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 12:17:45,431 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:17:45,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:17:45,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:17:45,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:45,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:17:45,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:45,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:45,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:45,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:45,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:45,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:45,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:45,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:17:45,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:45,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:17:45,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:45,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:45,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:45,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:45,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:45,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:45,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:45,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:17:46,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:46,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:17:46,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:46,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:46,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:46,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:46,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:46,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:46,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:46,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:17:46,181 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:46,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:17:46,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:46,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:46,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:46,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:46,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:46,316 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:46,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:46,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:17:46,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:46,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:17:46,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:46,423 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:46,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:46,459 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:46,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:46,494 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:46,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:46,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:17:46,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:46,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:17:46,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:46,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:46,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:46,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:46,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:46,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:46,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:46,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:17:46,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:46,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:17:46,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:46,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:46,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:46,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:46,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:46,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:46,849 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:46,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:17:46,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:46,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:17:46,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:46,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:46,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:46,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:46,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:47,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:47,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:47,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:17:47,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:47,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:17:47,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:47,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:47,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:47,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:47,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:47,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:47,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:47,214 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:17:47,232 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 12:17:47,269 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:17:47,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:17:47,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:17:47,641 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:17:47,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:17:47,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:17:47,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:17:47,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:17:47,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:17:47,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:17:47,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:17:47,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:17:47,770 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 12:17:47,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:17:47,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 15470 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:17:54,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:17:55,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:17:56,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:17:56,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:17:58,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:17:58,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:17:58,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:17:58,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:17:58,894 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 12:17:58,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:17:58,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:17:59,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:17:59,023 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 12:17:59,025 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:17:59,026 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:17:59,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:17:59,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:17:59,637 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:17:59,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:17:59,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:17:59,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:17:59,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:17:59,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:17:59,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:17:59,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:17:59,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:17:59,878 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 12:17:59,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:17:59,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:18:08,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:18:09,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:18:09,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:18:10,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:18:11,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:18:11,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:18:12,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:18:12,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:18:12,260 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 12:18:12,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:18:12,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:18:12,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:18:12,463 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 12:18:12,463 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:18:12,466 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 12:18:12,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:18:12,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 12:18:12,931 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:18:12,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 12:18:12,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 12:18:12,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 12:18:12,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 12:18:13,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 12:18:13,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 12:18:13,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 12:18:13,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:18:13,117 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 12:18:13,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:18:13,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:18:13,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:18:13,209 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:18:13,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:18:13,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:18:13,269 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 12:18:13,270 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:18:13,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:18:13,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:18:13,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:13,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:18:13,443 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:13,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:13,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:13,579 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:13,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:13,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:13,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:13,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:13,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:13,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:18:13,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:13,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:13,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:13,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:13,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:13,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:13,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:13,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:14,020 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:14,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:18:14,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:14,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:14,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:14,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:14,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:14,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:14,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:14,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:14,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:14,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:18:14,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:14,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:14,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:14,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:14,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:14,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:14,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:14,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:14,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:14,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:18:14,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:14,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:14,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:14,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:14,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:14,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:14,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:14,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:14,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:14,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:18:14,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:14,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:14,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:14,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:14,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:14,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:14,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:14,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:14,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:15,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:18:15,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:15,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:15,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:15,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:15,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:15,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:15,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:15,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:15,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:15,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:18:15,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:15,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:15,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:15,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:15,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:15,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:15,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:15,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:15,492 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:15,515 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:18:15,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:15,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:15,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:15,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:15,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:15,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:15,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:15,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:15,733 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 12:18:15,736 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 12:18:15,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:18:16,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 12:18:16,124 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:18:16,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 12:18:16,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 12:18:16,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 12:18:16,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 12:18:16,186 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 12:18:16,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 12:18:16,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 12:18:16,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:18:16,246 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 12:18:16,275 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:18:16,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:18:16,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:18:16,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:18:16,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:18:16,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:18:16,362 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 12:18:16,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:18:16,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:18:16,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:18:16,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:16,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:18:16,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:16,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:16,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:16,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:16,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:16,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:16,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:16,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:18:16,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:16,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:18:16,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:16,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:16,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:16,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:16,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:16,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:16,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:16,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:18:16,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:16,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:18:17,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:17,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:17,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:17,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:17,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:17,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:17,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:17,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:18:17,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:17,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:18:17,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:17,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:17,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:17,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:17,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:17,303 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:17,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:17,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:18:17,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:17,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:18:17,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:17,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:17,423 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:17,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:17,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:17,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:17,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:17,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:18:17,520 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:17,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:18:17,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:17,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:17,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:17,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:17,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:17,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:17,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:17,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:18:17,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:17,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:18:17,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:17,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:17,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:17,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:17,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:17,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:17,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:17,841 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:18:17,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:17,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:18:17,890 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:17,905 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:17,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:17,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:17,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:17,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:17,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:18,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:18:18,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:18,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:18:18,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:18,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:18,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:18,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:18,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:18,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:18,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:18,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:18:18,207 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 12:18:18,211 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:18:18,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:18:19,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:18:19,433 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:18:19,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:18:19,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:18:19,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:18:19,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:18:19,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:18:19,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:18:19,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:18:19,627 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:18:19,663 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 12:18:19,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:18:19,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:18:27,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:18:28,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:18:28,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:18:29,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:18:30,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:18:30,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:18:30,538 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:18:30,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:18:30,667 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 12:18:30,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:18:30,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:18:30,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:18:30,786 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 12:18:30,786 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:18:30,789 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:18:30,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:18:31,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:18:31,302 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:18:31,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:18:31,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:18:31,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:18:31,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:18:31,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:18:31,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:18:31,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:18:31,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:18:31,515 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 12:18:31,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:18:31,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:18:38,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:18:39,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:18:40,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:18:40,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:18:40,569 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 12:18:40,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:18:40,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:18:40,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:18:40,648 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 12:18:40,648 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:18:40,651 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:18:41,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:18:41,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:18:41,809 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:18:41,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:18:41,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:18:41,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:18:41,869 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:18:41,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:18:41,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:18:41,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:18:41,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:18:42,009 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 12:18:42,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:18:42,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:18:47,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:18:48,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:18:49,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:18:49,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:18:49,658 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 12:18:49,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:18:49,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:18:49,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:18:49,743 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 12:18:49,743 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:18:49,746 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 5 queries submitted, 0 failed, 15494 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:18:50,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:18:50,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:18:50,996 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:18:50,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:18:51,017 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:18:51,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:18:51,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:18:51,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:18:51,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:18:51,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:18:51,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:18:51,194 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 12:18:51,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:18:51,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:18:59,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:19:00,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:19:00,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:19:01,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:19:03,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:19:03,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:19:03,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:19:03,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:19:03,934 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 12:19:03,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:19:03,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:19:04,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:19:04,147 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 12:19:04,147 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:19:04,151 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 12:19:04,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:19:04,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 12:19:04,566 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:19:04,566 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 12:19:04,584 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 12:19:04,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 12:19:04,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 12:19:04,640 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 12:19:04,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 12:19:04,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 12:19:04,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:19:04,713 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 12:19:04,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:19:04,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:19:04,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:19:04,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:19:04,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:19:04,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:19:04,828 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 12:19:04,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:19:04,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:19:04,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:19:04,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:19:04,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:19:04,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:19:04,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:19:05,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:19:05,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:19:05,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:19:05,124 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:19:05,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:19:05,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:19:05,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:19:05,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:19:05,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:19:05,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:19:05,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:19:05,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:19:05,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:19:05,364 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:19:05,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:19:05,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:19:05,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:19:05,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:19:05,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:19:05,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:19:05,499 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:19:05,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:19:05,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:19:05,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:19:05,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:19:05,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:19:05,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:19:05,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:19:05,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:19:05,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:19:05,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:19:05,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:19:05,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:19:05,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:19:05,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:19:05,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:19:05,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:19:05,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:19:05,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:19:05,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:19:05,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:19:05,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:19:05,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:19:05,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:19:05,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:19:05,964 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:19:05,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:19:05,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:19:06,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:19:06,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:19:06,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:19:06,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:19:06,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:19:06,102 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:19:06,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:19:06,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:19:06,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:19:06,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:19:06,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:19:06,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:19:06,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:19:06,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:19:06,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:19:06,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:19:06,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:19:06,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:19:06,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:19:06,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:19:06,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:19:06,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:19:06,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:19:06,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:19:06,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:19:06,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:19:06,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:19:06,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:19:06,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:19:06,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:19:06,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:19:06,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:19:06,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:19:06,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:19:06,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:19:06,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:19:06,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:19:06,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:19:06,747 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 12:19:06,747 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:19:06,750 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:19:07,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:19:07,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:19:07,858 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:19:07,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:19:07,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:19:07,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:19:07,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:19:07,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:19:07,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:19:07,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:19:07,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:19:08,469 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:19:08,469 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.72100019455. 2022-02-08 12:19:08,470 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:19:08,471 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:19:08,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:19:09,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:19:09,443 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:19:09,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:19:09,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:19:09,489 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:19:09,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:19:09,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:19:09,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:19:09,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:19:09,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:19:10,220 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:19:10,220 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.75099992752. 2022-02-08 12:19:10,220 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:19:10,223 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:19:11,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:19:11,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:19:11,861 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:19:11,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:19:11,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:19:11,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:19:11,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:19:11,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:19:11,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:19:12,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:19:12,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:19:12,683 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:19:12,683 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.46200013161. 2022-02-08 12:19:12,683 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:19:12,684 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:19:13,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:19:14,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:19:14,299 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:19:14,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:19:14,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:19:14,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:19:14,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:19:14,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:19:14,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:19:14,466 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:19:14,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:19:15,217 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:19:15,217 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.53499984741. 2022-02-08 12:19:15,217 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:19:15,220 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:19:16,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:19:16,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:19:16,805 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:19:16,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:19:16,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:19:16,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:19:16,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:19:16,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:19:16,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:19:17,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:19:17,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:19:17,747 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:19:17,747 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.52999997139. 2022-02-08 12:19:17,749 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:19:17,750 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:19:18,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:19:18,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:19:18,746 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:19:18,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:19:18,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:19:18,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:19:18,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:19:18,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:19:18,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:19:18,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:19:18,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:19:19,756 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:19:19,756 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.00699996948. 2022-02-08 12:19:19,756 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:19:19,759 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:19:20,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:19:21,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:19:21,588 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:19:21,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:19:21,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:19:21,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:19:21,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:19:21,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:19:21,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:19:21,760 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:19:21,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:19:22,427 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:19:22,427 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.67000007629. 2022-02-08 12:19:22,427 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:19:22,428 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:19:23,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:19:23,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:19:24,046 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:19:24,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:19:24,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:19:24,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:19:24,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:19:24,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:19:24,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:19:24,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:19:24,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:19:29,548 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:19:29,549 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.1210000515. 2022-02-08 12:19:29,549 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:19:29,551 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:19:30,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:19:31,150 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:19:31,368 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:19:31,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:19:31,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:19:31,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:19:31,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:19:31,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:19:31,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:19:31,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:19:31,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:19:37,171 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:19:37,171 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.62299990654. ############################################################ # # RAM FootPrint: 4351.484375 # ############################################################ 2022-02-08 12:19:37,210 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 12:19:37,210 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 76 seconds. Reason: Iteration:1 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 15518 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 15544 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:20:53,216 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 12:20:53,217 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 12:20:58,220 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:20:58,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:20:59,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:20:59,648 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:20:59,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:20:59,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:20:59,742 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:20:59,757 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:20:59,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:20:59,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:20:59,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:20:59,862 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:20:59,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:20:59,928 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['67711 memcached\n'], []) 2022-02-08 12:20:59,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 12:21:00,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:21:00,059 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['67711 memcached\n'], []) 2022-02-08 12:21:00,061 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:21:00,062 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:21:01,124 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:21:01,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:21:01,801 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:21:01,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:21:01,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:21:01,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:21:01,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:21:01,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:21:01,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:21:01,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:21:02,000 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:21:02,002 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:21:02,062 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['124020 memcached\n'], []) 2022-02-08 12:21:02,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 12:21:02,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:21:02,141 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['124020 memcached\n'], []) 2022-02-08 12:21:02,141 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:21:02,144 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:21:02,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:21:03,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:21:03,404 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:21:03,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:21:03,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:21:03,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:21:03,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:21:03,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:21:03,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:21:03,627 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:21:03,654 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:21:03,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:21:03,736 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['106837 memcached\n'], []) 2022-02-08 12:21:03,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 12:21:03,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:21:03,806 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['106837 memcached\n'], []) 2022-02-08 12:21:03,808 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:21:03,809 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:21:04,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:21:05,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:21:05,608 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:21:05,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:21:05,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:21:05,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:21:05,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:21:05,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:21:05,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:21:05,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:21:05,805 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:21:05,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:21:05,881 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['1553 memcached\n'], []) 2022-02-08 12:21:05,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 12:21:05,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:21:05,946 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['1553 memcached\n'], []) 2022-02-08 12:21:05,947 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:21:05,948 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:21:06,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:21:07,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:21:07,645 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:21:07,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:21:07,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:21:07,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:21:07,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:21:07,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:21:07,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:21:07,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:21:07,809 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:21:07,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:21:07,881 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['126123 memcached\n'], []) 2022-02-08 12:21:07,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 12:21:07,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:21:07,936 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['126123 memcached\n'], []) 2022-02-08 12:21:07,936 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:21:07,937 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:21:08,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:21:09,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:21:09,328 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:21:09,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:21:09,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:21:09,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:21:09,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:21:09,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:21:09,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:21:09,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:21:09,517 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:21:09,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:21:09,595 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['30588 memcached\n'], []) 2022-02-08 12:21:09,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 12:21:09,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:21:09,641 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['30588 memcached\n'], []) 2022-02-08 12:21:09,641 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:21:09,644 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:21:10,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:21:10,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:21:11,075 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:21:11,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:21:11,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:21:11,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:21:11,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:21:11,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:21:11,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:21:11,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:21:11,201 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:21:11,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:21:11,247 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['94388 memcached\n'], []) 2022-02-08 12:21:11,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 12:21:11,270 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:21:11,292 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['94388 memcached\n'], []) 2022-02-08 12:21:11,293 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:21:11,296 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:21:12,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:21:12,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:21:12,500 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:21:12,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:21:12,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:21:12,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:21:12,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:21:12,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:21:12,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:21:12,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:21:12,638 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:21:12,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:21:12,707 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['109605 memcached\n'], []) 2022-02-08 12:21:12,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 12:21:12,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:21:12,756 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['109605 memcached\n'], []) 2022-02-08 12:21:12,757 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:21:12,759 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:21:13,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:21:13,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:21:14,094 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:21:14,094 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:21:14,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:21:14,124 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:21:14,150 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:21:14,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:21:14,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:21:14,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:21:14,230 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:21:14,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:21:14,286 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['46295 memcached\n'], []) 2022-02-08 12:21:14,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 12:21:14,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:21:14,334 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['46295 memcached\n'], []) 2022-02-08 12:21:14,334 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:21:14,335 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:21:14,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:21:14,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:21:14,734 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:21:14,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:21:14,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:21:14,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:21:14,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:21:14,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:21:14,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:21:14,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:21:14,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:21:14,871 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 12:21:14,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:21:14,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:21:21,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:21:22,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:21:23,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:21:23,905 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:21:25,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:21:25,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:21:25,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:21:25,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:21:25,688 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 12:21:25,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:21:25,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:21:25,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:21:25,821 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 12:21:25,822 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:21:25,823 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:21:25,918 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:21:26,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:21:26,201 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:21:26,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:21:26,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:21:26,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:21:26,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:21:26,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:21:26,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:21:26,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:21:26,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:21:26,316 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 12:21:26,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:21:26,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:21:33,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:21:34,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:21:35,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:21:35,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:21:36,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:21:37,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:21:37,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:21:37,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:21:37,321 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 12:21:37,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:21:37,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:21:37,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:21:37,447 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 12:21:37,447 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:21:37,453 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 12:21:37,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:21:37,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 12:21:37,871 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:21:37,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 12:21:37,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 12:21:37,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 12:21:37,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 12:21:37,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 12:21:37,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 12:21:37,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 12:21:37,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:21:37,996 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 12:21:38,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:21:38,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:21:38,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:21:38,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:21:38,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:21:38,102 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:21:38,119 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 12:21:38,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:21:38,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:21:38,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:21:38,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:38,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:21:38,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:38,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:38,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:38,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:38,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:38,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:38,466 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:38,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:38,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:38,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:21:38,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:38,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:38,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:38,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:38,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:38,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:38,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:38,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:38,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:38,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:21:38,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:38,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:38,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:38,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:38,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:38,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:38,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:38,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:38,918 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:38,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:21:38,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:38,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:38,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:39,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:39,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:39,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:39,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:39,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:39,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:39,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:21:39,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:39,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:39,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:39,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:39,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:39,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:39,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:39,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:39,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:39,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:21:39,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:39,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:39,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:39,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:39,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:39,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:39,431 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:39,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:39,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:39,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:21:39,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:39,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:39,548 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:39,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:39,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:39,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:39,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:39,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:39,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:39,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:21:39,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:39,701 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:39,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:39,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:39,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:39,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:39,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:39,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:39,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:39,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:21:39,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:39,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:39,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:39,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:39,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:39,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:39,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:39,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:40,007 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 12:21:40,012 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:21:40,280 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:21:40,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:21:41,200 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:21:41,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:21:41,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:21:41,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:21:41,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:21:41,288 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:21:41,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:21:41,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:21:41,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:21:41,437 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 12:21:41,464 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:21:41,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:21:47,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:21:48,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:21:49,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:21:49,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 5 queries submitted, 0 failed, 15569 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:21:49,931 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 12:21:49,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:21:50,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:21:50,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:21:50,190 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 12:21:50,190 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:21:50,191 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 12:21:50,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:21:50,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 12:21:50,605 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:21:50,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 12:21:50,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 12:21:50,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 12:21:50,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 12:21:50,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 12:21:50,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 12:21:50,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 12:21:50,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:21:50,773 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 12:21:50,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:21:50,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:21:50,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:21:50,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:21:50,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:21:50,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:21:50,910 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 12:21:50,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:21:50,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:21:50,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:21:50,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:50,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:21:51,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:51,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:51,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:51,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:51,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:51,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:51,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:51,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:21:51,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:51,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:21:51,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:51,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:51,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:51,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:51,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:51,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:51,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:51,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:21:51,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:51,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:21:51,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:51,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:51,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:51,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:51,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:51,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:51,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:51,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:21:51,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:51,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:21:51,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:51,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:51,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:51,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:51,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:51,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:51,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:51,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:21:51,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:51,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:21:51,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:51,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:51,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:52,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:52,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:52,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:52,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:52,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:21:52,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:52,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:21:52,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:52,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:52,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:52,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:52,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:52,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:52,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:52,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:21:52,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:52,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:21:52,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:52,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:52,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:52,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:52,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:52,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:52,392 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:52,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:21:52,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:52,443 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:21:52,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:52,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:52,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:52,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:52,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:52,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:52,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:52,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:21:52,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:52,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:21:52,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:52,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:52,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:52,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:52,709 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:52,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:52,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:52,760 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:21:52,779 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 12:21:52,780 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:21:52,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:21:53,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:21:53,164 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:21:53,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:21:53,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:21:53,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:21:53,214 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:21:53,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:21:53,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:21:53,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:21:53,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:21:53,302 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 12:21:53,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:21:53,346 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:21:59,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:22:00,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:22:00,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:22:01,418 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:22:02,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:22:03,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:22:03,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:22:03,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:22:03,471 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 12:22:03,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:22:03,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:22:03,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:22:03,572 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 12:22:03,572 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:22:03,575 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:22:03,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:22:03,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:22:04,046 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:22:04,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:22:04,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:22:04,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:22:04,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:22:04,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:22:04,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:22:04,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:22:04,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:22:04,220 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 12:22:04,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:22:04,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:22:12,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:22:13,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:22:13,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:22:14,494 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:22:16,186 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:22:16,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:22:16,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:22:16,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:22:16,667 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 12:22:16,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:22:16,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:22:16,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:22:16,891 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 12:22:16,891 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:22:16,894 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 12:22:17,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:22:17,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 12:22:17,345 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:22:17,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 12:22:17,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 12:22:17,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 12:22:17,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 12:22:17,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 12:22:17,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 12:22:17,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 12:22:17,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:22:17,500 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 12:22:17,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:22:17,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:22:17,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:22:17,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:22:17,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:22:17,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:22:17,651 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 12:22:17,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:22:17,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:22:17,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:22:17,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:17,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:22:17,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:17,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:17,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:17,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:17,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:18,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:18,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:18,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:18,122 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:18,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:22:18,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:18,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:18,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:18,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:18,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:18,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:18,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:18,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:18,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:18,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:22:18,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:18,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:18,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:18,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:18,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:18,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:18,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:18,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:18,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:18,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:22:18,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:18,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:18,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:18,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:18,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:18,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:18,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:18,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:18,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:18,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:22:18,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:18,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:18,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:18,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:18,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:18,915 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:18,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:18,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:18,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:19,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:22:19,028 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:19,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:19,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:19,086 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:19,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:19,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:19,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:19,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:19,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:19,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:22:19,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:19,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:19,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:19,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:19,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:19,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:19,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:19,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:19,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:19,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:22:19,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:19,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:19,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:19,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:19,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:19,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:19,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:19,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:19,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:19,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:22:19,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:19,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:19,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:19,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:19,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:19,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:19,788 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:19,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:19,831 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 12:22:19,834 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 12:22:19,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:22:20,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 12:22:20,216 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:22:20,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 12:22:20,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 12:22:20,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 12:22:20,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 12:22:20,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 12:22:20,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 12:22:20,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 12:22:20,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:22:20,335 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 12:22:20,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:22:20,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:22:20,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:22:20,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:22:20,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:22:20,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:22:20,454 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 12:22:20,456 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:22:20,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:22:20,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:22:20,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:20,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:22:20,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:20,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:20,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:20,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:20,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:20,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:20,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:20,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:22:20,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:20,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:22:20,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:20,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:20,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:20,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:20,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:20,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:20,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:20,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:22:20,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:20,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:22:20,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:20,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:20,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:20,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:20,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:20,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:20,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:21,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:22:21,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:21,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:22:21,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:21,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:21,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:21,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:21,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:21,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:21,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:21,183 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:22:21,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:21,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:22:21,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:21,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:21,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:21,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:21,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:21,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:21,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:21,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:22:21,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:21,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:22:21,418 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:21,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:21,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:21,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:21,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:21,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:21,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:21,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:22:21,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:21,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:22:21,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:21,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:21,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:21,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:21,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:21,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:21,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:21,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:22:21,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:21,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:22:21,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:21,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:21,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:21,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:21,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:21,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:21,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:21,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:22:21,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:21,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:22:21,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:21,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:21,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:21,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:21,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:21,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:22,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:22,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:22:22,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:22:22,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:22:22,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:22:22,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:22:22,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:22:22,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:22:22,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:22:22,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:22:22,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:22:22,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:22:22,210 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 12:22:22,211 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:22:22,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:22:23,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:22:23,496 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:22:23,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:22:23,515 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:22:23,640 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:22:23,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:22:23,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:22:23,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:22:23,831 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:22:23,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:22:23,888 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 12:22:23,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:22:23,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:22:31,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:22:32,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:22:33,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:22:33,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:22:34,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:22:34,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:22:34,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:22:35,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:22:35,092 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 12:22:35,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:22:35,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:22:35,214 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:22:35,233 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 12:22:35,233 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:22:35,236 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:22:35,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:22:35,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:22:35,690 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:22:35,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:22:35,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:22:35,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:22:35,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:22:35,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:22:35,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:22:35,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:22:35,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:22:35,858 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 12:22:35,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:22:35,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:22:42,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:22:43,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:22:43,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:22:44,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:22:44,242 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 12:22:44,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:22:44,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:22:44,492 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:22:44,538 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 12:22:44,539 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:22:44,540 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:22:45,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:22:45,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:22:45,592 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:22:45,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:22:45,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:22:45,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:22:45,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:22:45,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:22:45,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:22:45,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:22:45,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:22:45,740 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 12:22:45,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:22:45,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 15596 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:22:51,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:22:52,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:22:53,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:22:53,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:22:53,448 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 12:22:53,448 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:22:53,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:22:53,515 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:22:53,530 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 12:22:53,532 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:22:53,533 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:22:54,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:22:54,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:22:55,331 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:22:55,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:22:55,364 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:22:55,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:22:55,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:22:55,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:22:55,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:22:55,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:22:55,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:22:55,615 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 12:22:55,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:22:55,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:23:04,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:23:05,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:23:05,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:23:06,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:23:08,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:23:08,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:23:09,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:23:09,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:23:09,354 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 12:23:09,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:23:09,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:23:09,443 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:23:09,464 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 12:23:09,466 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:23:09,467 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 12:23:09,548 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:23:09,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 12:23:09,839 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:23:09,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 12:23:09,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 12:23:09,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 12:23:09,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 12:23:09,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 12:23:09,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 12:23:09,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 12:23:09,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:23:09,964 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 12:23:09,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:23:10,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:23:10,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:23:10,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:23:10,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:23:10,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:23:10,088 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 12:23:10,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:23:10,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:23:10,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:23:10,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:23:10,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:23:10,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:23:10,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:23:10,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:23:10,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:23:10,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:23:10,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:23:10,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:23:10,477 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:23:10,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:23:10,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:23:10,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:23:10,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:23:10,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:23:10,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:23:10,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:23:10,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:23:10,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:23:10,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:23:10,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:23:10,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:23:10,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:23:10,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:23:10,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:23:10,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:23:10,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:23:10,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:23:10,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:23:10,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:23:10,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:23:10,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:23:10,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:23:10,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:23:10,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:23:10,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:23:10,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:23:11,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:23:11,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:23:11,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:23:11,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:23:11,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:23:11,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:23:11,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:23:11,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:23:11,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:23:11,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:23:11,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:23:11,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:23:11,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:23:11,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:23:11,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:23:11,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:23:11,293 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:23:11,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:23:11,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:23:11,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:23:11,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:23:11,377 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:23:11,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:23:11,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:23:11,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:23:11,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:23:11,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:23:11,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:23:11,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:23:11,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:23:11,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:23:11,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:23:11,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:23:11,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:23:11,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:23:11,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:23:11,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:23:11,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:23:11,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:23:11,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:23:11,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:23:11,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:23:11,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:23:11,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:23:11,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:23:11,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:23:11,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:23:11,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:23:11,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:23:11,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:23:11,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:23:11,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:23:11,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:23:11,931 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 12:23:11,933 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:23:11,934 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:23:12,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:23:12,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:23:13,138 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:23:13,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:23:13,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:23:13,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:23:13,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:23:13,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:23:13,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:23:13,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:23:13,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:23:13,778 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:23:13,779 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.84599995613. 2022-02-08 12:23:13,779 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:23:13,780 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:23:14,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:23:14,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:23:14,598 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:23:14,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:23:14,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:23:14,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:23:14,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:23:14,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:23:14,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:23:14,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:23:14,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:23:15,395 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:23:15,397 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.61800003052. 2022-02-08 12:23:15,397 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:23:15,398 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:23:16,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:23:16,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:23:17,094 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:23:17,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:23:17,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:23:17,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:23:17,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:23:17,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:23:17,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:23:17,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:23:17,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:23:17,894 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:23:17,895 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.49899983406. 2022-02-08 12:23:17,895 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:23:17,898 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:23:18,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:23:19,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:23:19,512 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:23:19,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:23:19,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:23:19,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:23:19,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:23:19,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:23:19,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:23:19,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:23:19,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:23:20,385 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:23:20,385 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.49000000954. 2022-02-08 12:23:20,387 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:23:20,388 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:23:21,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:23:21,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:23:21,888 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:23:21,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:23:21,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:23:21,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:23:21,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:23:21,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:23:21,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:23:22,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:23:22,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:23:22,719 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:23:22,719 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.33300018311. 2022-02-08 12:23:22,720 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:23:22,720 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:23:23,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:23:23,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:23:23,875 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:23:23,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:23:23,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:23:23,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:23:23,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:23:23,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:23:23,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:23:24,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:23:24,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:23:24,615 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:23:24,617 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.89800000191. 2022-02-08 12:23:24,617 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:23:24,618 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:23:25,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:23:25,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:23:26,131 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:23:26,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:23:26,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:23:26,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:23:26,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:23:26,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:23:26,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:23:26,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:23:26,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:23:26,769 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:23:26,769 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.15299987793. 2022-02-08 12:23:26,770 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:23:26,773 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:23:27,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:23:27,698 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:23:27,811 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:23:27,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:23:27,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:23:27,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:23:27,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:23:27,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:23:27,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:23:27,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:23:27,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:23:28,892 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:23:28,892 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.12199997902. 2022-02-08 12:23:28,892 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:23:28,894 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:23:29,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:23:30,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:23:30,622 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:23:30,624 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:23:30,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:23:30,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:23:30,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:23:31,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:23:31,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:23:31,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:23:31,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:23:32,825 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 12:23:32,825 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:23:34,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:23:35,069 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 12:23:35,071 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:23:37,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:23:37,328 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 12:23:37,328 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:23:39,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:23:39,500 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:23:39,502 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 10.609000206. ############################################################ # # RAM FootPrint: 3776.45703125 # ############################################################ 2022-02-08 12:23:39,532 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 12:23:39,532 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 81 seconds. Reason: Iteration:2 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 15618 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 15644 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:25:00,536 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 12:25:00,538 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 12:25:05,539 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:25:06,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:25:06,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:25:06,941 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:25:06,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:25:06,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:25:07,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:25:07,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:25:07,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:25:07,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:25:07,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:25:07,224 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:25:07,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:25:07,305 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['68436 memcached\n'], []) 2022-02-08 12:25:07,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 12:25:07,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:25:07,404 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['68436 memcached\n'], []) 2022-02-08 12:25:07,405 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:25:07,407 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:25:08,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:25:09,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:25:09,216 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:25:09,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:25:09,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:25:09,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:25:09,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:25:09,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:25:09,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:25:09,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:25:09,513 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:25:09,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:25:09,582 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['124776 memcached\n'], []) 2022-02-08 12:25:09,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 12:25:09,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:25:09,638 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['124776 memcached\n'], []) 2022-02-08 12:25:09,640 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:25:09,641 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:25:10,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:25:11,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:25:11,466 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:25:11,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:25:11,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:25:11,515 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:25:11,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:25:11,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:25:11,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:25:11,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:25:11,677 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:25:11,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:25:11,753 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['107699 memcached\n'], []) 2022-02-08 12:25:11,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 12:25:11,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:25:11,825 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['107699 memcached\n'], []) 2022-02-08 12:25:11,826 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:25:11,841 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:25:12,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:25:13,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:25:13,842 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:25:13,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:25:13,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:25:13,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:25:13,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:25:13,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:25:13,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:25:13,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:25:14,006 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:25:14,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:25:14,075 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['2320 memcached\n'], []) 2022-02-08 12:25:14,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 12:25:14,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:25:14,148 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['2320 memcached\n'], []) 2022-02-08 12:25:14,150 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:25:14,151 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:25:14,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:25:15,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:25:15,657 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:25:15,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:25:15,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:25:15,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:25:15,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:25:15,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:25:15,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:25:15,823 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:25:15,842 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:25:15,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:25:15,917 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['126879 memcached\n'], []) 2022-02-08 12:25:15,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 12:25:15,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:25:15,979 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['126879 memcached\n'], []) 2022-02-08 12:25:15,980 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:25:15,984 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:25:17,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:25:17,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:25:17,861 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:25:17,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:25:17,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:25:17,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:25:17,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:25:17,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:25:17,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:25:17,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:25:17,994 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:25:17,994 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:25:18,056 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['31343 memcached\n'], []) 2022-02-08 12:25:18,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 12:25:18,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:25:18,144 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['31343 memcached\n'], []) 2022-02-08 12:25:18,144 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:25:18,148 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:25:18,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:25:19,566 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:25:19,733 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:25:19,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:25:19,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:25:19,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:25:19,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:25:19,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:25:19,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:25:19,849 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:25:19,867 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:25:19,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:25:19,904 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['95123 memcached\n'], []) 2022-02-08 12:25:19,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 12:25:19,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:25:19,956 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['95123 memcached\n'], []) 2022-02-08 12:25:19,957 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:25:19,960 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:25:20,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:25:20,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:25:20,913 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:25:20,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:25:20,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:25:20,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:25:20,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:25:20,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:25:21,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:25:21,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:25:21,058 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:25:21,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:25:21,118 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['110335 memcached\n'], []) 2022-02-08 12:25:21,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 12:25:21,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:25:21,167 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['110335 memcached\n'], []) 2022-02-08 12:25:21,167 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:25:21,170 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:25:21,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:25:21,698 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:25:21,803 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:25:21,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:25:21,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:25:21,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:25:21,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:25:21,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:25:21,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:25:21,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:25:21,941 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:25:21,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:25:22,164 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['47058 memcached\n'], []) 2022-02-08 12:25:22,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 12:25:22,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:25:22,301 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['47058 memcached\n'], []) 2022-02-08 12:25:22,301 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:25:22,303 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:25:22,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:25:22,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:25:22,680 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:25:22,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:25:22,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:25:22,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:25:22,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:25:22,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:25:22,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:25:22,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:25:22,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:25:22,818 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 12:25:22,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:25:22,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:25:30,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:25:30,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:25:31,477 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:25:32,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:25:33,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:25:33,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:25:33,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:25:33,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:25:33,868 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 12:25:33,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:25:33,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:25:33,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:25:33,966 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 12:25:33,966 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:25:33,967 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:25:34,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:25:34,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:25:34,305 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:25:34,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:25:34,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:25:34,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:25:34,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:25:34,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:25:34,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:25:34,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:25:34,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:25:34,411 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 12:25:34,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:25:34,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:25:41,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:25:42,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:25:43,227 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:25:43,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:25:45,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:25:45,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:25:45,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:25:45,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:25:45,346 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 12:25:45,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:25:45,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:25:45,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:25:45,467 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 12:25:45,467 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:25:45,469 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 12:25:45,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:25:45,793 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 12:25:45,859 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:25:45,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 12:25:45,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 12:25:45,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 12:25:45,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 12:25:45,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 12:25:45,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 12:25:45,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 12:25:45,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:25:45,986 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 12:25:46,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:25:46,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:25:46,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:25:46,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:25:46,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:25:46,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:25:46,107 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 12:25:46,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:25:46,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:25:46,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:25:46,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:25:46,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:25:46,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:25:46,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:25:46,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:25:46,239 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:25:46,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:25:46,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:25:46,293 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:25:46,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:25:46,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:25:46,346 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:25:46,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:25:46,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:25:46,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:25:46,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:25:46,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:25:46,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:25:46,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:25:46,492 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:25:46,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:25:46,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:25:46,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:25:46,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:25:46,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:25:46,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:25:46,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:25:46,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:25:46,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:25:46,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:25:46,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:25:46,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:25:46,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:25:46,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:25:46,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:25:46,788 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:25:46,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:25:46,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:25:46,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:25:46,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:25:46,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:25:46,900 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:25:46,918 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:25:46,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:25:46,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:25:46,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:25:46,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:25:47,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:25:47,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:25:47,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:25:47,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:25:47,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:25:47,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:25:47,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:25:47,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:25:47,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:25:47,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:25:47,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:25:47,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:25:47,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:25:47,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:25:47,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:25:47,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:25:47,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:25:47,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:25:47,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:25:47,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:25:47,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:25:47,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:25:47,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:25:47,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:25:47,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:25:47,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:25:47,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:25:47,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:25:47,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:25:47,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:25:47,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:25:47,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:25:47,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:25:47,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:25:47,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:25:47,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:25:47,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:25:47,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:25:47,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:25:47,698 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:25:47,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:25:47,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:25:47,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:25:47,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:25:47,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:25:47,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:25:47,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:25:47,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:25:47,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:25:47,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:25:47,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:25:47,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:25:47,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:25:47,960 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 12:25:47,963 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:25:48,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:25:48,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:25:49,494 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:25:49,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:25:49,701 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:25:49,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 5 queries submitted, 0 failed, 15668 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:25:50,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:25:50,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:25:50,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:25:50,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:25:50,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:25:50,852 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 12:25:50,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:25:51,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:25:57,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:25:58,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:25:59,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:26:00,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:26:00,292 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 12:26:00,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:26:00,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:26:00,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:26:00,561 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 12:26:00,561 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:26:00,562 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 12:26:00,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:26:00,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 12:26:00,961 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:26:00,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 12:26:00,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 12:26:00,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 12:26:01,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 12:26:01,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 12:26:01,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 12:26:01,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 12:26:01,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:26:01,086 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 12:26:01,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:26:01,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:26:01,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:26:01,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:26:01,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:26:01,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:26:01,209 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 12:26:01,209 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:26:01,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:26:01,239 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:26:01,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:01,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:26:01,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:01,303 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:01,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:01,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:01,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:01,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:01,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:01,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:01,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:01,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:26:01,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:01,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:01,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:01,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:01,528 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:01,545 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:01,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:01,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:01,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:01,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:26:01,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:01,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:01,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:01,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:01,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:01,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:01,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:01,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:01,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:01,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:26:01,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:01,823 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:01,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:01,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:01,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:01,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:01,918 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:01,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:01,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:01,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:26:01,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:02,010 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:02,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:02,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:02,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:02,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:02,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:02,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:02,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:02,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:26:02,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:02,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:02,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:02,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:02,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:02,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:02,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:02,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:02,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:02,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:26:02,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:02,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:02,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:02,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:02,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:02,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:02,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:02,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:02,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:02,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:26:02,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:02,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:02,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:02,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:02,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:02,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:02,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:02,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:02,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:02,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:26:02,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:02,698 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:02,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:02,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:02,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:02,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:02,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:02,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:02,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:02,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:26:02,849 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:02,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:02,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:02,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:02,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:02,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:02,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:02,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:02,990 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 12:26:02,993 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:26:03,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:26:03,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:26:03,357 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:26:03,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:26:03,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:26:03,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:26:03,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:26:03,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:26:03,431 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:26:03,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:26:03,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:26:03,480 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 12:26:03,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:26:03,538 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:26:09,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:26:10,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:26:11,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:26:11,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:26:13,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:26:13,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:26:13,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:26:13,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:26:13,736 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 12:26:13,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:26:13,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:26:13,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:26:13,819 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 12:26:13,819 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:26:13,822 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:26:13,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:26:14,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:26:14,256 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:26:14,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:26:14,275 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:26:14,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:26:14,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:26:14,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:26:14,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:26:14,431 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:26:14,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:26:14,473 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 12:26:14,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:26:14,522 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:26:22,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:26:23,566 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:26:24,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:26:24,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:26:25,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:26:26,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:26:26,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:26:26,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:26:26,364 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 12:26:26,364 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:26:26,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:26:26,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:26:26,509 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 12:26:26,509 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:26:26,512 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 12:26:26,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:26:26,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 12:26:26,926 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:26:26,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 12:26:26,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 12:26:26,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 12:26:26,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 12:26:27,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 12:26:27,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 12:26:27,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 12:26:27,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:26:27,068 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 12:26:27,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:26:27,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:26:27,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:26:27,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:26:27,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:26:27,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:26:27,194 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 12:26:27,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:26:27,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:26:27,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:26:27,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:27,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:26:27,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:27,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:27,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:27,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:27,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:27,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:27,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:27,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:27,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:27,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:26:27,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:27,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:27,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:27,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:27,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:27,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:27,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:27,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:27,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:27,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:26:27,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:27,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:27,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:27,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:27,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:27,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:27,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:27,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:27,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:27,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:26:27,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:27,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:27,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:27,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:27,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:27,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:28,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:28,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:28,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:28,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:26:28,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:28,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:28,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:28,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:28,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:28,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:28,214 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:28,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:28,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:28,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:26:28,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:28,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:28,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:28,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:28,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:28,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:28,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:28,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:28,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:28,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:26:28,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:28,520 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:28,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:28,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:28,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:28,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:28,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:28,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:28,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:28,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:26:28,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:28,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:28,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:28,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:28,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:28,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:28,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:28,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:28,831 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:28,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:26:28,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:28,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:28,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:28,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:28,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:28,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:28,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:28,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:29,010 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:29,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:26:29,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:29,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:29,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:29,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:29,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:29,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:29,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:29,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:29,207 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 12:26:29,209 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 12:26:29,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:26:29,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 12:26:29,569 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:26:29,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 12:26:29,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 12:26:29,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 12:26:29,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 12:26:29,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 12:26:29,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 12:26:29,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 12:26:29,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:26:29,687 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 12:26:29,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:26:29,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:26:29,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:26:29,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:26:29,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:26:29,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:26:29,803 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 12:26:29,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:26:29,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:26:29,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:26:29,849 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:29,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:26:29,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:29,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:29,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:29,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:29,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:29,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:29,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:30,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:26:30,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:30,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:26:30,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:30,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:30,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:30,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:30,124 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:30,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:30,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:30,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:26:30,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:30,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:26:30,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:30,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:30,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:30,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:30,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:30,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:30,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:30,349 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:26:30,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:30,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:26:30,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:30,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:30,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:30,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:30,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:30,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:30,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:30,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:26:30,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:30,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:26:30,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:30,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:30,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:30,624 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:30,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:30,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:30,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:30,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:26:30,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:30,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:26:30,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:30,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:30,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:30,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:30,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:30,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:30,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:30,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:26:30,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:30,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:26:30,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:30,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:30,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:30,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:30,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:30,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:31,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:31,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:26:31,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:31,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:26:31,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:31,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:31,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:31,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:31,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:31,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:31,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:31,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:26:31,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:31,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:26:31,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:31,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:31,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:31,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:31,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:31,308 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:31,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:31,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:26:31,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:31,377 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:26:31,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:31,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:31,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:31,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:31,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:31,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:31,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:31,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:26:31,532 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 12:26:31,535 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:26:32,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:26:32,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:26:33,065 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:26:33,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:26:33,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:26:33,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:26:33,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:26:33,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:26:33,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:26:33,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:26:33,316 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:26:33,392 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 12:26:33,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:26:33,548 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:26:40,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:26:41,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:26:42,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:26:42,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:26:44,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:26:44,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:26:44,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:26:44,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:26:44,377 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 12:26:44,377 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:26:44,392 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:26:44,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:26:44,494 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 12:26:44,496 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:26:44,499 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:26:44,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:26:44,869 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:26:44,950 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:26:44,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:26:44,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:26:44,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:26:45,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:26:45,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:26:45,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:26:45,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:26:45,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:26:45,108 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 12:26:45,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:26:45,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 15692 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:26:51,854 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:26:52,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:26:53,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:26:53,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:26:53,950 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 12:26:53,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:26:53,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:26:54,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:26:54,052 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 12:26:54,052 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:26:54,055 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:26:54,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:26:54,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:26:54,982 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:26:54,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:26:54,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:26:55,020 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:26:55,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:26:55,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:26:55,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:26:55,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:26:55,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:26:55,128 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 12:26:55,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:26:55,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:27:01,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:27:02,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:27:02,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:27:02,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:27:02,943 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 12:27:02,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:27:02,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:27:03,017 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:27:03,038 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 12:27:03,039 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:27:03,040 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:27:03,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:27:03,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:27:03,792 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:27:03,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:27:03,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:27:03,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:27:03,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:27:03,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:27:03,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:27:03,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:27:03,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:27:03,964 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 12:27:03,999 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:27:04,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:27:13,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:27:23,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:27:23,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:27:24,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:27:26,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:27:27,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:27:27,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:27:27,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:27:27,614 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 12:27:27,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:27:27,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:27:27,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:27:27,729 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 12:27:27,730 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:27:27,732 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 12:27:27,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:27:28,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 12:27:28,150 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:27:28,150 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 12:27:28,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 12:27:28,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 12:27:28,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 12:27:28,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 12:27:28,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 12:27:28,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 12:27:28,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:27:28,273 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 12:27:28,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:27:28,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:27:28,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:27:28,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:27:28,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:27:28,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:27:28,388 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 12:27:28,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:27:28,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:27:28,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:27:28,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:27:28,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:27:28,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:27:28,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:27:28,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:27:28,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:27:28,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:27:28,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:27:28,566 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:27:28,584 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:27:28,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:27:28,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:27:28,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:27:28,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:27:28,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:27:28,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:27:28,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:27:28,742 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:27:28,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:27:28,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:27:28,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:27:28,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:27:28,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:27:28,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:27:28,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:27:28,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:27:28,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:27:28,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:27:28,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:27:28,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:27:28,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:27:28,987 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:27:29,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:27:29,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:27:29,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:27:29,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:27:29,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:27:29,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:27:29,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:27:29,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:27:29,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:27:29,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:27:29,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:27:29,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:27:29,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:27:29,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:27:29,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:27:29,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:27:29,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:27:29,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:27:29,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:27:29,364 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:27:29,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:27:29,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:27:29,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:27:29,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:27:29,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:27:29,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:27:29,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:27:29,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:27:29,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:27:29,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:27:29,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:27:29,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:27:29,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:27:29,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:27:29,627 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:27:29,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:27:29,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:27:29,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:27:29,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:27:29,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:27:29,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:27:29,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:27:29,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:27:29,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:27:29,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:27:29,823 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:27:29,841 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:27:29,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:27:29,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:27:29,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:27:29,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:27:29,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:27:29,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:27:29,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:27:29,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:27:30,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:27:30,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:27:30,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:27:30,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:27:30,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:27:30,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:27:30,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:27:30,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:27:30,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:27:30,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:27:30,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:27:30,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:27:30,227 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:27:30,246 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 12:27:30,246 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:27:30,249 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:27:30,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:27:31,204 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:27:31,384 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:27:31,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:27:31,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:27:31,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:27:31,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:27:31,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:27:31,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:27:31,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:27:31,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:27:32,072 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:27:32,073 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.8259999752. 2022-02-08 12:27:32,073 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:27:32,076 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:27:32,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:27:33,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:27:33,585 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:27:33,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:27:33,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:27:33,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:27:33,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:27:33,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:27:33,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:27:33,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:27:33,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:27:34,426 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:27:34,427 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.35300016403. 2022-02-08 12:27:34,427 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:27:34,428 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:27:35,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:27:35,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:27:36,032 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:27:36,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:27:36,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:27:36,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:27:36,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:27:36,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:27:36,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:27:36,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:27:36,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:27:36,941 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:27:36,943 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.51600003242. 2022-02-08 12:27:36,943 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:27:36,944 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:27:37,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:27:38,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:27:38,392 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:27:38,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:27:38,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:27:38,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:27:38,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:27:38,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:27:38,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:27:38,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:27:38,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:27:39,283 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:27:39,285 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.34200000763. 2022-02-08 12:27:39,285 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:27:39,286 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:27:40,102 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:27:40,566 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:27:40,703 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:27:40,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:27:40,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:27:40,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:27:40,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:27:40,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:27:40,823 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:27:40,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:27:40,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:27:41,676 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:27:41,677 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.39199995995. 2022-02-08 12:27:41,677 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:27:41,678 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:27:42,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:27:42,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:27:42,476 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:27:42,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:27:42,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:27:42,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:27:42,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:27:42,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:27:42,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:27:42,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:27:42,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:27:43,335 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:27:43,335 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.65899991989. 2022-02-08 12:27:43,335 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:27:43,338 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:27:44,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:27:44,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:27:44,673 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:27:44,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:27:44,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:27:44,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:27:44,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:27:44,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:27:44,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:27:44,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:27:44,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:27:45,345 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:27:45,345 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.00999999046. 2022-02-08 12:27:45,346 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:27:45,348 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:27:46,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:27:46,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:27:46,846 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:27:46,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:27:46,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:27:46,897 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:27:46,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:27:46,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:27:46,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:27:47,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:27:47,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:27:47,617 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:27:47,617 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.27100014687. 2022-02-08 12:27:47,617 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:27:47,618 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:27:49,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:27:49,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 5 queries submitted, 0 failed, 15714 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:27:50,039 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:27:50,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:27:50,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:27:50,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:27:50,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:27:50,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:27:50,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:27:50,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:27:50,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:27:51,201 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:27:51,203 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.58599996567. ############################################################ # # RAM FootPrint: 3680.8046875 # ############################################################ 2022-02-08 12:27:51,234 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 12:27:51,234 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 102 seconds. Reason: Iteration:3 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 15735 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:29:33,240 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 12:29:33,240 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 12:29:38,243 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:29:38,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:29:39,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:29:39,907 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:29:39,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:29:39,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:29:40,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:29:40,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:29:40,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:29:40,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:29:40,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:29:40,244 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:29:40,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:29:40,325 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['69167 memcached\n'], []) 2022-02-08 12:29:40,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 12:29:40,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:29:40,382 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['69167 memcached\n'], []) 2022-02-08 12:29:40,384 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:29:40,385 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:29:41,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:29:41,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:29:41,762 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:29:41,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:29:41,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:29:41,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:29:41,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:29:41,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:29:41,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:29:41,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:29:41,961 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:29:41,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:29:42,073 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['125537 memcached\n'], []) 2022-02-08 12:29:42,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 12:29:42,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:29:42,138 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['125537 memcached\n'], []) 2022-02-08 12:29:42,138 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:29:42,142 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:29:43,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:29:43,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:29:44,058 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:29:44,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:29:44,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:29:44,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:29:44,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:29:44,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:29:44,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:29:44,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:29:44,345 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:29:44,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:29:44,421 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['108561 memcached\n'], []) 2022-02-08 12:29:44,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 12:29:44,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:29:44,497 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['108561 memcached\n'], []) 2022-02-08 12:29:44,500 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:29:44,505 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:29:45,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:29:45,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:29:46,124 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:29:46,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:29:46,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:29:46,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:29:46,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:29:46,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:29:46,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:29:46,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:29:46,279 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:29:46,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:29:46,375 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['3083 memcached\n'], []) 2022-02-08 12:29:46,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 12:29:46,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:29:46,438 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['3083 memcached\n'], []) 2022-02-08 12:29:46,438 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:29:46,441 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:29:47,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:29:47,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:29:48,028 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:29:48,028 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:29:48,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:29:48,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:29:48,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:29:48,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:29:48,124 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:29:48,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:29:48,181 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:29:48,183 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:29:48,253 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['127640 memcached\n'], []) 2022-02-08 12:29:48,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 12:29:48,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:29:48,305 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['127640 memcached\n'], []) 2022-02-08 12:29:48,306 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:29:48,309 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:29:49,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 15759 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:29:49,964 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:29:50,092 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:29:50,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:29:50,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:29:50,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:29:50,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:29:50,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:29:50,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:29:50,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:29:50,247 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:29:50,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:29:50,426 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['32109 memcached\n'], []) 2022-02-08 12:29:50,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 12:29:50,456 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:29:50,480 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['32109 memcached\n'], []) 2022-02-08 12:29:50,482 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:29:50,483 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:29:51,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:29:51,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:29:52,154 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:29:52,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:29:52,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:29:52,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:29:52,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:29:52,255 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:29:52,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:29:52,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:29:52,318 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:29:52,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:29:52,362 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['95856 memcached\n'], []) 2022-02-08 12:29:52,364 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 12:29:52,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:29:52,414 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['95856 memcached\n'], []) 2022-02-08 12:29:52,414 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:29:52,417 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:29:53,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:29:53,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:29:53,573 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:29:53,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:29:53,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:29:53,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:29:53,640 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:29:53,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:29:53,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:29:53,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:29:53,713 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:29:53,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:29:53,772 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['111071 memcached\n'], []) 2022-02-08 12:29:53,772 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 12:29:53,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:29:53,822 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['111071 memcached\n'], []) 2022-02-08 12:29:53,822 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:29:53,823 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:29:54,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:29:54,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:29:54,891 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:29:54,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:29:54,905 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:29:54,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:29:54,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:29:55,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:29:55,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:29:55,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:29:55,096 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:29:55,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:29:55,240 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['47858 memcached\n'], []) 2022-02-08 12:29:55,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 12:29:55,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:29:55,289 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['47858 memcached\n'], []) 2022-02-08 12:29:55,289 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:29:55,292 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:29:55,364 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:29:55,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:29:55,674 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:29:55,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:29:55,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:29:55,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:29:55,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:29:55,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:29:55,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:29:55,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:29:55,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:29:55,793 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 12:29:55,823 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:29:55,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:02,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:30:03,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:30:03,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:30:04,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:05,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:30:06,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:30:06,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:30:06,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:30:06,286 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 12:30:06,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:30:06,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:06,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:30:06,382 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 12:30:06,384 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:30:06,385 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:30:06,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:30:06,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:30:06,740 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:30:06,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:30:06,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:30:06,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:30:06,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:30:06,793 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:30:06,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:30:06,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:30:06,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:30:06,851 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 12:30:06,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:30:06,890 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:14,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:30:15,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:30:15,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:30:16,219 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:17,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:30:17,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:30:17,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:30:17,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:30:17,894 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 12:30:17,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:30:17,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:17,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:30:18,005 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 12:30:18,005 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:30:18,006 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 12:30:18,094 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:30:18,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 12:30:18,398 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:30:18,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 12:30:18,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 12:30:18,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 12:30:18,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 12:30:18,459 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 12:30:18,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 12:30:18,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 12:30:18,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:30:18,516 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 12:30:18,545 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:30:18,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:18,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:30:18,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:30:18,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:30:18,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:30:18,638 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 12:30:18,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:30:18,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:18,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:30:18,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:18,698 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:30:18,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:18,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:18,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:18,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:18,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:18,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:18,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:18,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:18,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:18,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:30:18,915 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:18,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:18,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:18,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:18,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:19,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:19,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:19,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:19,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:19,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:30:19,102 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:19,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:19,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:19,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:19,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:19,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:19,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:19,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:19,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:19,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:30:19,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:19,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:19,316 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:19,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:19,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:19,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:19,392 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:19,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:19,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:19,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:30:19,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:19,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:19,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:19,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:19,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:19,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:19,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:19,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:19,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:19,627 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:30:19,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:19,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:19,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:19,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:19,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:19,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:19,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:19,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:19,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:19,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:30:19,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:19,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:19,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:19,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:19,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:19,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:19,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:19,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:19,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:19,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:30:19,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:19,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:20,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:20,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:20,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:20,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:20,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:20,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:20,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:20,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:30:20,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:20,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:20,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:20,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:20,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:20,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:20,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:20,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:20,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:20,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:30:20,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:20,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:20,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:20,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:20,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:20,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:20,431 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:20,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:20,469 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 12:30:20,470 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:30:20,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:30:21,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:30:21,634 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:30:21,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:30:21,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:30:21,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:30:21,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:30:21,831 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:30:21,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:30:21,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:30:21,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:30:22,058 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 12:30:22,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:30:22,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:28,201 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:30:29,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:30:29,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:30:30,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:30:30,584 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 12:30:30,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:30:30,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:30,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:30:30,940 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 12:30:30,941 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:30:30,944 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 12:30:31,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:30:31,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 12:30:31,319 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:30:31,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 12:30:31,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 12:30:31,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 12:30:31,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 12:30:31,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 12:30:31,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 12:30:31,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 12:30:31,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:30:31,447 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 12:30:31,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:30:31,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:31,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:30:31,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:30:31,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:30:31,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:30:31,556 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 12:30:31,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:30:31,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:31,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:30:31,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:31,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:30:31,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:31,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:31,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:31,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:31,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:31,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:31,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:31,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:31,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:31,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:30:31,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:31,831 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:31,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:31,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:31,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:31,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:31,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:31,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:31,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:31,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:30:31,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:31,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:32,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:32,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:32,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:32,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:32,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:32,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:32,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:32,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:30:32,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:32,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:32,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:32,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:32,209 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:32,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:32,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:32,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:32,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:32,293 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:30:32,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:32,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:32,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:32,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:32,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:32,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:32,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:32,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:32,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:32,466 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:30:32,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:32,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:32,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:32,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:32,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:32,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:32,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:32,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:30:32,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:32,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:30:32,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:32,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:32,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:32,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:32,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:32,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:32,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:32,772 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:30:32,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:32,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:30:32,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:32,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:32,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:32,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:32,890 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:32,905 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:32,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:32,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:30:32,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:32,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:30:32,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:33,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:33,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:33,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:33,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:33,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:33,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:33,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:30:33,150 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:33,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:30:33,186 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:33,209 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:33,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:33,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:33,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:33,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:33,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:33,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:30:33,331 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 12:30:33,332 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:30:33,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:30:34,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:30:34,253 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:30:34,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:30:34,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:30:34,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:30:34,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:30:34,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:30:34,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:30:34,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:30:34,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:30:34,414 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 12:30:34,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:30:34,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:40,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:30:41,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:30:42,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:30:42,905 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:44,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:30:45,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:30:45,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:30:45,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:30:45,397 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 12:30:45,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:30:45,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:45,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:30:45,529 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 12:30:45,530 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:30:45,532 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:30:45,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:30:45,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:30:45,950 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:30:45,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:30:45,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:30:45,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:30:46,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:30:46,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:30:46,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:30:46,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:30:46,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:30:46,131 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 12:30:46,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:30:46,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 15780 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:30:54,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:30:55,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:30:55,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:30:55,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:57,464 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:30:57,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:30:57,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:30:58,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:30:58,081 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 12:30:58,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:30:58,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:58,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:30:58,260 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 12:30:58,260 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:30:58,263 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 12:30:58,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:30:58,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 12:30:58,664 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:30:58,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 12:30:58,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 12:30:58,701 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 12:30:58,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 12:30:58,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 12:30:58,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 12:30:58,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 12:30:58,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:30:58,806 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 12:30:58,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:30:58,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:58,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:30:58,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:30:58,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:30:58,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:30:58,940 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 12:30:58,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:30:58,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:30:58,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:30:58,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:59,010 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:30:59,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:59,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:59,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:59,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:59,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:59,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:59,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:59,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:30:59,186 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:59,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:30:59,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:59,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:59,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:59,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:59,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:59,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:59,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:59,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:30:59,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:59,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:30:59,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:59,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:59,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:59,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:59,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:59,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:59,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:59,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:30:59,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:59,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:30:59,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:59,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:59,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:59,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:59,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:59,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:59,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:59,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:30:59,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:59,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:30:59,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:59,836 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:59,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:59,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:59,900 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:59,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:59,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:59,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:30:59,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:00,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:31:00,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:00,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:00,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:00,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:00,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:00,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:00,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:00,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:00,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:00,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:31:00,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:00,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:00,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:00,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:00,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:00,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:00,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:00,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:00,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:00,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:31:00,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:00,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:00,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:00,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:00,494 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:00,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:00,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:00,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:00,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:00,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:31:00,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:00,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:00,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:00,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:00,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:00,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:00,742 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:00,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:00,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:00,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:31:00,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:00,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:00,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:00,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:00,900 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:00,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:00,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:00,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:00,979 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 12:31:00,982 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 12:31:01,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:31:01,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 12:31:01,351 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:31:01,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 12:31:01,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 12:31:01,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 12:31:01,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 12:31:01,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 12:31:01,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 12:31:01,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 12:31:01,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:31:01,474 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 12:31:01,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:31:01,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:31:01,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:31:01,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:31:01,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:31:01,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:31:01,605 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 12:31:01,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:31:01,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:31:01,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:31:01,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:01,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:31:01,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:01,709 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:01,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:01,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:01,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:01,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:01,808 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:01,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:01,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:01,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:31:01,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:01,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:01,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:01,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:01,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:01,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:01,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:02,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:02,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:02,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:31:02,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:02,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:02,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:02,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:02,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:02,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:02,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:02,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:02,181 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:02,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:31:02,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:02,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:02,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:02,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:02,280 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:02,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:02,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:02,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:02,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:02,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:31:02,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:02,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:02,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:02,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:02,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:02,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:02,492 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:02,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:02,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:02,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:31:02,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:02,579 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:02,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:02,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:02,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:02,701 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:02,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:02,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:02,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:02,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:31:02,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:02,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:02,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:02,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:02,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:02,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:02,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:02,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:02,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:02,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:31:02,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:02,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:02,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:02,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:03,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:03,020 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:03,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:03,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:03,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:03,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:31:03,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:03,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:03,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:03,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:03,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:03,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:03,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:03,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:03,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:03,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:31:03,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:03,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:03,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:03,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:03,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:03,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:03,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:03,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:03,397 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 12:31:03,398 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:31:03,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:31:03,788 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:31:03,911 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:31:03,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:31:03,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:31:03,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:31:03,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:31:03,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:31:03,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:31:04,010 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:31:04,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:31:04,049 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 12:31:04,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:31:04,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:31:11,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:31:12,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:31:13,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:31:13,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:31:14,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:31:14,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:31:15,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:31:15,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:31:15,216 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 12:31:15,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:31:15,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:31:15,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:31:15,361 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 12:31:15,361 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:31:15,361 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:31:15,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:31:15,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:31:15,786 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:31:15,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:31:15,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:31:15,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:31:15,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:31:15,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:31:15,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:31:15,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:31:15,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:31:15,934 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 12:31:15,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:31:15,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:31:22,323 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:31:23,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:31:23,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:31:24,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:31:24,089 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 12:31:24,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:31:24,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:31:24,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:31:24,193 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 12:31:24,194 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:31:24,196 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:31:24,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:31:24,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:31:25,075 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:31:25,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:31:25,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:31:25,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:31:25,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:31:25,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:31:25,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:31:25,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:31:25,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:31:25,207 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 12:31:25,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:31:25,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:31:31,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:31:32,255 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:31:32,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:31:33,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:31:33,101 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 12:31:33,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:31:33,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:31:33,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:31:33,193 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 12:31:33,194 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:31:33,196 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:31:33,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:31:33,579 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:31:33,678 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:31:33,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:31:33,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:31:33,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:31:33,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:31:33,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:31:33,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:31:33,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:31:33,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:31:33,815 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 12:31:33,841 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:31:33,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:31:41,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:31:42,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:31:43,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:31:43,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:31:46,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:31:46,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:31:46,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:31:47,209 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:31:47,335 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 12:31:47,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:31:47,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:31:47,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:31:47,724 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 12:31:47,724 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:31:47,726 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 12:31:47,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:31:48,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 12:31:48,144 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:31:48,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 12:31:48,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 12:31:48,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 12:31:48,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 12:31:48,209 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 12:31:48,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 12:31:48,239 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 12:31:48,255 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:31:48,273 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 12:31:48,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:31:48,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:31:48,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:31:48,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:31:48,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:31:48,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:31:48,388 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 12:31:48,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:31:48,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:31:48,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:31:48,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:48,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:31:48,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:48,492 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:48,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:48,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:48,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:48,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:48,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:48,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:31:48,624 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:48,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:31:48,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:48,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:48,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:48,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:48,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:48,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:48,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:48,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:31:48,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:48,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:31:48,836 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:48,854 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:48,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:48,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:48,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:48,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:48,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:48,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:31:48,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:48,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:31:49,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:49,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:49,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:49,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:49,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:49,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:49,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:49,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:31:49,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:49,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:31:49,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:49,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:49,234 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:49,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:49,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:49,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:49,303 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:49,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:31:49,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:49,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:31:49,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:49,392 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:49,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:49,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:49,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:49,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:49,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:49,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:31:49,522 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:49,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:31:49,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:49,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:49,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:49,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:49,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:49,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:49,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:49,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:31:49,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:49,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:31:49,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:49,757 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:49,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:49,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:49,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:49,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:49,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:49,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:31:49,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:49,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:31:49,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 5 queries submitted, 0 failed, 15803 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:31:49,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:49,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:49,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:49,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:50,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:50,028 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:50,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:31:50,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:50,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:31:50,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:50,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:50,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:50,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:50,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:50,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:50,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:50,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:31:50,250 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 12:31:50,250 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:31:50,253 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:31:50,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:31:51,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:31:51,207 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:31:51,209 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:31:51,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:31:51,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:31:51,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:31:51,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:31:51,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:31:51,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:31:51,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:31:51,869 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:31:51,871 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.62000012398. 2022-02-08 12:31:51,871 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:31:51,872 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:31:52,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:31:53,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:31:53,423 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:31:53,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:31:53,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:31:53,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:31:53,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:31:53,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:31:53,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:31:53,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:31:53,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:31:54,257 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:31:54,257 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.3869998455. 2022-02-08 12:31:54,257 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:31:54,259 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:31:55,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:31:55,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:31:55,740 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:31:55,742 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:31:55,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:31:55,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:31:55,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:31:55,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:31:55,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:31:55,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:31:55,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:31:56,509 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:31:56,509 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.25200009346. 2022-02-08 12:31:56,509 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:31:56,510 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:31:57,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:31:57,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:31:58,055 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:31:58,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:31:58,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:31:58,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:31:58,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:31:58,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:31:58,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:31:58,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:31:58,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:31:58,795 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:31:58,796 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.28600001335. 2022-02-08 12:31:58,796 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:31:58,796 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:31:59,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:32:00,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:32:00,492 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:32:00,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:32:00,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:32:00,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:32:00,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:32:00,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:32:00,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:32:00,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:32:00,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:32:01,362 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:32:01,362 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.56699991226. 2022-02-08 12:32:01,364 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:32:01,365 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:32:01,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:32:02,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:32:02,493 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:32:02,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:32:02,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:32:02,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:32:02,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:32:02,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:32:02,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:32:02,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:32:02,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:32:03,302 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:32:03,302 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.9390001297. 2022-02-08 12:32:03,302 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:32:03,303 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:32:04,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:32:04,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:32:05,131 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:32:05,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:32:05,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:32:05,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:32:05,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:32:05,219 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:32:05,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:32:05,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:32:05,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:32:05,818 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:32:05,818 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.51600003242. 2022-02-08 12:32:05,819 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:32:05,819 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:32:06,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:32:06,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:32:06,874 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:32:06,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:32:06,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:32:06,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:32:07,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:32:07,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:32:07,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:32:07,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:32:07,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:32:07,654 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:32:07,654 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.83500003815. 2022-02-08 12:32:07,654 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:32:07,655 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:32:07,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:32:08,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:32:08,894 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:32:08,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:32:09,002 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:32:09,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:32:09,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:32:09,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:32:09,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:32:09,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:32:09,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:32:10,569 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 12:32:10,569 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:32:12,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:32:12,790 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 12:32:12,792 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:32:14,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:32:15,114 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 12:32:15,115 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:32:17,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:32:17,762 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 12:32:17,762 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:32:19,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:32:20,305 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 12:32:20,306 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:32:22,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:32:22,878 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 12:32:22,880 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:32:24,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:32:25,408 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 12:32:25,408 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:32:27,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:32:27,911 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 12:32:27,913 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:32:29,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:32:30,105 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 12:32:30,105 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:32:32,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:32:32,309 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:32:32,309 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 24.6559998989. ############################################################ # # RAM FootPrint: 3301.63671875 # ############################################################ 2022-02-08 12:32:32,344 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 12:32:32,344 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 73 seconds. Reason: Iteration:4 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 15830 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:33:45,348 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 12:33:45,349 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 5 queries submitted, 0 failed, 15855 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:33:50,351 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:33:51,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:33:51,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:33:51,926 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:33:51,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:33:51,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:33:52,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:33:52,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:33:52,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:33:52,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:33:52,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:33:52,371 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:33:52,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:33:52,480 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['69925 memcached\n'], []) 2022-02-08 12:33:52,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 12:33:52,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:33:52,578 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['69925 memcached\n'], []) 2022-02-08 12:33:52,578 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:33:52,581 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:33:53,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:33:53,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:33:53,993 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:33:53,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:33:54,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:33:54,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:33:54,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:33:54,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:33:54,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:33:54,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:33:54,160 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:33:54,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:33:54,226 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['126352 memcached\n'], []) 2022-02-08 12:33:54,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 12:33:54,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:33:54,286 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['126352 memcached\n'], []) 2022-02-08 12:33:54,286 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:33:54,289 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:33:55,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:33:56,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:33:56,243 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:33:56,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:33:56,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:33:56,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:33:56,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:33:56,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:33:56,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:33:56,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:33:56,411 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:33:56,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:33:56,487 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['109478 memcached\n'], []) 2022-02-08 12:33:56,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 12:33:56,520 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:33:56,551 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['109478 memcached\n'], []) 2022-02-08 12:33:56,551 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:33:56,553 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:33:57,377 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:33:57,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:33:58,121 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:33:58,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:33:58,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:33:58,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:33:58,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:33:58,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:33:58,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:33:58,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:33:58,358 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:33:58,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:33:58,426 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['3911 memcached\n'], []) 2022-02-08 12:33:58,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 12:33:58,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:33:58,494 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['3911 memcached\n'], []) 2022-02-08 12:33:58,494 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:33:58,496 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:33:59,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:33:59,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:34:00,012 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:34:00,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:34:00,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:34:00,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:34:00,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:34:00,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:34:00,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:34:00,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:34:00,163 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:34:00,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:34:00,229 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['128458 memcached\n'], []) 2022-02-08 12:34:00,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 12:34:00,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:34:00,282 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['128458 memcached\n'], []) 2022-02-08 12:34:00,282 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:34:00,286 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:34:01,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:34:02,010 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:34:02,131 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:34:02,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:34:02,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:34:02,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:34:02,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:34:02,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:34:02,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:34:02,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:34:02,288 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:34:02,288 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:34:02,345 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['32927 memcached\n'], []) 2022-02-08 12:34:02,346 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 12:34:02,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:34:02,395 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['32927 memcached\n'], []) 2022-02-08 12:34:02,395 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:34:02,398 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:34:03,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:34:03,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:34:03,966 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:34:03,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:34:03,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:34:03,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:34:04,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:34:04,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:34:04,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:34:04,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:34:04,098 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:34:04,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:34:04,148 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['96613 memcached\n'], []) 2022-02-08 12:34:04,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 12:34:04,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:34:04,197 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['96613 memcached\n'], []) 2022-02-08 12:34:04,197 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:34:04,203 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:34:05,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:34:05,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:34:05,566 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:34:05,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:34:05,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:34:05,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:34:05,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:34:05,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:34:05,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:34:05,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:34:05,717 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:34:05,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:34:05,776 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['111873 memcached\n'], []) 2022-02-08 12:34:05,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 12:34:05,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:34:05,838 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['111873 memcached\n'], []) 2022-02-08 12:34:05,838 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:34:05,841 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:34:06,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:34:07,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:34:07,163 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:34:07,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:34:07,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:34:07,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:34:07,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:34:07,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:34:07,280 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:34:07,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:34:07,326 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:34:07,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:34:07,411 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['48686 memcached\n'], []) 2022-02-08 12:34:07,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 12:34:07,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:34:07,505 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['48686 memcached\n'], []) 2022-02-08 12:34:07,506 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:34:07,509 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:34:07,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:34:07,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:34:07,877 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:34:07,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:34:07,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:34:07,905 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:34:07,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:34:07,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:34:08,002 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:34:08,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:34:08,028 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:34:08,043 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 12:34:08,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:34:08,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:34:14,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:34:15,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:34:16,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:34:16,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:34:18,150 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:34:18,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:34:18,492 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:34:18,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:34:18,621 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 12:34:18,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:34:18,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:34:18,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:34:18,719 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 12:34:18,720 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:34:18,723 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:34:18,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:34:19,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:34:19,075 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:34:19,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:34:19,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:34:19,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:34:19,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:34:19,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:34:19,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:34:19,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:34:19,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:34:19,188 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 12:34:19,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:34:19,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:34:26,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:34:27,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:34:28,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:34:28,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:34:29,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:34:30,017 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:34:30,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:34:30,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:34:30,203 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 12:34:30,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:34:30,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:34:30,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:34:30,309 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 12:34:30,309 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:34:30,312 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 12:34:30,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:34:30,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 12:34:30,703 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:34:30,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 12:34:30,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 12:34:30,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 12:34:30,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 12:34:30,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 12:34:30,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 12:34:30,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 12:34:30,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:34:30,832 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 12:34:30,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:34:30,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:34:30,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:34:30,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:34:30,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:34:30,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:34:30,951 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 12:34:30,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:34:30,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:34:30,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:34:31,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:31,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:34:31,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:31,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:31,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:31,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:31,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:31,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:31,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:31,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:31,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:31,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:34:31,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:31,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:31,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:31,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:31,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:31,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:31,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:31,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:31,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:31,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:34:31,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:31,464 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:31,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:31,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:31,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:31,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:31,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:31,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:31,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:31,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:34:31,627 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:31,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:31,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:31,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:31,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:31,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:31,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:31,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:31,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:31,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:34:31,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:31,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:31,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:31,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:31,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:31,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:31,918 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:31,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:31,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:31,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:34:31,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:32,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:32,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:32,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:32,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:32,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:32,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:32,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:32,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:32,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:34:32,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:32,201 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:32,219 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:32,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:32,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:32,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:32,288 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:32,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:32,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:32,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:34:32,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:32,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:32,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:32,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:32,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:32,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:32,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:32,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:32,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:32,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:34:32,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:32,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:32,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:32,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:32,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:32,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:32,640 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:32,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:32,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:32,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:34:32,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:32,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:32,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:32,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:32,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:32,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:32,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:32,836 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:32,854 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 12:34:32,857 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:34:33,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:34:33,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:34:33,812 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:34:33,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:34:33,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:34:34,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:34:34,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:34:34,443 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:34:34,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:34:34,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:34:34,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:34:34,539 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 12:34:34,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:34:34,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:34:40,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:34:41,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:34:42,227 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:34:42,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:34:42,826 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 12:34:42,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:34:42,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:34:43,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:34:43,104 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 12:34:43,105 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:34:43,108 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 12:34:43,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:34:43,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 12:34:43,484 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:34:43,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 12:34:43,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 12:34:43,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 12:34:43,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 12:34:43,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 12:34:43,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 12:34:43,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 12:34:43,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:34:43,598 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 12:34:43,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:34:43,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:34:43,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:34:43,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:34:43,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:34:43,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:34:43,711 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 12:34:43,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:34:43,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:34:43,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:34:43,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:43,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:34:43,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:43,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:43,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:43,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:43,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:43,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:43,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:43,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:34:43,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:43,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:34:44,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:44,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:44,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:44,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:44,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:44,094 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:44,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:44,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:34:44,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:44,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:34:44,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:44,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:44,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:44,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:44,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:44,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:44,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:44,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:34:44,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:44,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:34:44,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:44,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:44,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:44,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:44,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:44,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:44,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:44,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:34:44,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:44,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:34:44,538 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:44,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:44,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:44,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:44,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:44,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:44,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:44,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:34:44,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:44,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:34:44,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:44,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:44,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:44,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:44,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:44,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:44,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:44,836 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:34:44,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:44,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:34:44,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:44,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:44,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:44,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:44,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:44,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:44,987 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:45,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:34:45,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:45,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:34:45,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:45,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:45,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:45,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:45,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:45,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:45,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:45,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:34:45,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:45,201 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:34:45,219 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:45,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:45,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:45,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:45,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:45,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:45,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:45,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:34:45,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:45,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:34:45,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:45,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:45,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:45,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:45,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:45,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:45,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:45,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:34:45,565 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 12:34:45,566 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:34:45,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:34:45,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:34:45,928 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:34:45,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:34:45,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:34:45,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:34:45,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:34:45,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:34:46,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:34:46,017 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:34:46,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:34:46,052 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 12:34:46,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:34:46,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 15875 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:34:52,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:34:53,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:34:53,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:34:54,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:34:55,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:34:56,219 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:34:56,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:34:56,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:34:56,489 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 12:34:56,489 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:34:56,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:34:56,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:34:56,573 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 12:34:56,573 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:34:56,576 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:34:56,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:34:56,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:34:57,078 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:34:57,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:34:57,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:34:57,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:34:57,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:34:57,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:34:57,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:34:57,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:34:57,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:34:57,247 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 12:34:57,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:34:57,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:35:05,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:35:06,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:35:06,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:35:07,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:35:09,028 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:35:09,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:35:09,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:35:09,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:35:09,546 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 12:35:09,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:35:09,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:35:09,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:35:09,678 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 12:35:09,678 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:35:09,680 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 12:35:09,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:35:10,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 12:35:10,131 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:35:10,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 12:35:10,150 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 12:35:10,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 12:35:10,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 12:35:10,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 12:35:10,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 12:35:10,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 12:35:10,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:35:10,282 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 12:35:10,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:35:10,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:35:10,346 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:35:10,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:35:10,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:35:10,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:35:10,421 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 12:35:10,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:35:10,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:35:10,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:35:10,477 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:10,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:35:10,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:10,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:10,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:10,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:10,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:10,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:10,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:10,701 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:10,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:10,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:35:10,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:10,788 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:10,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:10,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:10,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:10,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:10,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:10,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:10,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:10,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:35:10,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:11,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:11,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:11,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:11,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:11,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:11,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:11,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:11,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:11,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:35:11,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:11,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:11,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:11,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:11,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:11,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:11,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:11,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:11,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:11,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:35:11,423 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:11,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:11,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:11,494 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:11,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:11,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:11,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:11,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:11,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:11,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:35:11,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:11,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:11,709 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:11,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:11,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:11,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:11,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:11,823 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:11,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:11,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:35:11,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:11,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:11,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:11,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:11,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:11,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:12,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:12,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:12,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:12,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:35:12,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:12,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:12,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:12,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:12,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:12,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:12,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:12,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:12,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:12,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:35:12,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:12,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:12,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:12,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:12,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:12,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:12,423 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:12,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:12,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:12,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:35:12,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:12,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:12,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:12,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:12,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:12,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:12,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:12,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:12,684 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 12:35:12,686 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 12:35:12,760 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:35:12,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 12:35:13,052 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:35:13,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 12:35:13,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 12:35:13,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 12:35:13,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 12:35:13,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 12:35:13,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 12:35:13,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 12:35:13,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:35:13,164 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 12:35:13,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:35:13,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:35:13,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:35:13,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:35:13,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:35:13,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:35:13,270 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 12:35:13,270 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:35:13,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:35:13,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:35:13,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:13,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:35:13,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:13,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:13,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:13,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:13,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:13,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:13,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:13,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:35:13,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:13,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:35:13,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:13,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:13,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:13,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:13,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:13,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:13,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:13,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:35:13,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:13,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:35:13,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:13,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:13,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:13,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:13,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:13,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:13,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:13,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:35:13,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:13,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:35:13,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:13,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:13,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:13,964 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:13,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:14,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:14,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:14,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:35:14,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:14,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:35:14,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:14,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:14,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:14,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:14,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:14,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:14,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:14,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:35:14,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:14,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:35:14,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:14,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:14,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:14,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:14,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:14,346 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:14,364 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:14,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:35:14,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:14,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:35:14,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:14,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:14,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:14,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:14,499 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:14,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:14,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:14,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:35:14,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:14,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:35:14,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:14,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:14,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:14,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:14,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:14,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:14,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:14,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:35:14,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:14,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:35:14,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:14,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:14,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:14,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:14,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:14,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:14,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:14,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:35:14,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:14,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:35:14,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:14,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:14,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:14,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:15,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:15,020 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:15,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:15,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:35:15,071 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 12:35:15,072 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:35:15,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:35:16,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:35:16,622 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:35:16,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:35:16,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:35:16,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:35:16,772 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:35:16,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:35:16,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:35:16,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:35:16,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:35:16,905 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 12:35:16,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:35:16,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:35:24,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:35:25,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:35:25,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:35:26,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:35:27,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:35:27,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:35:27,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:35:27,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:35:28,068 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 12:35:28,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:35:28,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:35:28,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:35:28,229 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 12:35:28,229 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:35:28,230 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:35:28,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:35:28,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:35:28,625 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:35:28,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:35:28,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:35:28,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:35:28,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:35:28,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:35:28,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:35:28,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:35:28,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:35:28,765 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 12:35:28,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:35:28,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:35:35,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:35:36,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:35:36,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:35:37,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:35:37,326 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 12:35:37,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:35:37,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:35:37,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:35:37,436 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 12:35:37,436 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:35:37,463 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:35:37,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:35:38,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:35:38,367 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:35:38,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:35:38,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:35:38,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:35:38,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:35:38,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:35:38,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:35:38,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:35:38,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:35:38,505 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 12:35:38,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:35:38,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:35:44,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:35:45,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:35:46,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:35:46,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:35:46,635 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 12:35:46,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:35:46,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:35:46,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:35:46,716 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 12:35:46,717 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:35:46,719 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:35:47,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:35:47,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:35:48,237 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:35:48,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:35:48,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:35:48,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:35:48,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:35:48,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:35:48,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:35:48,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:35:48,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:35:48,526 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 12:35:48,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:35:48,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 15898 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:35:56,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:35:58,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:35:58,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:35:59,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:36:02,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:36:02,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:36:02,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:36:03,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:36:03,240 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 12:36:03,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:36:03,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:36:03,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:36:03,404 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 12:36:03,404 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:36:03,407 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 12:36:03,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:36:03,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 12:36:03,799 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:36:03,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 12:36:03,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 12:36:03,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 12:36:03,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 12:36:03,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 12:36:03,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 12:36:03,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 12:36:03,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:36:03,926 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 12:36:03,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:36:03,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:36:03,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:36:04,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:36:04,028 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:36:04,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:36:04,059 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 12:36:04,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:36:04,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:36:04,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:36:04,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:36:04,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:36:04,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:36:04,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:36:04,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:36:04,214 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:36:04,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:36:04,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:36:04,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:36:04,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:36:04,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:36:04,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:36:04,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:36:04,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:36:04,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:36:04,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:36:04,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:36:04,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:36:04,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:36:04,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:36:04,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:36:04,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:36:04,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:36:04,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:36:04,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:36:04,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:36:04,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:36:04,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:36:04,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:36:04,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:36:04,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:36:04,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:36:04,709 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:36:04,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:36:04,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:36:04,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:36:04,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:36:04,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:36:04,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:36:04,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:36:04,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:36:04,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:36:04,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:36:04,915 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:36:04,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:36:04,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:36:04,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:36:04,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:36:05,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:36:05,028 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:36:05,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:36:05,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:36:05,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:36:05,102 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:36:05,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:36:05,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:36:05,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:36:05,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:36:05,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:36:05,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:36:05,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:36:05,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:36:05,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:36:05,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:36:05,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:36:05,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:36:05,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:36:05,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:36:05,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:36:05,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:36:05,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:36:05,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:36:05,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:36:05,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:36:05,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:36:05,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:36:05,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:36:05,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:36:05,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:36:05,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:36:05,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:36:05,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:36:05,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:36:05,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:36:05,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:36:05,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:36:05,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:36:05,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:36:05,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:36:05,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:36:05,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:36:05,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:36:05,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:36:05,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:36:05,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:36:05,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:36:05,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:36:05,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:36:05,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:36:05,905 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:36:05,923 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 12:36:05,924 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:36:05,926 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:36:06,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:36:06,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:36:07,071 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:36:07,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:36:07,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:36:07,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:36:07,122 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:36:07,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:36:07,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:36:07,186 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:36:07,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:36:07,726 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:36:07,727 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.80399990082. 2022-02-08 12:36:07,727 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:36:07,730 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:36:08,545 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:36:08,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:36:09,059 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:36:09,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:36:09,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:36:09,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:36:09,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:36:09,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:36:09,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:36:09,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:36:09,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:36:09,852 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:36:09,852 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.125. 2022-02-08 12:36:09,852 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:36:09,855 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:36:10,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:36:11,288 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:36:11,460 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:36:11,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:36:11,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:36:11,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:36:11,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:36:11,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:36:11,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:36:11,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:36:11,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:36:12,243 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:36:12,243 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.3900001049. 2022-02-08 12:36:12,243 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:36:12,246 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:36:13,204 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:36:13,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:36:13,882 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:36:13,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:36:13,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:36:13,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:36:13,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:36:13,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:36:13,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:36:14,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:36:14,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:36:14,743 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:36:14,744 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.50199985504. 2022-02-08 12:36:14,744 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:36:14,746 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:36:15,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:36:16,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:36:16,427 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:36:16,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:36:16,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:36:16,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:36:16,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:36:16,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:36:16,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:36:16,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:36:16,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:36:17,243 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:36:17,244 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.5. 2022-02-08 12:36:17,244 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:36:17,247 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:36:17,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:36:17,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:36:17,927 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:36:17,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:36:17,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:36:17,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:36:18,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:36:18,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:36:18,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:36:18,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:36:18,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:36:18,660 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:36:18,660 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.41500020027. 2022-02-08 12:36:18,661 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:36:18,661 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:36:19,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:36:20,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:36:20,196 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:36:20,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:36:20,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:36:20,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:36:20,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:36:20,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:36:20,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:36:20,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:36:20,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:36:20,842 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:36:20,842 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.18099999428. 2022-02-08 12:36:20,842 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:36:20,844 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:36:21,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:36:21,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:36:21,523 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:36:21,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:36:21,545 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:36:21,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:36:21,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:36:21,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:36:21,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:36:21,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:36:21,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:36:22,223 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:36:22,223 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.38199996948. 2022-02-08 12:36:22,224 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:36:22,226 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:36:23,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:36:23,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:36:24,158 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:36:24,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:36:24,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:36:24,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:36:24,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:36:24,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:36:24,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:36:24,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:36:25,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:36:26,174 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 12:36:26,176 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:36:28,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:36:28,361 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:36:28,361 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.1360001564. ############################################################ # # RAM FootPrint: 3336.3125 # ############################################################ 2022-02-08 12:36:28,391 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 12:36:28,391 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 100 seconds. Reason: Iteration:5 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 15923 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 15949 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:38:08,395 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 12:38:08,395 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 12:38:13,398 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:38:14,219 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:38:14,869 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:38:15,134 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:38:15,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:38:15,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:38:15,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:38:15,275 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:38:15,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:38:15,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:38:15,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:38:15,382 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:38:15,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:38:15,434 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['70662 memcached\n'], []) 2022-02-08 12:38:15,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 12:38:15,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:38:15,486 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['70662 memcached\n'], []) 2022-02-08 12:38:15,486 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:38:15,489 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:38:16,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:38:16,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:38:17,033 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:38:17,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:38:17,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:38:17,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:38:17,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:38:17,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:38:17,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:38:17,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:38:17,197 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:38:17,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:38:17,276 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['127120 memcached\n'], []) 2022-02-08 12:38:17,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 12:38:17,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:38:17,338 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['127120 memcached\n'], []) 2022-02-08 12:38:17,338 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:38:17,339 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:38:18,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:38:18,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:38:18,844 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:38:18,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:38:18,869 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:38:18,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:38:18,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:38:18,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:38:18,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:38:18,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:38:19,016 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:38:19,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:38:19,084 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['110355 memcached\n'], []) 2022-02-08 12:38:19,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 12:38:19,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:38:19,144 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['110355 memcached\n'], []) 2022-02-08 12:38:19,144 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:38:19,147 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:38:20,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:38:20,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:38:21,115 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:38:21,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:38:21,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:38:21,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:38:21,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:38:21,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:38:21,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:38:21,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:38:21,262 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:38:21,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:38:21,329 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['4680 memcached\n'], []) 2022-02-08 12:38:21,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 12:38:21,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:38:21,378 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['4680 memcached\n'], []) 2022-02-08 12:38:21,380 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:38:21,381 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:38:22,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:38:22,693 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:38:22,822 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:38:22,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:38:22,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:38:22,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:38:22,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:38:22,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:38:22,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:38:22,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:38:23,007 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:38:23,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:38:23,078 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['129227 memcached\n'], []) 2022-02-08 12:38:23,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 12:38:23,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:38:23,138 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['129227 memcached\n'], []) 2022-02-08 12:38:23,138 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:38:23,140 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:38:24,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:38:25,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:38:25,625 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:38:25,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:38:25,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:38:25,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:38:25,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:38:25,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:38:25,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:38:25,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:38:25,763 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:38:25,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:38:25,845 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['33696 memcached\n'], []) 2022-02-08 12:38:25,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 12:38:25,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:38:25,894 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['33696 memcached\n'], []) 2022-02-08 12:38:25,894 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:38:25,895 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:38:26,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:38:27,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:38:27,496 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:38:27,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:38:27,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:38:27,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:38:27,545 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:38:27,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:38:27,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:38:27,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:38:27,625 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:38:27,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:38:27,671 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['97355 memcached\n'], []) 2022-02-08 12:38:27,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 12:38:27,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:38:27,729 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['97355 memcached\n'], []) 2022-02-08 12:38:27,729 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:38:27,732 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:38:28,464 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:38:28,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:38:29,015 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:38:29,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:38:29,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:38:29,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:38:29,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:38:29,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:38:29,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:38:29,150 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:38:29,164 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:38:29,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:38:29,220 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['112611 memcached\n'], []) 2022-02-08 12:38:29,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 12:38:29,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:38:29,266 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['112611 memcached\n'], []) 2022-02-08 12:38:29,266 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:38:29,269 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:38:29,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:38:30,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:38:30,815 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:38:30,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:38:30,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:38:30,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:38:30,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:38:30,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:38:30,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:38:30,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:38:30,934 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:38:30,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:38:30,996 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['49566 memcached\n'], []) 2022-02-08 12:38:30,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 12:38:31,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:38:31,042 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['49566 memcached\n'], []) 2022-02-08 12:38:31,042 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:38:31,045 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:38:31,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:38:31,341 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:38:31,401 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:38:31,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:38:31,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:38:31,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:38:31,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:38:31,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:38:31,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:38:31,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:38:31,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:38:31,523 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 12:38:31,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:38:31,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:38:38,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:38:39,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:38:39,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:38:40,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:38:41,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:38:42,020 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:38:42,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:38:42,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:38:42,292 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 12:38:42,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:38:42,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:38:42,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:38:42,382 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 12:38:42,384 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:38:42,385 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:38:42,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:38:42,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:38:42,756 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:38:42,757 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:38:42,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:38:42,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:38:42,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:38:42,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:38:42,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:38:42,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:38:42,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:38:42,867 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 12:38:42,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:38:42,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 15973 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:38:50,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:38:51,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:38:52,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:38:52,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:38:53,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:38:54,084 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:38:54,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:38:54,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:38:54,280 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 12:38:54,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:38:54,293 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:38:54,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:38:54,388 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 12:38:54,388 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:38:54,390 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 12:38:54,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:38:54,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 12:38:54,785 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:38:54,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 12:38:54,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 12:38:54,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 12:38:54,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 12:38:54,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 12:38:54,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 12:38:54,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 12:38:54,890 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:38:54,905 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 12:38:54,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:38:54,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:38:54,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:38:54,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:38:54,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:38:55,002 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:38:55,017 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 12:38:55,017 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:38:55,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:38:55,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:38:55,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:38:55,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:38:55,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:38:55,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:38:55,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:38:55,181 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:38:55,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:38:55,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:38:55,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:38:55,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:38:55,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:38:55,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:38:55,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:38:55,377 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:38:55,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:38:55,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:38:55,443 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:38:55,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:38:55,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:38:55,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:38:55,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:38:55,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:38:55,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:38:55,579 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:38:55,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:38:55,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:38:55,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:38:55,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:38:55,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:38:55,698 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:38:55,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:38:55,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:38:55,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:38:55,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:38:55,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:38:55,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:38:55,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:38:55,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:38:55,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:38:55,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:38:55,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:38:55,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:38:55,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:38:55,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:38:55,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:38:55,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:38:56,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:38:56,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:38:56,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:38:56,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:38:56,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:38:56,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:38:56,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:38:56,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:38:56,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:38:56,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:38:56,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:38:56,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:38:56,214 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:38:56,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:38:56,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:38:56,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:38:56,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:38:56,303 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:38:56,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:38:56,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:38:56,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:38:56,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:38:56,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:38:56,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:38:56,423 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:38:56,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:38:56,459 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:38:56,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:38:56,492 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:38:56,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:38:56,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:38:56,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:38:56,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:38:56,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:38:56,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:38:56,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:38:56,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:38:56,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:38:56,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:38:56,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:38:56,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:38:56,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:38:56,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:38:56,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:38:56,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:38:56,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:38:56,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:38:56,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:38:56,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:38:56,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:38:56,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:38:56,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:38:56,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:38:56,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:38:56,927 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 12:38:56,930 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:38:57,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:38:57,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:38:58,040 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:38:58,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:38:58,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:38:58,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:38:58,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:38:58,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:38:58,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:38:58,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:38:58,303 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:38:58,319 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 12:38:58,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:38:58,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:39:04,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:39:05,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:39:06,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:39:06,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:39:06,871 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 12:39:06,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:39:06,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:39:07,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:39:07,032 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 12:39:07,032 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:39:07,035 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 12:39:07,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:39:07,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 12:39:07,450 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:39:07,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 12:39:07,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 12:39:07,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 12:39:07,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 12:39:07,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 12:39:07,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 12:39:07,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 12:39:07,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:39:07,591 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 12:39:07,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:39:07,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:39:07,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:39:07,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:39:07,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:39:07,701 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:39:07,736 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 12:39:07,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:39:07,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:39:07,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:39:07,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:07,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:39:07,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:07,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:07,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:07,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:07,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:07,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:07,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:08,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:08,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:08,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:39:08,102 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:08,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:08,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:08,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:08,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:08,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:08,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:08,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:08,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:08,275 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:39:08,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:08,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:08,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:08,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:08,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:08,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:08,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:08,431 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:08,456 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:08,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:39:08,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:08,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:08,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:08,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:08,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:08,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:08,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:08,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:08,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:08,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:39:08,698 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:08,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:08,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:08,760 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:08,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:08,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:08,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:08,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:08,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:08,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:39:08,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:08,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:08,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:08,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:08,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:08,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:09,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:09,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:09,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:09,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:39:09,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:09,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:09,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:09,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:09,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:09,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:09,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:09,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:09,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:09,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:39:09,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:09,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:09,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:09,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:09,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:09,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:09,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:09,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:09,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:09,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:39:09,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:09,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:09,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:09,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:09,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:09,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:09,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:09,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:09,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:09,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:39:09,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:09,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:09,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:09,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:09,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:09,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:09,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:09,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:09,815 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 12:39:09,818 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:39:09,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:39:10,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:39:10,243 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:39:10,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:39:10,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:39:10,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:39:10,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:39:10,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:39:10,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:39:10,346 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:39:10,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:39:10,395 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 12:39:10,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:39:10,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:39:16,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:39:17,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:39:18,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:39:19,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:39:21,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:39:21,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:39:21,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:39:21,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:39:22,013 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 12:39:22,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:39:22,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:39:22,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:39:22,104 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 12:39:22,104 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:39:22,107 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:39:22,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:39:22,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:39:22,546 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:39:22,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:39:22,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:39:22,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:39:22,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:39:22,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:39:22,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:39:22,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:39:22,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:39:22,714 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 12:39:22,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:39:22,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:39:30,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:39:31,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:39:31,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:39:32,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:39:34,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:39:34,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:39:34,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:39:34,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:39:34,572 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 12:39:34,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:39:34,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:39:34,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:39:34,726 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 12:39:34,726 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:39:34,729 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 12:39:34,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:39:35,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 12:39:35,130 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:39:35,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 12:39:35,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 12:39:35,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 12:39:35,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 12:39:35,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 12:39:35,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 12:39:35,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 12:39:35,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:39:35,272 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 12:39:35,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:39:35,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:39:35,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:39:35,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:39:35,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:39:35,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:39:35,397 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 12:39:35,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:39:35,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:39:35,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:39:35,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:35,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:39:35,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:35,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:35,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:35,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:35,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:35,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:35,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:35,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:35,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:35,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:39:35,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:35,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:35,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:35,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:35,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:35,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:35,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:35,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:35,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:35,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:39:35,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:35,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:35,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:36,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:36,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:36,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:36,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:36,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:36,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:36,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:39:36,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:36,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:36,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:36,204 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:36,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:36,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:36,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:36,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:36,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:36,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:39:36,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:36,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:36,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:36,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:36,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:36,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:36,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:36,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:36,499 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:36,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:39:36,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:36,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:36,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:36,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:36,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:36,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:36,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:36,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:36,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:36,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:39:36,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:36,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:36,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:36,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:36,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:36,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:36,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:36,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:36,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:36,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:39:36,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:36,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:36,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:36,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:36,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:36,999 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:37,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:37,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:37,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:37,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:39:37,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:37,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:37,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:37,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:37,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:37,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:37,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:37,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:37,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:37,255 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:39:37,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:37,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:37,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:37,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:37,349 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:37,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:37,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:37,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:37,423 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 12:39:37,424 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 12:39:37,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:39:37,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 12:39:37,785 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:39:37,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 12:39:37,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 12:39:37,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 12:39:37,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 12:39:37,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 12:39:37,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 12:39:37,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 12:39:37,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:39:37,904 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 12:39:37,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:39:37,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:39:37,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:39:37,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:39:37,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:39:38,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:39:38,023 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 12:39:38,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:39:38,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:39:38,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:39:38,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:38,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:39:38,124 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:38,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:38,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:38,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:38,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:38,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:38,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:38,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:39:38,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:38,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:39:38,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:38,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:38,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:38,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:38,418 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:38,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:38,456 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:38,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:39:38,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:38,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:39:38,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:38,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:38,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:38,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:38,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:38,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:38,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:38,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:39:38,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:38,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:39:38,698 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:38,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:38,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:38,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:38,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:38,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:38,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:38,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:39:38,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:38,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:39:38,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:38,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:38,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:38,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:38,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:38,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:38,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:39,010 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:39:39,028 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:39,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:39:39,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:39,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:39,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:39,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:39,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:39,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:39,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:39,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:39:39,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:39,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:39:39,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:39,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:39,270 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:39,288 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:39,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:39,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:39,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:39,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:39:39,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:39,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:39:39,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:39,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:39,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:39,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:39,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:39,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:39,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:39,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:39:39,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:39,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:39:39,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:39,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:39,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:39,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:39,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:39,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:39,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:39,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:39:39,693 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:39,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:39:39,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:39,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:39,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:39,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:39,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:39,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:39,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:39,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:39:39,875 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 12:39:39,878 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:39:39,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:39:40,204 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:39:40,292 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:39:40,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:39:40,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:39:40,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:39:40,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:39:40,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:39:40,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:39:40,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:39:40,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:39:40,457 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 12:39:40,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:39:40,499 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:39:47,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:39:48,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:39:49,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 15999 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:39:49,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:39:51,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:39:51,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:39:51,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:39:51,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:39:51,789 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 12:39:51,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:39:51,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:39:51,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:39:51,954 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 12:39:51,956 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:39:51,957 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:39:52,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:39:53,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:39:53,447 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:39:53,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:39:53,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:39:53,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:39:53,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:39:53,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:39:53,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:39:53,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:39:53,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:39:53,611 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 12:39:53,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:39:53,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:40:00,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:40:00,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:40:01,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:40:02,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:40:02,069 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 12:40:02,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:40:02,086 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:40:02,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:40:02,167 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 12:40:02,167 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:40:02,170 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:40:02,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:40:02,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:40:03,095 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:40:03,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:40:03,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:40:03,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:40:03,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:40:03,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:40:03,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:40:03,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:40:03,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:40:03,227 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 12:40:03,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:40:03,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:40:09,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:40:10,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:40:10,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:40:11,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:40:11,411 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 12:40:11,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:40:11,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:40:11,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:40:11,500 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 12:40:11,502 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:40:11,503 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:40:12,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:40:12,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:40:13,125 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:40:13,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:40:13,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:40:13,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:40:13,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:40:13,341 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:40:13,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:40:13,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:40:13,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:40:13,421 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 12:40:13,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:40:13,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:40:22,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:40:29,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:40:30,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:40:30,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:40:33,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:40:33,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:40:33,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:40:34,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:40:34,055 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 12:40:34,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:40:34,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:40:34,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:40:34,174 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 12:40:34,176 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:40:34,178 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 12:40:34,270 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:40:34,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 12:40:34,572 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:40:34,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 12:40:34,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 12:40:34,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 12:40:34,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 12:40:34,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 12:40:34,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 12:40:34,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 12:40:34,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:40:34,691 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 12:40:34,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:40:34,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:40:34,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:40:34,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:40:34,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:40:34,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:40:34,811 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 12:40:34,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:40:34,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:40:34,841 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:40:34,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:40:34,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:40:34,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:40:34,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:40:34,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:40:34,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:40:34,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:40:35,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:40:35,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:40:35,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:40:35,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:40:35,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:40:35,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:40:35,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:40:35,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:40:35,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:40:35,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:40:35,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:40:35,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:40:35,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:40:35,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:40:35,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:40:35,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:40:35,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:40:35,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:40:35,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:40:35,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:40:35,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:40:35,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:40:35,423 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:40:35,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:40:35,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:40:35,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:40:35,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:40:35,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:40:35,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:40:35,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:40:35,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:40:35,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:40:35,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:40:35,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:40:35,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:40:35,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:40:35,693 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:40:35,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:40:35,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:40:35,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:40:35,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:40:35,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:40:35,808 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:40:35,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:40:35,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:40:35,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:40:35,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:40:35,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:40:35,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:40:35,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:40:35,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:40:35,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:40:35,987 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:40:36,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:40:36,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:40:36,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:40:36,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:40:36,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:40:36,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:40:36,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:40:36,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:40:36,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:40:36,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:40:36,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:40:36,204 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:40:36,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:40:36,239 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:40:36,255 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:40:36,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:40:36,288 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:40:36,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:40:36,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:40:36,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:40:36,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:40:36,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:40:36,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:40:36,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:40:36,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:40:36,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:40:36,459 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:40:36,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:40:36,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:40:36,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:40:36,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:40:36,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:40:36,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:40:36,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:40:36,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:40:36,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:40:36,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:40:36,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:40:36,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:40:36,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:40:36,720 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 12:40:36,720 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:40:36,723 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:40:37,341 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:40:37,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:40:37,743 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:40:37,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:40:37,757 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:40:37,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:40:37,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:40:37,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:40:37,836 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:40:37,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:40:37,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:40:38,428 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:40:38,430 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.70799994469. 2022-02-08 12:40:38,430 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:40:38,430 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:40:39,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:40:39,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:40:39,946 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:40:39,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:40:39,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:40:39,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:40:40,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:40:40,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:40:40,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:40:40,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:40:40,122 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:40:40,775 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:40:40,776 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.34599995613. 2022-02-08 12:40:40,776 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:40:40,779 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:40:41,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:40:42,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:40:42,249 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:40:42,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:40:42,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:40:42,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:40:42,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:40:42,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:40:42,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:40:42,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:40:42,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:40:43,168 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:40:43,168 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.39299988747. 2022-02-08 12:40:43,170 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:40:43,171 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:40:44,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:40:44,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:40:45,157 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:40:45,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:40:45,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:40:45,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:40:45,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:40:45,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:40:45,293 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:40:45,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:40:45,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:40:45,996 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:40:45,996 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.8259999752. 2022-02-08 12:40:45,996 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:40:45,997 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:40:46,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:40:47,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:40:47,651 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:40:47,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:40:47,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:40:47,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:40:47,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:40:47,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:40:47,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:40:47,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:40:47,849 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:40:48,407 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:40:48,407 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.41199994087. 2022-02-08 12:40:48,407 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:40:48,411 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:40:48,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:40:48,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:40:49,055 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:40:49,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:40:49,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:40:49,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:40:49,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:40:49,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:40:49,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:40:49,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:40:49,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:40:49,930 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:40:49,930 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.52200007439. 2022-02-08 12:40:49,930 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:40:49,931 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 5 queries submitted, 0 failed, 16019 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:40:50,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:40:51,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:40:51,286 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:40:51,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:40:51,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:40:51,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:40:51,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:40:51,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:40:51,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:40:51,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:40:51,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:40:51,990 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:40:51,990 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.0609998703. 2022-02-08 12:40:51,990 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:40:51,993 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:40:52,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:40:52,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:40:52,681 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:40:52,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:40:52,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:40:52,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:40:52,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:40:52,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:40:52,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:40:52,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:40:52,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:40:53,506 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:40:53,507 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.51699995995. 2022-02-08 12:40:53,507 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:40:53,509 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:40:54,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:40:54,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:40:55,493 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:40:55,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:40:55,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:40:55,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:40:55,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:40:55,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:40:56,028 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:40:56,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:40:56,255 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:40:57,388 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 12:40:57,388 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 12:40:59,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:40:59,548 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:40:59,548 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.04000020027. ############################################################ # # RAM FootPrint: 3461.3359375 # ############################################################ 2022-02-08 12:40:59,585 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 12:40:59,586 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 76 seconds. Reason: Iteration:6 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 16047 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:42:15,592 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 12:42:15,594 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 12:42:20,595 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:42:21,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:42:22,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:42:22,323 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:42:22,323 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:42:22,349 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:42:22,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:42:22,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:42:22,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:42:22,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:42:22,448 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:42:22,506 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:42:22,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:42:22,578 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['71436 memcached\n'], []) 2022-02-08 12:42:22,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 12:42:22,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:42:22,680 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['71436 memcached\n'], []) 2022-02-08 12:42:22,680 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:42:22,684 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:42:23,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:42:23,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:42:24,101 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:42:24,102 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:42:24,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:42:24,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:42:24,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:42:24,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:42:24,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:42:24,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:42:24,288 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:42:24,288 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:42:24,351 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['127895 memcached\n'], []) 2022-02-08 12:42:24,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 12:42:24,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:42:24,418 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['127895 memcached\n'], []) 2022-02-08 12:42:24,418 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:42:24,421 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:42:25,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:42:26,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:42:26,272 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:42:26,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:42:26,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:42:26,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:42:26,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:42:26,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:42:26,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:42:26,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:42:26,460 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:42:26,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:42:26,533 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['111234 memcached\n'], []) 2022-02-08 12:42:26,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 12:42:26,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:42:26,601 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['111234 memcached\n'], []) 2022-02-08 12:42:26,602 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:42:26,605 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:42:27,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:42:28,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:42:28,302 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:42:28,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:42:28,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:42:28,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:42:28,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:42:28,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:42:28,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:42:28,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:42:28,479 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:42:28,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:42:28,543 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['5454 memcached\n'], []) 2022-02-08 12:42:28,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 12:42:28,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:42:28,601 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['5454 memcached\n'], []) 2022-02-08 12:42:28,602 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:42:28,604 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:42:29,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:42:30,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:42:30,581 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:42:30,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:42:30,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:42:30,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:42:30,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:42:30,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:42:30,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:42:30,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:42:30,750 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:42:30,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:42:30,809 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['129999 memcached\n'], []) 2022-02-08 12:42:30,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 12:42:30,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:42:30,865 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['129999 memcached\n'], []) 2022-02-08 12:42:30,867 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:42:30,868 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:42:31,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:42:32,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:42:32,529 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:42:32,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:42:32,545 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:42:32,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:42:32,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:42:32,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:42:32,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:42:32,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:42:32,691 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:42:32,693 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:42:32,753 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['34469 memcached\n'], []) 2022-02-08 12:42:32,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 12:42:32,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:42:32,806 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['34469 memcached\n'], []) 2022-02-08 12:42:32,808 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:42:32,809 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:42:33,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:42:34,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:42:34,191 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:42:34,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:42:34,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:42:34,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:42:34,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:42:34,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:42:34,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:42:34,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:42:34,326 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:42:34,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:42:34,400 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['98133 memcached\n'], []) 2022-02-08 12:42:34,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 12:42:34,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:42:34,448 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['98133 memcached\n'], []) 2022-02-08 12:42:34,448 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:42:34,450 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:42:35,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:42:35,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:42:35,582 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:42:35,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:42:35,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:42:35,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:42:35,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:42:35,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:42:35,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:42:35,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:42:35,717 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:42:35,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:42:35,776 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['113358 memcached\n'], []) 2022-02-08 12:42:35,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 12:42:35,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:42:35,836 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['113358 memcached\n'], []) 2022-02-08 12:42:35,838 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:42:35,838 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:42:36,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:42:37,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:42:37,338 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:42:37,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:42:37,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:42:37,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:42:37,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:42:37,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:42:37,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:42:37,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:42:37,467 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:42:37,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:42:37,523 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['50357 memcached\n'], []) 2022-02-08 12:42:37,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 12:42:37,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:42:37,571 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['50357 memcached\n'], []) 2022-02-08 12:42:37,572 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:42:37,573 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:42:37,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:42:37,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:42:37,937 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:42:37,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:42:37,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:42:37,964 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:42:37,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:42:38,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:42:38,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:42:38,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:42:38,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:42:38,118 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 12:42:38,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:42:38,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:42:44,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:42:45,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:42:46,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:42:46,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:42:48,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:42:48,308 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:42:48,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:42:48,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:42:48,572 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 12:42:48,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:42:48,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:42:48,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:42:48,678 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 12:42:48,678 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:42:48,681 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:42:48,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:42:48,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:42:49,058 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:42:49,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:42:49,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:42:49,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:42:49,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:42:49,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:42:49,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:42:49,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:42:49,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:42:49,173 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 12:42:49,201 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:42:49,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 16070 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:42:57,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:42:58,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:42:59,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:42:59,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:00,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:43:00,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:43:01,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:43:01,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:43:01,135 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 12:43:01,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:43:01,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:01,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:43:01,240 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 12:43:01,242 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:43:01,243 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 12:43:01,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:43:01,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 12:43:01,635 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:43:01,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 12:43:01,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 12:43:01,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 12:43:01,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 12:43:01,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 12:43:01,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 12:43:01,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 12:43:01,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:43:01,753 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 12:43:01,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:43:01,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:01,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:43:01,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:43:01,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:43:01,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:43:01,871 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 12:43:01,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:43:01,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:01,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:43:01,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:01,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:43:01,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:01,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:02,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:02,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:02,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:02,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:02,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:02,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:02,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:02,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:43:02,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:02,227 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:02,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:02,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:02,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:02,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:02,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:02,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:02,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:02,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:43:02,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:02,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:02,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:02,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:02,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:02,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:02,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:02,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:02,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:02,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:43:02,548 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:02,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:02,584 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:02,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:02,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:02,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:02,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:02,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:02,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:02,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:43:02,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:02,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:02,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:02,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:02,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:02,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:02,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:02,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:02,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:02,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:43:02,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:02,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:02,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:02,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:02,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:02,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:03,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:03,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:03,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:03,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:43:03,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:03,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:03,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:03,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:03,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:03,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:03,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:03,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:03,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:03,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:43:03,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:03,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:03,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:03,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:03,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:03,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:03,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:03,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:03,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:03,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:43:03,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:03,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:03,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:03,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:03,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:03,515 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:03,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:03,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:03,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:03,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:43:03,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:03,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:03,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:03,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:03,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:03,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:03,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:03,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:03,753 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 12:43:03,755 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:43:03,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:43:04,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:43:04,157 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:43:04,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:43:04,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:43:04,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:43:04,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:43:04,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:43:04,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:43:04,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:43:04,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:43:04,477 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 12:43:04,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:43:04,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:10,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:43:12,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:43:13,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:43:13,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:43:13,948 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 12:43:13,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:43:14,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:14,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:43:14,459 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 12:43:14,460 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:43:14,461 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 12:43:14,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:43:14,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 12:43:14,834 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:43:14,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 12:43:14,849 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 12:43:14,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 12:43:14,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 12:43:14,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 12:43:14,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 12:43:14,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 12:43:14,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:43:14,957 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 12:43:14,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:43:14,994 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:15,010 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:43:15,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:43:15,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:43:15,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:43:15,065 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 12:43:15,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:43:15,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:15,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:43:15,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:15,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:43:15,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:15,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:15,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:15,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:15,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:15,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:15,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:15,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:15,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:15,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:43:15,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:15,423 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:15,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:15,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:15,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:15,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:15,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:15,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:15,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:15,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:43:15,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:15,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:15,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:15,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:15,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:15,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:15,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:15,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:15,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:15,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:43:15,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:15,772 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:15,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:15,808 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:15,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:15,841 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:15,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:15,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:15,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:15,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:43:15,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:15,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:15,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:15,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:15,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:16,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:16,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:16,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:16,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:16,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:43:16,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:16,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:16,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:16,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:16,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:16,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:16,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:16,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:16,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:16,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:43:16,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:16,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:16,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:16,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:16,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:16,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:16,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:16,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:16,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:16,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:43:16,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:16,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:16,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:16,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:16,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:16,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:16,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:16,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:16,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:16,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:43:16,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:16,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:16,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:16,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:16,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:16,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:16,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:16,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:16,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:16,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:43:16,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:16,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:16,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:16,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:16,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:16,869 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:16,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:16,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:16,920 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 12:43:16,921 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:43:17,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:43:17,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:43:17,309 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:43:17,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:43:17,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:43:17,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:43:17,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:43:17,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:43:17,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:43:17,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:43:17,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:43:17,437 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 12:43:17,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:43:17,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:23,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:43:24,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:43:25,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:43:25,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:27,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:43:28,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:43:28,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:43:28,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:43:28,423 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 12:43:28,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:43:28,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:28,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:43:28,515 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 12:43:28,516 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:43:28,517 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:43:28,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:43:29,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:43:29,338 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:43:29,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:43:29,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:43:29,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:43:29,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:43:29,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:43:29,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:43:29,477 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:43:29,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:43:29,631 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 12:43:29,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:43:29,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:37,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:43:38,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:43:39,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:43:39,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:41,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:43:41,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:43:41,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:43:41,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:43:41,641 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 12:43:41,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:43:41,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:41,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:43:41,862 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 12:43:41,864 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:43:41,865 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 12:43:41,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:43:42,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 12:43:42,272 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:43:42,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 12:43:42,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 12:43:42,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 12:43:42,323 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 12:43:42,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 12:43:42,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 12:43:42,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 12:43:42,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:43:42,426 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 12:43:42,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:43:42,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:42,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:43:42,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:43:42,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:43:42,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:43:42,575 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 12:43:42,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:43:42,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:42,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:43:42,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:42,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:43:42,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:42,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:42,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:42,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:42,808 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:42,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:42,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:42,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:42,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:42,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:43:42,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:42,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:42,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:43,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:43,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:43,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:43,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:43,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:43,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:43,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:43:43,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:43,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:43,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:43,214 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:43,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:43,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:43,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:43,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:43,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:43,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:43:43,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:43,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:43,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:43,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:43,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:43,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:43,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:43,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:43,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:43,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:43:43,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:43,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:43,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:43,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:43,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:43,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:43,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:43,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:43,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:43,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:43:43,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:43,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:43,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:43,831 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:43,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:43,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:43,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:43,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:43,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:43,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:43:43,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:43,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:44,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:44,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:44,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:44,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:44,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:44,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:44,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:44,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:43:44,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:44,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:44,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:44,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:44,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:44,288 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:44,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:44,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:44,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:44,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:43:44,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:44,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:44,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:44,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:44,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:44,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:44,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:44,528 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:44,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:44,566 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:43:44,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:44,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:44,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:44,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:44,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:44,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:44,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:44,742 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:44,760 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 12:43:44,763 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 12:43:44,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:43:45,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 12:43:45,138 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:43:45,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 12:43:45,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 12:43:45,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 12:43:45,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 12:43:45,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 12:43:45,219 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 12:43:45,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 12:43:45,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:43:45,266 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 12:43:45,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:43:45,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:45,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:43:45,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:43:45,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:43:45,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:43:45,387 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 12:43:45,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:43:45,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:45,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:43:45,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:45,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:43:45,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:45,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:45,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:45,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:45,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:45,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:45,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:45,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:43:45,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:45,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:43:45,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:45,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:45,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:45,772 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:45,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:45,808 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:45,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:45,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:43:45,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:45,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:43:45,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:45,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:45,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:45,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:45,987 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:46,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:46,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:46,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:43:46,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:46,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:43:46,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:46,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:46,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:46,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:46,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:46,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:46,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:46,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:43:46,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:46,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:43:46,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:46,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:46,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:46,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:46,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:46,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:46,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:46,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:43:46,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:46,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:43:46,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:46,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:46,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:46,492 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:46,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:46,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:46,545 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:46,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:43:46,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:46,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:43:46,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:46,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:46,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:46,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:46,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:46,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:46,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:46,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:43:46,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:46,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:43:46,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:46,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:46,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:46,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:46,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:46,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:46,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:46,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:43:46,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:46,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:43:46,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:46,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:46,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:46,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:47,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:47,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:47,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:47,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:43:47,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:47,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:43:47,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:47,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:47,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:47,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:47,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:47,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:47,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:47,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:43:47,250 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 12:43:47,253 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:43:48,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:43:48,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:43:48,742 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:43:48,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:43:48,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:43:48,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:43:48,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:43:48,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:43:48,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:43:48,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:43:48,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:43:48,943 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 12:43:48,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:43:48,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 16095 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:43:56,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:43:57,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:43:58,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:43:58,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:43:59,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:44:00,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:44:00,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:44:00,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:44:00,433 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 12:44:00,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:44:00,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:44:00,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:44:00,563 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 12:44:00,563 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:44:00,565 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:44:00,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:44:00,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:44:01,009 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:44:01,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:44:01,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:44:01,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:44:01,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:44:01,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:44:01,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:44:01,102 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:44:01,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:44:01,134 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 12:44:01,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:44:01,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:44:08,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:44:09,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:44:10,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:44:10,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:44:10,637 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 12:44:10,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:44:10,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:44:10,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:44:10,733 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 12:44:10,733 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:44:10,736 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:44:11,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:44:11,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:44:11,615 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:44:11,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:44:11,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:44:11,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:44:11,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:44:11,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:44:11,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:44:11,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:44:11,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:44:11,750 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 12:44:11,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:44:11,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:44:18,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:44:19,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:44:19,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:44:20,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:44:20,157 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 12:44:20,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:44:20,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:44:20,227 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:44:20,256 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 12:44:20,256 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:44:20,259 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:44:21,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:44:21,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:44:22,108 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:44:22,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:44:22,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:44:22,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:44:22,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:44:22,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:44:22,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:44:22,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:44:22,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:44:22,431 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 12:44:22,456 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:44:22,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:44:31,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:44:35,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:44:36,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:44:36,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:44:39,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:44:39,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:44:39,964 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:44:40,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:44:40,171 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 12:44:40,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:44:40,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:44:40,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:44:40,282 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 12:44:40,282 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:44:40,285 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 12:44:40,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:44:40,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 12:44:40,661 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:44:40,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 12:44:40,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 12:44:40,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 12:44:40,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 12:44:40,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 12:44:40,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 12:44:40,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 12:44:40,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:44:40,782 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 12:44:40,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:44:40,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:44:40,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:44:40,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:44:40,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:44:40,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:44:40,900 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 12:44:40,900 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:44:40,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:44:40,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:44:40,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:44:40,964 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:44:40,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:44:41,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:44:41,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:44:41,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:44:41,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:44:41,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:44:41,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:44:41,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:44:41,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:44:41,234 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:44:41,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:44:41,270 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:44:41,288 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:44:41,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:44:41,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:44:41,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:44:41,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:44:41,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:44:41,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:44:41,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:44:41,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:44:41,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:44:41,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:44:41,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:44:41,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:44:41,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:44:41,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:44:41,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:44:41,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:44:41,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:44:41,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:44:41,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:44:41,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:44:41,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:44:41,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:44:41,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:44:41,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:44:41,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:44:41,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:44:41,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:44:41,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:44:41,788 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:44:41,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:44:41,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:44:41,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:44:41,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:44:41,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:44:41,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:44:41,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:44:41,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:44:41,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:44:41,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:44:41,987 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:44:42,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:44:42,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:44:42,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:44:42,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:44:42,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:44:42,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:44:42,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:44:42,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:44:42,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:44:42,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:44:42,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:44:42,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:44:42,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:44:42,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:44:42,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:44:42,275 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:44:42,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:44:42,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:44:42,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:44:42,346 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:44:42,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:44:42,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:44:42,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:44:42,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:44:42,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:44:42,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:44:42,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:44:42,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:44:42,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:44:42,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:44:42,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:44:42,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:44:42,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:44:42,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:44:42,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:44:42,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:44:42,627 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:44:42,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:44:42,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:44:42,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:44:42,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:44:42,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:44:42,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:44:42,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:44:42,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:44:42,789 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 12:44:42,789 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:44:42,790 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:44:43,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:44:43,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:44:43,826 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:44:43,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:44:43,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:44:43,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:44:43,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:44:43,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:44:43,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:44:43,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:44:43,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:44:44,546 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:44:44,546 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.757999897. 2022-02-08 12:44:44,546 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:44:44,548 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:44:44,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:44:45,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:44:45,375 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:44:45,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:44:45,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:44:45,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:44:45,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:44:45,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:44:45,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:44:45,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:44:45,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:44:46,546 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:44:46,546 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.99900007248. 2022-02-08 12:44:46,546 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:44:46,549 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:44:47,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:44:47,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:44:48,155 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:44:48,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:44:48,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:44:48,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:44:48,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:44:48,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:44:48,280 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:44:48,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:44:48,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:44:48,891 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:44:48,891 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.34400010109. 2022-02-08 12:44:48,891 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:44:48,892 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:44:49,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 16116 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:44:50,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:44:50,680 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:44:50,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:44:50,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:44:50,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:44:50,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:44:50,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:44:50,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:44:50,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:44:50,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:44:51,509 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:44:51,509 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.61899995804. 2022-02-08 12:44:51,510 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:44:51,512 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:44:52,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:44:53,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:44:53,375 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:44:53,377 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:44:53,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:44:53,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:44:53,443 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:44:53,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:44:53,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:44:53,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:44:53,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:44:54,066 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:44:54,066 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.55599999428. 2022-02-08 12:44:54,068 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:44:54,069 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:44:54,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:44:54,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:44:54,838 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:44:54,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:44:54,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:44:54,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:44:54,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:44:54,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:44:54,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:44:54,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:44:55,017 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:44:55,634 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:44:55,634 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.56599998474. 2022-02-08 12:44:55,634 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:44:55,635 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:44:56,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:44:57,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:44:57,230 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:44:57,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:44:57,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:44:57,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:44:57,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:44:57,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:44:57,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:44:57,349 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:44:57,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:44:57,901 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:44:57,901 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.26600003242. 2022-02-08 12:44:57,901 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:44:57,904 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:44:58,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:44:58,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:44:58,940 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:44:58,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:44:58,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:44:58,987 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:44:59,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:44:59,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:44:59,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:44:59,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:44:59,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:44:59,632 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:44:59,632 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.73099994659. 2022-02-08 12:44:59,632 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:44:59,634 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:45:00,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:45:01,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:45:01,825 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:45:01,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:45:01,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:45:01,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:45:01,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:45:02,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:45:02,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:45:02,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:45:02,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:45:02,947 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:45:02,947 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.3140001297. ############################################################ # # RAM FootPrint: 3580.45703125 # ############################################################ 2022-02-08 12:45:02,976 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 12:45:02,977 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 103 seconds. Reason: Iteration:7 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 16141 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:46:45,983 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 12:46:45,983 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 5 queries submitted, 0 failed, 16166 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:46:50,986 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:46:51,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:46:52,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:46:52,845 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:46:52,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:46:52,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:46:52,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:46:52,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:46:53,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:46:53,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:46:53,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:46:53,068 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:46:53,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:46:53,115 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['72166 memcached\n'], []) 2022-02-08 12:46:53,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 12:46:53,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:46:53,176 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['72166 memcached\n'], []) 2022-02-08 12:46:53,177 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:46:53,180 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:46:54,010 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:46:54,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:46:54,644 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:46:54,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:46:54,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:46:54,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:46:54,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:46:54,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:46:54,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:46:54,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:46:54,862 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:46:54,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:46:54,941 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['128656 memcached\n'], []) 2022-02-08 12:46:54,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 12:46:54,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:46:55,010 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['128656 memcached\n'], []) 2022-02-08 12:46:55,010 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:46:55,013 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:46:55,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:46:56,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:46:56,874 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:46:56,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:46:56,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:46:56,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:46:56,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:46:56,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:46:57,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:46:57,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:46:57,089 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:46:57,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:46:57,168 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['112105 memcached\n'], []) 2022-02-08 12:46:57,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 12:46:57,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:46:57,224 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['112105 memcached\n'], []) 2022-02-08 12:46:57,226 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:46:57,229 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:46:57,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:46:58,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:46:58,865 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:46:58,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:46:58,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:46:58,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:46:58,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:46:58,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:46:58,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:46:58,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:46:59,000 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:46:59,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:46:59,081 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['6213 memcached\n'], []) 2022-02-08 12:46:59,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 12:46:59,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:46:59,141 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['6213 memcached\n'], []) 2022-02-08 12:46:59,141 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:46:59,144 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:46:59,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:47:00,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:47:00,667 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:47:00,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:47:00,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:47:00,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:47:00,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:47:00,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:47:00,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:47:00,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:47:00,855 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:47:00,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:47:00,921 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['130761 memcached\n'], []) 2022-02-08 12:47:00,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 12:47:00,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:47:01,005 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['130761 memcached\n'], []) 2022-02-08 12:47:01,005 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:47:01,007 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:47:02,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:47:02,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:47:02,986 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:47:02,987 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:47:03,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:47:03,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:47:03,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:47:03,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:47:03,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:47:03,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:47:03,131 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:47:03,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:47:03,200 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['35240 memcached\n'], []) 2022-02-08 12:47:03,201 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 12:47:03,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:47:03,255 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['35240 memcached\n'], []) 2022-02-08 12:47:03,255 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:47:03,257 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:47:04,181 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:47:04,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:47:04,983 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:47:04,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:47:04,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:47:05,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:47:05,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:47:05,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:47:05,086 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:47:05,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:47:05,153 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:47:05,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:47:05,198 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['98872 memcached\n'], []) 2022-02-08 12:47:05,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 12:47:05,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:47:05,252 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['98872 memcached\n'], []) 2022-02-08 12:47:05,252 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:47:05,253 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:47:06,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:47:06,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:47:06,533 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:47:06,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:47:06,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:47:06,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:47:06,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:47:06,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:47:06,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:47:06,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:47:06,658 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:47:06,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:47:06,719 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['114091 memcached\n'], []) 2022-02-08 12:47:06,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 12:47:06,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:47:06,769 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['114091 memcached\n'], []) 2022-02-08 12:47:06,770 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:47:06,773 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:47:07,489 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:47:08,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:47:08,286 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:47:08,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:47:08,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:47:08,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:47:08,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:47:08,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:47:08,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:47:08,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:47:08,473 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:47:08,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:47:08,543 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['51162 memcached\n'], []) 2022-02-08 12:47:08,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 12:47:08,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:47:08,599 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['51162 memcached\n'], []) 2022-02-08 12:47:08,599 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:47:08,602 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:47:08,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:47:08,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:47:08,970 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:47:08,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:47:08,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:47:09,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:47:09,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:47:09,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:47:09,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:47:09,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:47:09,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:47:09,089 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 12:47:09,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:47:09,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:47:15,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:47:16,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:47:17,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:47:17,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:47:19,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:47:19,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:47:19,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:47:19,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:47:19,941 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 12:47:19,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:47:19,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:47:20,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:47:20,039 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 12:47:20,040 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:47:20,043 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:47:20,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:47:20,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:47:20,398 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:47:20,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:47:20,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:47:20,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:47:20,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:47:20,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:47:20,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:47:20,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:47:20,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:47:20,513 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 12:47:20,545 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:47:20,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:47:28,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:47:28,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:47:29,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:47:29,890 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:47:31,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:47:31,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:47:31,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:47:31,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:47:31,469 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 12:47:31,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:47:31,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:47:31,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:47:31,565 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 12:47:31,565 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:47:31,568 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 12:47:31,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:47:31,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 12:47:31,967 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:47:31,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 12:47:31,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 12:47:31,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 12:47:32,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 12:47:32,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 12:47:32,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 12:47:32,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 12:47:32,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:47:32,098 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 12:47:32,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:47:32,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:47:32,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:47:32,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:47:32,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:47:32,204 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:47:32,220 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 12:47:32,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:47:32,234 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:47:32,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:47:32,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:32,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:47:32,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:32,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:32,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:32,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:32,443 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:32,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:32,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:32,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:32,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:32,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:47:32,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:32,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:32,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:32,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:32,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:32,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:32,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:32,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:32,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:32,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:47:32,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:32,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:32,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:32,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:32,836 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:32,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:32,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:32,890 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:32,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:32,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:47:32,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:32,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:32,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:33,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:33,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:33,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:33,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:33,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:33,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:33,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:47:33,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:33,150 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:33,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:33,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:33,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:33,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:33,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:33,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:33,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:33,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:47:33,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:33,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:33,349 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:33,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:33,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:33,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:33,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:33,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:33,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:33,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:47:33,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:33,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:33,528 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:33,545 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:33,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:33,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:33,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:33,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:33,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:33,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:47:33,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:33,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:33,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:33,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:33,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:33,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:33,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:33,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:33,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:33,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:47:33,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:33,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:33,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:33,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:33,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:33,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:33,918 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:33,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:33,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:33,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:47:33,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:34,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:34,020 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:34,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:34,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:34,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:34,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:34,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:34,127 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 12:47:34,131 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:47:34,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:47:34,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:47:34,940 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:47:34,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:47:34,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:47:35,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:47:35,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:47:35,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:47:35,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:47:35,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:47:35,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:47:35,219 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 12:47:35,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:47:35,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:47:42,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:47:43,275 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:47:43,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:47:44,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:47:44,453 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 12:47:44,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:47:44,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:47:44,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:47:44,556 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 12:47:44,558 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:47:44,561 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 12:47:44,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:47:44,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 12:47:44,931 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:47:44,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 12:47:44,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 12:47:44,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 12:47:44,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 12:47:44,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 12:47:45,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 12:47:45,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 12:47:45,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:47:45,052 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 12:47:45,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:47:45,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:47:45,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:47:45,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:47:45,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:47:45,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:47:45,165 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 12:47:45,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:47:45,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:47:45,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:47:45,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:45,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:47:45,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:45,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:45,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:45,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:45,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:45,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:45,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:45,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:47:45,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:45,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:47:45,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:45,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:45,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:45,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:45,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:45,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:45,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:45,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:47:45,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:45,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:47:45,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:45,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:45,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:45,757 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:45,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:45,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:45,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:45,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:47:45,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:45,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:47:45,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:45,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:45,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:45,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:45,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:45,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:45,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:45,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:47:46,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:46,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:47:46,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:46,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:46,084 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:46,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:46,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:46,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:46,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:46,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:47:46,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:46,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:47:46,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:46,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:46,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:46,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:46,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:46,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:46,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:46,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:47:46,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:46,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:47:46,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:46,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:46,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:46,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:46,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:46,492 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:46,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:46,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:47:46,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:46,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:47:46,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:46,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:46,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:46,624 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:46,640 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:46,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:46,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:46,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:47:46,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:46,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:47:46,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:46,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:46,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:46,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:46,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:46,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:46,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:46,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:47:46,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:46,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:47:46,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:46,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:46,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:46,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:46,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:46,994 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:47,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:47,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:47:47,048 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 12:47:47,051 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:47:47,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:47:47,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:47:47,437 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:47:47,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:47:47,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:47:47,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:47:47,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:47:47,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:47:47,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:47:47,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:47:47,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:47:47,565 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 12:47:47,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:47:47,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 16192 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:47:54,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:47:55,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:47:56,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:47:57,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:47:59,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:47:59,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:47:59,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:48:00,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:48:00,042 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 12:48:00,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:48:00,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:48:00,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:48:00,134 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 12:48:00,134 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:48:00,137 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:48:00,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:48:00,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:48:00,585 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:48:00,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:48:00,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:48:00,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:48:00,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:48:00,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:48:00,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:48:00,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:48:00,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:48:00,743 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 12:48:00,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:48:00,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:48:08,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:48:10,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:48:10,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:48:11,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:48:12,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:48:12,987 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:48:13,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:48:13,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:48:13,446 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 12:48:13,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:48:13,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:48:13,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:48:13,618 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 12:48:13,618 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:48:13,621 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 12:48:13,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:48:13,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 12:48:14,023 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:48:14,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 12:48:14,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 12:48:14,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 12:48:14,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 12:48:14,094 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 12:48:14,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 12:48:14,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 12:48:14,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:48:14,167 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 12:48:14,227 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:48:14,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:48:14,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:48:14,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:48:14,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:48:14,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:48:14,332 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 12:48:14,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:48:14,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:48:14,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:48:14,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:14,418 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:48:14,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:14,489 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:14,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:14,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:14,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:14,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:14,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:14,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:14,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:14,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:48:14,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:14,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:14,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:14,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:14,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:14,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:14,897 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:14,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:14,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:14,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:48:14,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:14,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:15,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:15,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:15,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:15,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:15,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:15,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:15,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:15,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:48:15,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:15,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:15,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:15,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:15,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:15,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:15,308 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:15,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:15,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:15,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:48:15,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:15,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:15,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:15,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:15,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:15,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:15,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:15,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:15,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:15,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:48:15,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:15,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:15,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:15,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:15,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:15,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:15,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:15,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:15,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:15,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:48:15,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:15,823 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:15,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:15,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:15,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:15,900 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:15,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:15,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:15,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:15,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:48:16,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:16,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:16,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:16,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:16,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:16,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:16,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:16,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:16,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:16,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:48:16,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:16,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:16,234 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:16,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:16,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:16,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:16,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:16,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:16,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:16,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:48:16,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:16,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:16,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:16,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:16,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:16,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:16,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:16,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:16,572 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 12:48:16,575 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 12:48:16,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:48:16,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 12:48:16,976 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:48:16,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 12:48:16,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 12:48:17,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 12:48:17,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 12:48:17,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 12:48:17,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 12:48:17,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 12:48:17,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:48:17,101 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 12:48:17,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:48:17,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:48:17,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:48:17,181 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:48:17,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:48:17,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:48:17,226 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 12:48:17,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:48:17,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:48:17,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:48:17,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:17,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:48:17,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:17,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:17,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:17,418 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:17,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:17,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:17,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:17,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:48:17,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:17,584 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:48:17,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:17,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:17,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:17,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:17,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:17,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:17,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:17,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:48:17,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:17,757 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:48:17,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:17,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:17,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:17,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:17,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:17,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:17,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:17,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:48:17,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:17,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:48:17,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:17,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:17,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:18,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:18,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:18,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:18,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:18,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:48:18,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:18,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:48:18,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:18,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:18,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:18,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:18,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:18,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:18,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:18,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:48:18,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:18,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:48:18,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:18,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:18,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:18,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:18,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:18,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:18,423 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:18,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:48:18,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:18,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:48:18,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:18,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:18,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:18,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:18,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:18,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:18,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:18,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:48:18,627 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:18,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:48:18,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:18,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:18,693 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:18,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:18,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:18,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:18,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:18,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:48:18,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:18,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:48:18,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:18,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:18,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:18,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:18,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:18,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:18,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:18,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:48:18,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:18,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:48:18,994 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:19,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:19,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:19,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:19,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:19,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:19,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:19,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:48:19,138 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 12:48:19,140 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:48:20,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:48:20,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:48:20,677 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:48:20,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:48:20,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:48:20,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:48:20,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:48:20,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:48:20,836 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:48:20,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:48:20,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:48:20,928 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 12:48:20,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:48:20,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:48:28,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:48:29,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:48:30,028 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:48:30,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:48:31,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:48:32,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:48:32,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:48:32,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:48:32,299 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 12:48:32,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:48:32,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:48:32,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:48:32,474 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 12:48:32,476 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:48:32,479 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:48:32,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:48:33,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:48:33,217 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:48:33,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:48:33,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:48:33,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:48:33,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:48:33,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:48:33,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:48:33,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:48:33,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:48:33,479 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 12:48:33,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:48:33,579 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:48:41,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:48:42,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:48:42,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:48:43,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:48:43,427 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 12:48:43,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:48:43,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:48:43,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:48:43,530 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 12:48:43,532 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:48:43,535 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:48:44,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:48:44,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:48:44,737 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:48:44,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:48:44,760 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:48:44,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:48:44,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:48:44,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:48:44,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:48:44,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:48:44,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:48:44,895 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 12:48:44,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:48:44,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 16217 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:48:51,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:48:52,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:48:52,520 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:48:52,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:48:52,947 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 12:48:52,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:48:52,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:48:53,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:48:53,026 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 12:48:53,026 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:48:53,029 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:48:53,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:48:53,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:48:54,036 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:48:54,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:48:54,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:48:54,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:48:54,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:48:54,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:48:54,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:48:54,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:48:54,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:48:54,897 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 12:48:54,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:48:54,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:49:03,270 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:49:04,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:49:05,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:49:05,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:49:09,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:49:09,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:49:09,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:49:09,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:49:10,068 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 12:49:10,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:49:10,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:49:10,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:49:10,186 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 12:49:10,187 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:49:10,191 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 12:49:10,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:49:10,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 12:49:10,578 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:49:10,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 12:49:10,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 12:49:10,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 12:49:10,624 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 12:49:10,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 12:49:10,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 12:49:10,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 12:49:10,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:49:10,701 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 12:49:10,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:49:10,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:49:10,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:49:10,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:49:10,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:49:10,808 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:49:10,821 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 12:49:10,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:49:10,836 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:49:10,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:49:10,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:49:10,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:49:10,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:49:10,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:49:11,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:49:11,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:49:11,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:49:11,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:49:11,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:49:11,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:49:11,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:49:11,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:49:11,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:49:11,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:49:11,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:49:11,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:49:11,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:49:11,303 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:49:11,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:49:11,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:49:11,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:49:11,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:49:11,392 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:49:11,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:49:11,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:49:11,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:49:11,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:49:11,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:49:11,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:49:11,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:49:11,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:49:11,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:49:11,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:49:11,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:49:11,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:49:11,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:49:11,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:49:11,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:49:11,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:49:11,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:49:11,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:49:11,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:49:11,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:49:11,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:49:11,788 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:49:11,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:49:11,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:49:11,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:49:11,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:49:11,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:49:11,915 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:49:11,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:49:11,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:49:11,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:49:11,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:49:12,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:49:12,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:49:12,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:49:12,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:49:12,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:49:12,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:49:12,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:49:12,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:49:12,150 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:49:12,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:49:12,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:49:12,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:49:12,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:49:12,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:49:12,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:49:12,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:49:12,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:49:12,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:49:12,323 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:49:12,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:49:12,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:49:12,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:49:12,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:49:12,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:49:12,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:49:12,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:49:12,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:49:12,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:49:12,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:49:12,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:49:12,520 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:49:12,538 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:49:12,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:49:12,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:49:12,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:49:12,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:49:12,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:49:12,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:49:12,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:49:12,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:49:12,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:49:12,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:49:12,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:49:12,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:49:12,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:49:12,788 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 12:49:12,789 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:49:12,790 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:49:13,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:49:13,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:49:13,910 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:49:13,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:49:13,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:49:13,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:49:13,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:49:13,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:49:13,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:49:14,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:49:14,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:49:14,711 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:49:14,713 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.9240000248. 2022-02-08 12:49:14,713 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:49:14,714 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:49:15,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:49:15,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:49:15,575 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:49:15,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:49:15,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:49:15,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:49:15,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:49:15,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:49:15,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:49:15,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:49:15,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:49:16,354 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:49:16,354 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.64100003242. 2022-02-08 12:49:16,354 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:49:16,355 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:49:17,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:49:17,849 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:49:17,983 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:49:17,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:49:18,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:49:18,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:49:18,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:49:18,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:49:18,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:49:18,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:49:18,209 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:49:18,782 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:49:18,783 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.42999982834. 2022-02-08 12:49:18,783 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:49:18,786 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:49:19,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:49:20,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:49:20,400 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:49:20,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:49:20,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:49:20,443 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:49:20,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:49:20,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:49:20,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:49:20,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:49:20,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:49:21,217 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:49:21,217 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.43400001526. 2022-02-08 12:49:21,217 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:49:21,220 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:49:22,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:49:22,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:49:22,805 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:49:22,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:49:22,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:49:22,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:49:22,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:49:22,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:49:22,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:49:22,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:49:22,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:49:23,635 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:49:23,635 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.41799998283. 2022-02-08 12:49:23,635 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:49:23,638 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:49:24,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:49:24,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:49:24,687 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:49:24,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:49:24,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:49:24,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:49:24,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:49:24,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:49:24,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:49:24,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:49:24,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:49:25,565 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:49:25,565 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.93000006676. 2022-02-08 12:49:25,565 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:49:25,566 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:49:26,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:49:26,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:49:27,032 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:49:27,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:49:27,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:49:27,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:49:27,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:49:27,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:49:27,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:49:27,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:49:27,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:49:27,693 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:49:27,694 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.12800002098. 2022-02-08 12:49:27,694 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:49:27,696 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:49:28,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:49:28,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:49:29,020 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:49:29,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:49:29,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:49:29,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:49:29,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:49:29,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:49:29,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:49:29,204 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:49:29,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:49:29,835 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:49:29,835 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.14100003242. 2022-02-08 12:49:29,835 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:49:29,836 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:49:31,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:49:31,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:49:31,788 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:49:31,788 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:49:31,808 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:49:31,831 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:49:31,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:49:31,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:49:31,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:49:31,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:49:31,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:49:32,642 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:49:32,642 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.80799984932. ############################################################ # # RAM FootPrint: 3364.5390625 # ############################################################ 2022-02-08 12:49:32,671 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 12:49:32,673 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 111 seconds. Reason: Iteration:8 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 16243 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 16267 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:51:23,687 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 12:51:23,688 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 12:51:28,697 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:51:29,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:51:30,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:51:30,381 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:51:30,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:51:30,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:51:30,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:51:30,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:51:30,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:51:30,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:51:30,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:51:30,655 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:51:30,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:51:30,778 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['72918 memcached\n'], []) 2022-02-08 12:51:30,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 12:51:30,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:51:30,838 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['72918 memcached\n'], []) 2022-02-08 12:51:30,838 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:51:30,845 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:51:31,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:51:32,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:51:32,528 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:51:32,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:51:32,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:51:32,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:51:32,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:51:32,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:51:32,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:51:32,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:51:32,743 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:51:32,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:51:32,891 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['129484 memcached\n'], []) 2022-02-08 12:51:32,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 12:51:32,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:51:32,959 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['129484 memcached\n'], []) 2022-02-08 12:51:32,960 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:51:32,961 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:51:33,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:51:34,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:51:34,740 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:51:34,742 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:51:34,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:51:34,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:51:34,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:51:34,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:51:34,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:51:34,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:51:34,951 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:51:34,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:51:35,039 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['113036 memcached\n'], []) 2022-02-08 12:51:35,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 12:51:35,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:51:35,105 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['113036 memcached\n'], []) 2022-02-08 12:51:35,107 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:51:35,111 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:51:35,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:51:36,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:51:36,654 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:51:36,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:51:36,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:51:36,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:51:36,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:51:36,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:51:36,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:51:36,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:51:36,852 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:51:36,854 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:51:36,924 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['7024 memcached\n'], []) 2022-02-08 12:51:36,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 12:51:36,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:51:36,996 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['7024 memcached\n'], []) 2022-02-08 12:51:36,997 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:51:37,003 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:51:38,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:51:38,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:51:38,756 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:51:38,757 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:51:38,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:51:38,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:51:38,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:51:38,869 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:51:38,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:51:38,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:51:38,957 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:51:38,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:51:39,016 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['878 memcached\n'], []) 2022-02-08 12:51:39,017 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 12:51:39,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:51:39,072 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['878 memcached\n'], []) 2022-02-08 12:51:39,073 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:51:39,078 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:51:40,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:51:41,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:51:41,377 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:51:41,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:51:41,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:51:41,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:51:41,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:51:41,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:51:41,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:51:41,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:51:41,625 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:51:41,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:51:41,693 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['36055 memcached\n'], []) 2022-02-08 12:51:41,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 12:51:41,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:51:41,767 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['36055 memcached\n'], []) 2022-02-08 12:51:41,769 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:51:41,776 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:51:42,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:51:43,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:51:43,358 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:51:43,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:51:43,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:51:43,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:51:43,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:51:43,459 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:51:43,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:51:43,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:51:43,542 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:51:43,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:51:43,605 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['99627 memcached\n'], []) 2022-02-08 12:51:43,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 12:51:43,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:51:43,655 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['99627 memcached\n'], []) 2022-02-08 12:51:43,655 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:51:43,658 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:51:44,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:51:44,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:51:44,924 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:51:44,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:51:44,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:51:44,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:51:45,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:51:45,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:51:45,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:51:45,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:51:45,099 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:51:45,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:51:45,161 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['114874 memcached\n'], []) 2022-02-08 12:51:45,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 12:51:45,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:51:45,226 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['114874 memcached\n'], []) 2022-02-08 12:51:45,227 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:51:45,232 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:51:45,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:51:46,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:51:46,515 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:51:46,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:51:46,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:51:46,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:51:46,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:51:46,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:51:46,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:51:46,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:51:46,664 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:51:46,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:51:46,733 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['51944 memcached\n'], []) 2022-02-08 12:51:46,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 12:51:46,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:51:46,792 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['51944 memcached\n'], []) 2022-02-08 12:51:46,792 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:51:46,795 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:51:46,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:51:47,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:51:47,213 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:51:47,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:51:47,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:51:47,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:51:47,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:51:47,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:51:47,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:51:47,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:51:47,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:51:47,352 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 12:51:47,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:51:47,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 16291 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:51:54,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:51:55,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:51:55,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:51:56,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:51:58,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:51:58,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:51:58,456 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:51:58,579 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:51:58,595 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 12:51:58,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:51:58,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:51:58,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:51:58,700 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 12:51:58,700 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:51:58,703 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:51:58,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:51:59,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:51:59,075 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:51:59,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:51:59,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:51:59,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:51:59,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:51:59,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:51:59,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:51:59,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:51:59,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:51:59,217 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 12:51:59,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:51:59,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:06,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:52:07,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:52:08,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:52:08,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:09,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:52:10,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:52:10,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:52:10,316 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:52:10,332 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 12:52:10,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:52:10,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:10,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:52:10,447 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 12:52:10,448 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:52:10,454 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 12:52:10,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:52:10,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 12:52:10,865 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:52:10,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 12:52:10,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 12:52:10,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 12:52:10,918 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 12:52:10,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 12:52:10,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 12:52:10,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 12:52:10,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:52:11,010 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 12:52:11,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:52:11,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:11,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:52:11,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:52:11,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:52:11,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:52:11,151 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 12:52:11,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:52:11,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:11,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:52:11,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:11,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:52:11,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:11,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:11,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:11,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:11,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:11,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:11,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:11,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:11,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:11,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:52:11,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:11,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:11,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:11,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:11,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:11,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:11,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:11,698 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:11,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:11,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:52:11,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:11,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:11,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:11,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:11,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:11,831 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:11,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:11,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:11,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:11,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:52:11,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:11,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:11,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:11,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:12,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:12,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:12,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:12,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:12,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:12,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:52:12,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:12,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:12,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:12,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:12,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:12,219 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:12,239 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:12,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:12,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:12,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:52:12,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:12,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:12,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:12,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:12,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:12,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:12,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:12,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:12,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:12,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:52:12,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:12,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:12,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:12,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:12,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:12,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:12,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:12,757 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:12,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:12,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:52:12,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:12,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:12,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:12,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:12,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:12,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:12,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:12,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:12,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:12,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:52:13,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:13,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:13,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:13,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:13,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:13,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:13,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:13,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:13,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:13,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:52:13,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:13,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:13,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:13,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:13,270 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:13,288 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:13,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:13,323 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:13,342 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 12:52:13,348 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:52:13,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:52:13,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:52:13,855 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:52:13,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:52:13,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:52:13,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:52:13,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:52:13,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:52:13,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:52:13,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:52:14,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:52:14,026 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 12:52:14,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:52:14,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:20,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:52:21,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:52:22,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:52:22,772 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:52:22,913 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 12:52:22,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:52:23,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:23,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:52:23,490 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 12:52:23,492 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:52:23,494 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 12:52:23,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:52:23,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 12:52:23,872 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:52:23,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 12:52:23,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 12:52:23,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 12:52:23,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 12:52:23,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 12:52:23,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 12:52:23,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 12:52:23,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:52:24,016 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 12:52:24,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:52:24,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:24,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:52:24,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:52:24,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:52:24,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:52:24,151 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 12:52:24,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:52:24,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:24,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:52:24,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:24,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:52:24,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:24,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:24,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:24,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:24,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:24,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:24,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:24,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:24,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:24,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:52:24,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:24,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:24,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:24,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:24,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:24,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:24,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:24,701 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:24,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:24,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:52:24,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:24,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:24,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:24,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:24,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:24,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:24,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:24,890 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:24,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:24,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:52:24,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:24,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:24,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:25,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:25,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:25,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:25,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:25,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:25,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:25,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:52:25,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:25,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:25,183 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:25,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:25,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:25,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:25,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:25,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:25,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:25,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:52:25,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:25,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:25,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:25,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:25,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:25,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:25,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:25,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:25,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:25,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:52:25,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:25,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:25,579 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:25,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:25,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:25,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:25,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:25,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:25,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:25,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:52:25,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:25,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:25,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:25,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:25,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:25,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:25,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:25,854 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:25,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:25,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:52:25,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:25,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:25,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:25,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:25,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:26,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:26,020 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:26,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:26,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:26,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:52:26,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:26,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:26,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:26,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:26,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:26,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:26,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:26,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:26,266 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 12:52:26,269 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:52:26,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:52:26,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:52:26,798 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:52:26,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:52:26,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:52:26,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:52:26,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:52:26,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:52:26,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:52:26,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:52:26,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:52:26,953 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 12:52:26,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:52:27,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:33,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:52:34,392 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:52:34,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:52:35,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:37,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:52:37,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:52:37,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:52:37,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:52:37,921 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 12:52:37,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:52:37,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:37,994 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:52:38,009 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 12:52:38,009 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:52:38,010 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:52:38,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:52:38,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:52:38,430 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:52:38,431 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:52:38,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:52:38,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:52:38,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:52:38,515 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:52:38,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:52:38,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:52:38,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:52:38,592 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 12:52:38,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:52:38,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:46,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:52:47,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:52:48,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:52:48,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 16316 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:52:50,186 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:52:50,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:52:50,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:52:50,627 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:52:50,650 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 12:52:50,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:52:50,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:50,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:52:50,801 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 12:52:50,802 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:52:50,805 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 12:52:50,890 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:52:51,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 12:52:51,210 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:52:51,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 12:52:51,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 12:52:51,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 12:52:51,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 12:52:51,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 12:52:51,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 12:52:51,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 12:52:51,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:52:51,358 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 12:52:51,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:52:51,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:51,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:52:51,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:52:51,489 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:52:51,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:52:51,523 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 12:52:51,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:52:51,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:51,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:52:51,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:51,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:52:51,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:51,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:51,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:51,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:51,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:51,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:51,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:51,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:51,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:51,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:52:51,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:51,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:52,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:52,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:52,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:52,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:52,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:52,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:52,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:52,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:52:52,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:52,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:52,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:52,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:52,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:52,308 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:52,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:52,349 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:52,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:52,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:52:52,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:52,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:52,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:52,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:52,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:52,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:52,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:52,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:52,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:52,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:52:52,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:52,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:52,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:52,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:52,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:52,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:52,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:52,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:52,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:52,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:52:52,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:52,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:52,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:52,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:52,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:52,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:52,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:52,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:53,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:53,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:52:53,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:53,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:53,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:53,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:53,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:53,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:53,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:53,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:53,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:53,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:52:53,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:53,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:53,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:53,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:53,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:53,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:53,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:53,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:53,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:53,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:52:53,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:53,515 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:53,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:53,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:53,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:53,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:53,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:53,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:53,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:53,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:52:53,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:53,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:53,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:53,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:53,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:53,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:53,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:53,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:53,880 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 12:52:53,882 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 12:52:53,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:52:54,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 12:52:54,285 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:52:54,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 12:52:54,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 12:52:54,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 12:52:54,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 12:52:54,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 12:52:54,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 12:52:54,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 12:52:54,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:52:54,423 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 12:52:54,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:52:54,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:54,499 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:52:54,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:52:54,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:52:54,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:52:54,680 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 12:52:54,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:52:54,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:52:54,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:52:54,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:54,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:52:54,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:54,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:54,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:54,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:54,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:54,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:55,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:55,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:52:55,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:55,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:52:55,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:55,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:55,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:55,181 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:55,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:55,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:55,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:55,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:52:55,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:55,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:52:55,316 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:55,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:55,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:55,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:55,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:55,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:55,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:55,443 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:52:55,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:55,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:52:55,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:55,528 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:55,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:55,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:55,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:55,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:55,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:55,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:52:55,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:55,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:52:55,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:55,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:55,742 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:55,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:55,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:55,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:55,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:55,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:52:55,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:55,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:52:55,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:55,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:55,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:55,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:55,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:56,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:56,020 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:56,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:52:56,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:56,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:52:56,094 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:56,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:56,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:56,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:56,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:56,181 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:56,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:56,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:52:56,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:56,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:52:56,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:56,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:56,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:56,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:56,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:56,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:56,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:56,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:52:56,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:56,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:52:56,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:56,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:56,448 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:56,464 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:56,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:56,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:56,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:56,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:52:56,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:56,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:52:56,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:56,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:56,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:56,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:56,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:56,693 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:56,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:56,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:52:56,753 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 12:52:56,757 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:52:56,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:52:57,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:52:57,193 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:52:57,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:52:57,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:52:57,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:52:57,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:52:57,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:52:57,293 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:52:57,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:52:57,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:52:57,355 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 12:52:57,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:52:57,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:53:04,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:53:05,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:53:06,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:53:07,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:53:08,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:53:08,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:53:08,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:53:09,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:53:09,187 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 12:53:09,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:53:09,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:53:09,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:53:09,344 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 12:53:09,345 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:53:09,351 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:53:10,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:53:10,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:53:11,058 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:53:11,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:53:11,084 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:53:11,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:53:11,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:53:11,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:53:11,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:53:11,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:53:11,219 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:53:11,243 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 12:53:11,288 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:53:11,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:53:18,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:53:19,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:53:19,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:53:19,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:53:20,028 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 12:53:20,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:53:20,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:53:20,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:53:20,151 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 12:53:20,153 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:53:20,157 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:53:21,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:53:21,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:53:21,776 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:53:21,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:53:21,793 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:53:21,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:53:21,841 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:53:21,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:53:21,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:53:21,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:53:21,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:53:21,980 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 12:53:22,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:53:22,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:53:28,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:53:29,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:53:29,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:53:30,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:53:30,256 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 12:53:30,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:53:30,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:53:30,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:53:30,348 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 12:53:30,349 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:53:30,354 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:53:31,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:53:31,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:53:32,071 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:53:32,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:53:32,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:53:32,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:53:32,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:53:32,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:53:32,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:53:32,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:53:32,275 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:53:32,325 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 12:53:32,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:53:32,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:53:40,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:53:41,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:53:42,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:53:43,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:53:45,915 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:53:46,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:53:46,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:53:46,640 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:53:46,709 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 12:53:46,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:53:46,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:53:46,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:53:46,832 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 12:53:46,832 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:53:46,835 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 12:53:46,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:53:47,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 12:53:47,232 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:53:47,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 12:53:47,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 12:53:47,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 12:53:47,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 12:53:47,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 12:53:47,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 12:53:47,341 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 12:53:47,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:53:47,374 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 12:53:47,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:53:47,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:53:47,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:53:47,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:53:47,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:53:47,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:53:47,515 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 12:53:47,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:53:47,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:53:47,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:53:47,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:53:47,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:53:47,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:53:47,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:53:47,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:53:47,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:53:47,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:53:47,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:53:47,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:53:47,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:53:47,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:53:47,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:53:47,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:53:47,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:53:47,999 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:53:48,017 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:53:48,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:53:48,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:53:48,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:53:48,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:53:48,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:53:48,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:53:48,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:53:48,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:53:48,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:53:48,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:53:48,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:53:48,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:53:48,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:53:48,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:53:48,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:53:48,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:53:48,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:53:48,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:53:48,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:53:48,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:53:48,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:53:48,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:53:48,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:53:48,520 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:53:48,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:53:48,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:53:48,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:53:48,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:53:48,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:53:48,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:53:48,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:53:48,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:53:48,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:53:48,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:53:48,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:53:48,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:53:48,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:53:48,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:53:48,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:53:48,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:53:48,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:53:48,900 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:53:48,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:53:48,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:53:48,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:53:48,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:53:48,999 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:53:49,017 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:53:49,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:53:49,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:53:49,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:53:49,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:53:49,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:53:49,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:53:49,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:53:49,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:53:49,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:53:49,214 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:53:49,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:53:49,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:53:49,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:53:49,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:53:49,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:53:49,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:53:49,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:53:49,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:53:49,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:53:49,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:53:49,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:53:49,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:53:49,459 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:53:49,477 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:53:49,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:53:49,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:53:49,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:53:49,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:53:49,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:53:49,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:53:49,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:53:49,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:53:49,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:53:49,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:53:49,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:53:49,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:53:49,760 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 12:53:49,763 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:53:49,770 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 5 queries submitted, 0 failed, 16341 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:53:50,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:53:50,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:53:50,816 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:53:50,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:53:50,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:53:50,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:53:50,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:53:50,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:53:50,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:53:50,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:53:50,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:53:51,565 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:53:51,565 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.80100011826. 2022-02-08 12:53:51,565 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:53:51,569 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:53:52,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:53:52,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:53:53,148 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:53:53,150 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:53:53,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:53:53,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:53:53,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:53:53,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:53:53,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:53:53,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:53:53,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:53:53,928 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:53:53,930 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.36500000954. 2022-02-08 12:53:53,930 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:53:53,933 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:53:54,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:53:54,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:53:55,079 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:53:55,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:53:55,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:53:55,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:53:55,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:53:55,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:53:55,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:53:55,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:53:55,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:53:56,404 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:53:56,405 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.47599983215. 2022-02-08 12:53:56,407 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:53:56,411 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:53:57,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:53:57,915 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:53:58,082 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:53:58,084 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:53:58,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:53:58,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:53:58,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:53:58,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:53:58,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:53:58,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:53:58,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:53:58,921 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:53:58,921 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.51499986649. 2022-02-08 12:53:58,921 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:53:58,924 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:53:59,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:54:00,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:54:00,572 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:54:00,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:54:00,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:54:00,627 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:54:00,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:54:00,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:54:00,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:54:00,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:54:00,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:54:01,312 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:54:01,313 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.39199995995. 2022-02-08 12:54:01,313 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:54:01,315 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:54:02,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:54:02,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:54:02,681 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:54:02,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:54:02,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:54:02,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:54:02,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:54:02,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:54:02,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:54:02,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:54:02,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:54:03,447 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:54:03,448 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.13499999046. 2022-02-08 12:54:03,450 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:54:03,453 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:54:04,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:54:05,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:54:05,651 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:54:05,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:54:05,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:54:05,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:54:05,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:54:05,742 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:54:05,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:54:05,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:54:05,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:54:06,706 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:54:06,707 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.257999897. 2022-02-08 12:54:06,707 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:54:06,711 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:54:07,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:54:08,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:54:08,618 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:54:08,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:54:08,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:54:08,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:54:08,698 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:54:08,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:54:08,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:54:08,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:54:08,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:54:10,335 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:54:10,335 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.62800002098. 2022-02-08 12:54:10,335 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:54:10,338 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:54:11,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:54:11,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:54:12,108 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:54:12,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:54:12,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:54:12,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:54:12,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:54:12,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:54:12,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:54:12,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:54:12,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:54:13,335 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:54:13,335 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.99900007248. ############################################################ # # RAM FootPrint: 4082.64453125 # ############################################################ 2022-02-08 12:54:13,377 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 12:54:13,377 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 74 seconds. Reason: Iteration:9 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 16361 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:55:27,451 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 12:55:27,453 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 12:55:32,500 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:55:33,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:55:33,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:55:34,128 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:55:34,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:55:34,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:55:34,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:55:34,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:55:34,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:55:34,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:55:34,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:55:34,681 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:55:34,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:55:34,759 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['73720 memcached\n'], []) 2022-02-08 12:55:34,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 12:55:34,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 12:55:34,845 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['73720 memcached\n'], []) 2022-02-08 12:55:34,848 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:55:34,890 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:55:35,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:55:36,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:55:36,688 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:55:36,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:55:36,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:55:36,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:55:36,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:55:36,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:55:36,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:55:36,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:55:36,921 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:55:36,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:55:37,058 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['130303 memcached\n'], []) 2022-02-08 12:55:37,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 12:55:37,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 12:55:37,144 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['130303 memcached\n'], []) 2022-02-08 12:55:37,147 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:55:37,157 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:55:38,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:55:38,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:55:39,088 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:55:39,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:55:39,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:55:39,150 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:55:39,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:55:39,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:55:39,255 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:55:39,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:55:39,359 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:55:39,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:55:39,451 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['113929 memcached\n'], []) 2022-02-08 12:55:39,456 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 12:55:39,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 12:55:39,551 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['113929 memcached\n'], []) 2022-02-08 12:55:39,556 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:55:39,585 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:55:40,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:55:41,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:55:41,486 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:55:41,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:55:41,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:55:41,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:55:41,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:55:41,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:55:41,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:55:41,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:55:41,822 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:55:41,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:55:41,920 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['7807 memcached\n'], []) 2022-02-08 12:55:41,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 12:55:41,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 12:55:42,002 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['7807 memcached\n'], []) 2022-02-08 12:55:42,003 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:55:42,012 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:55:42,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:55:43,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:55:43,994 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:55:43,999 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:55:44,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:55:44,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:55:44,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:55:44,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:55:44,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:55:44,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:55:44,267 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:55:44,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:55:44,496 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['1680 memcached\n'], []) 2022-02-08 12:55:44,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 12:55:44,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 12:55:44,565 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['1680 memcached\n'], []) 2022-02-08 12:55:44,568 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:55:44,578 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:55:46,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:55:46,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:55:46,701 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:55:46,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:55:46,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:55:46,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:55:46,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:55:46,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:55:46,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:55:46,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:55:46,913 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:55:46,915 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:55:46,980 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['36875 memcached\n'], []) 2022-02-08 12:55:46,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 12:55:47,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 12:55:47,048 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['36875 memcached\n'], []) 2022-02-08 12:55:47,051 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:55:47,072 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:55:47,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:55:48,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:55:48,796 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:55:48,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:55:48,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:55:48,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:55:48,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:55:48,915 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:55:48,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:55:48,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:55:48,994 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:55:49,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:55:49,079 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['100438 memcached\n'], []) 2022-02-08 12:55:49,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 12:55:49,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 12:55:49,187 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['100438 memcached\n'], []) 2022-02-08 12:55:49,190 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:55:49,206 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:55:49,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 16384 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:55:50,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:55:50,509 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:55:50,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:55:50,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:55:50,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:55:50,584 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:55:50,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:55:50,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:55:50,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:55:50,698 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:55:50,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:55:50,766 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['115635 memcached\n'], []) 2022-02-08 12:55:50,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 12:55:50,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 12:55:50,822 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['115635 memcached\n'], []) 2022-02-08 12:55:50,826 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:55:50,845 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:55:51,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:55:52,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:55:52,257 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:55:52,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:55:52,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:55:52,323 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:55:52,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:55:52,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:55:52,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:55:52,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:55:52,451 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 12:55:52,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:55:52,522 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['52787 memcached\n'], []) 2022-02-08 12:55:52,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 12:55:52,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 12:55:52,589 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['52787 memcached\n'], []) 2022-02-08 12:55:52,592 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:55:52,601 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:55:52,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:55:52,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:55:53,058 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:55:53,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:55:53,086 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:55:53,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:55:53,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:55:53,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:55:53,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:55:53,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:55:53,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:55:53,270 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 12:55:53,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:55:53,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:55:59,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:56:00,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:56:01,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:56:01,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:56:03,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:56:03,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:56:03,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:56:04,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:56:04,112 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 12:56:04,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:56:04,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:56:04,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:56:04,286 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 12:56:04,289 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:56:04,316 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:56:04,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:56:04,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:56:04,780 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:56:04,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:56:04,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:56:04,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:56:04,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:56:04,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:56:04,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:56:04,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:56:04,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:56:04,983 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 12:56:05,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:56:05,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:56:12,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:56:13,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:56:14,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:56:14,456 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:56:15,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:56:16,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:56:16,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:56:16,364 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:56:16,381 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 12:56:16,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:56:16,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:56:16,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:56:16,510 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 12:56:16,513 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:56:16,523 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 12:56:16,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:56:16,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 12:56:17,118 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:56:17,124 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 12:56:17,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 12:56:17,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 12:56:17,204 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 12:56:17,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 12:56:17,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 12:56:17,280 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 12:56:17,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:56:17,328 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 12:56:17,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:56:17,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:56:17,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:56:17,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:56:17,584 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:56:17,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:56:17,630 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 12:56:17,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:56:17,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:56:17,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:56:17,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:17,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:56:17,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:17,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:17,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:17,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:17,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:18,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:18,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:18,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:18,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:18,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:56:18,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:18,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:18,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:18,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:18,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:18,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:18,423 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:18,448 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:18,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:18,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:56:18,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:18,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:18,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:18,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:18,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:18,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:18,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:18,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:18,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:18,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:56:18,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:18,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:18,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:18,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:18,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:18,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:19,017 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:19,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:19,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:19,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:56:19,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:19,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:19,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:19,227 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:19,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:19,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:19,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:19,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:19,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:19,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:56:19,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:19,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:19,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:19,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:19,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:19,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:19,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:19,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:19,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:19,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:56:19,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:19,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:19,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:19,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:19,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:19,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:19,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:19,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:19,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:19,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:56:20,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:20,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:20,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:20,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:20,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:20,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:20,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:20,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:20,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:20,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:56:20,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:20,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:20,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:20,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:20,448 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:20,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:20,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:20,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:20,548 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:20,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:56:20,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:20,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:20,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:20,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:20,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:20,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:20,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:20,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:20,805 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 12:56:20,819 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:56:20,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:56:21,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:56:21,318 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:56:21,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:56:21,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:56:21,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:56:21,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:56:21,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:56:21,459 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:56:21,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:56:21,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:56:21,533 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 12:56:21,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:56:21,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:56:28,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:56:29,275 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:56:29,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:56:30,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:56:30,832 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 12:56:30,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:56:30,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:56:31,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:56:31,279 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 12:56:31,286 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:56:31,309 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 12:56:31,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:56:31,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 12:56:31,948 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:56:31,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 12:56:31,987 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 12:56:32,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 12:56:32,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 12:56:32,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 12:56:32,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 12:56:32,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 12:56:32,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:56:32,142 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 12:56:32,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:56:32,280 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:56:32,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:56:32,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:56:32,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:56:32,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:56:32,411 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 12:56:32,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:56:32,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:56:32,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:56:32,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:32,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:56:32,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:32,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:32,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:32,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:32,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:32,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:32,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:32,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:56:32,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:32,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:56:32,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:33,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:33,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:33,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:33,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:33,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:33,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:33,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:56:33,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:33,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:56:33,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:33,209 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:33,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:33,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:33,280 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:33,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:33,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:33,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:56:33,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:33,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:56:33,494 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:33,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:33,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:33,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:33,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:33,624 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:33,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:33,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:56:33,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:33,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:56:33,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:33,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:33,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:33,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:33,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:33,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:33,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:34,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:56:34,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:34,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:56:34,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:34,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:34,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:34,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:34,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:34,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:34,275 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:34,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:56:34,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:34,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:56:34,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:34,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:34,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:34,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:34,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:34,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:34,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:34,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:56:34,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:34,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:56:34,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:34,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:34,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:34,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:34,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:34,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:34,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:34,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:56:34,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:34,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:56:34,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:35,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:35,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:35,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:35,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:35,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:35,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:35,204 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:56:35,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:35,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:56:35,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:35,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:35,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:35,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:35,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:35,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:35,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:35,499 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:56:35,523 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 12:56:35,529 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:56:35,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:56:36,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:56:36,349 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:56:36,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:56:36,423 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:56:36,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:56:36,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:56:36,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:56:36,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:56:36,693 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:56:36,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:56:36,799 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 12:56:36,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:56:36,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:56:43,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:56:44,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:56:45,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:56:45,900 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:56:47,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:56:48,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:56:48,854 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:56:49,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:56:49,066 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 12:56:49,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:56:49,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:56:49,183 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:56:49,211 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 12:56:49,217 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:56:49,236 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:56:49,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:56:49,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:56:49,926 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:56:49,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:56:49,964 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:56:50,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:56:50,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:56:50,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 5 queries submitted, 0 failed, 16409 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:56:50,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:56:50,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:56:50,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:56:50,219 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 12:56:50,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:56:50,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:56:59,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:57:00,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:57:00,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:57:01,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:57:02,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:57:03,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:57:03,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:57:03,459 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:57:03,506 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 12:57:03,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:57:03,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:57:03,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:57:03,703 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 12:57:03,707 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:57:03,732 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 12:57:03,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:57:04,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 12:57:04,332 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:57:04,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 12:57:04,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 12:57:04,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 12:57:04,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 12:57:04,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 12:57:04,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 12:57:04,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 12:57:04,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:57:04,552 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 12:57:04,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:57:04,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:57:04,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:57:04,701 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:57:04,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:57:04,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:57:04,782 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 12:57:04,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:57:04,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:57:04,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:57:04,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:04,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:57:04,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:05,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:05,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:05,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:05,186 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:05,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:05,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:05,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:05,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:05,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:57:05,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:05,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:05,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:05,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:05,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:05,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:05,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:05,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:05,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:05,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:57:05,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:05,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:05,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:05,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:05,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:05,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:05,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:05,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:05,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:06,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:57:06,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:06,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:06,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:06,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:06,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:06,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:06,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:06,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:06,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:06,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:57:06,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:06,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:06,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:06,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:06,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:06,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:06,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:06,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:06,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:06,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:57:06,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:06,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:06,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:06,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:06,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:06,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:06,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:07,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:07,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:07,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:57:07,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:07,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:07,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:07,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:07,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:07,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:07,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:07,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:07,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:07,418 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:57:07,448 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:07,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:07,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:07,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:07,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:07,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:07,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:07,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:07,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:07,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:57:07,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:07,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:07,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:07,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:07,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:08,020 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:08,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:08,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:08,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:08,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:57:08,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:08,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:08,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:08,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:08,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:08,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:08,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:08,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:08,459 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 12:57:08,473 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 12:57:08,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:57:08,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 12:57:08,908 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:57:08,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 12:57:08,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 12:57:08,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 12:57:08,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 12:57:09,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 12:57:09,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 12:57:09,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 12:57:09,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:57:09,096 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 12:57:09,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:57:09,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:57:09,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:57:09,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:57:09,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:57:09,280 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:57:09,298 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 12:57:09,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:57:09,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:57:09,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:57:09,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:09,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:57:09,456 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:09,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:09,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:09,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:09,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:09,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:09,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:09,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:57:09,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:09,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:57:09,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:09,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:09,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:09,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:09,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:09,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:09,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:10,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:57:10,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:10,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:57:10,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:10,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:10,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:10,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:10,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:10,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:10,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:10,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:57:10,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:10,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:57:10,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:10,423 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:10,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:10,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:10,492 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:10,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:10,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:10,579 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:57:10,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:10,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:57:10,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:10,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:10,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:10,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:10,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:10,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:10,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:10,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:57:10,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:10,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:57:10,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:11,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:11,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:11,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:11,084 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:11,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:11,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:11,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:57:11,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:11,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:57:11,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:11,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:11,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:11,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:11,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:11,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:11,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:11,515 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:57:11,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:11,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:57:11,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:11,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:11,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:11,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:11,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:11,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:11,742 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:11,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:57:11,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:11,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:57:11,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:11,869 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:11,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:11,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:11,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:11,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:12,028 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:12,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:57:12,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:12,094 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:57:12,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:12,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:12,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:12,181 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:12,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:12,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:12,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:12,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:57:12,322 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 12:57:12,335 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:57:13,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:57:13,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:57:13,927 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:57:13,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:57:13,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:57:13,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:57:14,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:57:14,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:57:14,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:57:14,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:57:14,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:57:14,243 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 12:57:14,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:57:14,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:57:21,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:57:22,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:57:23,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:57:23,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:57:24,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:57:25,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:57:25,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:57:25,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:57:25,421 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 12:57:25,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:57:25,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:57:25,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:57:25,622 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 12:57:25,625 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:57:25,634 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:57:26,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:57:26,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:57:27,167 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:57:27,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:57:27,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:57:27,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:57:27,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:57:27,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:57:27,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:57:27,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:57:27,456 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:57:27,483 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 12:57:27,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:57:27,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:57:35,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:57:36,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:57:36,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:57:37,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:57:37,220 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 12:57:37,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:57:37,255 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:57:37,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:57:37,351 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 12:57:37,354 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:57:37,365 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:57:37,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:57:37,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:57:38,006 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:57:38,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:57:38,028 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:57:38,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:57:38,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:57:38,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:57:38,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:57:38,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:57:38,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:57:38,190 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 12:57:38,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:57:38,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:57:44,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:57:45,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:57:46,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:57:46,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:57:46,822 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 12:57:46,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:57:46,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:57:46,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:57:46,967 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 12:57:46,971 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:57:46,990 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:57:47,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:57:48,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:57:48,608 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:57:48,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:57:48,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:57:48,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:57:48,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:57:48,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:57:48,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:57:48,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:57:48,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:57:48,923 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 12:57:49,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:57:49,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 16435 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 12:57:58,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:57:59,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:57:59,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:58:00,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:58:02,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:58:03,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:58:03,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 12:58:03,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:58:03,836 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 12:58:03,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:58:03,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:58:03,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:58:04,023 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 12:58:04,026 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:58:04,032 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 12:58:04,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:58:04,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 12:58:04,503 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:58:04,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 12:58:04,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 12:58:04,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 12:58:04,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 12:58:04,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 12:58:04,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 12:58:04,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 12:58:04,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 12:58:04,743 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 12:58:04,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 12:58:04,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:58:04,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 12:58:04,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:58:04,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 12:58:05,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 12:58:05,036 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 12:58:05,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 12:58:05,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 12:58:05,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 12:58:05,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:58:05,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 12:58:05,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:58:05,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:58:05,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:58:05,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:58:05,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:58:05,418 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:58:05,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:58:05,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 12:58:05,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:58:05,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 12:58:05,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:58:05,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:58:05,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:58:05,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:58:05,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:58:05,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:58:05,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:58:05,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 12:58:05,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:58:05,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 12:58:05,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:58:05,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:58:05,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:58:05,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:58:05,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:58:06,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:58:06,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:58:06,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 12:58:06,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:58:06,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 12:58:06,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:58:06,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:58:06,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:58:06,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:58:06,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:58:06,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:58:06,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:58:06,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 12:58:06,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:58:06,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 12:58:06,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:58:06,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:58:06,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:58:06,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:58:06,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:58:06,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:58:06,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:58:06,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 12:58:06,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:58:06,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 12:58:06,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:58:06,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:58:06,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:58:06,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:58:06,841 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:58:06,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:58:06,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:58:06,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 12:58:06,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:58:06,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 12:58:07,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:58:07,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:58:07,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:58:07,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:58:07,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:58:07,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:58:07,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:58:07,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 12:58:07,293 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:58:07,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 12:58:07,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:58:07,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:58:07,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:58:07,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:58:07,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:58:07,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:58:07,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:58:07,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 12:58:07,579 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:58:07,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 12:58:07,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:58:07,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:58:07,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:58:07,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:58:07,772 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:58:07,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:58:07,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:58:07,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 12:58:07,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:58:07,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 12:58:07,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:58:07,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:58:08,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:58:08,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:58:08,084 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:58:08,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:58:08,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:58:08,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 12:58:08,194 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 12:58:08,198 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:58:08,211 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 12:58:08,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:58:09,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 12:58:09,187 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:58:09,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 12:58:09,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 12:58:09,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 12:58:09,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 12:58:09,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 12:58:09,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 12:58:09,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 12:58:09,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:58:09,971 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 12:58:09,974 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.77699995041. 2022-02-08 12:58:09,977 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:58:09,993 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 12:58:10,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:58:10,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 12:58:10,855 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:58:10,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 12:58:10,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 12:58:10,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 12:58:10,987 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 12:58:11,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 12:58:11,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 12:58:11,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 12:58:11,150 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:58:11,908 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 12:58:11,911 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.93499994278. 2022-02-08 12:58:11,914 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:58:11,927 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 12:58:12,849 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:58:13,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 12:58:13,766 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:58:13,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 12:58:13,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 12:58:13,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 12:58:13,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 12:58:13,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 12:58:13,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 12:58:14,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 12:58:14,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:58:14,776 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 12:58:14,779 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.86500000954. 2022-02-08 12:58:14,779 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:58:14,786 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 12:58:15,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:58:16,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 12:58:16,517 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:58:16,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 12:58:16,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 12:58:16,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 12:58:16,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 12:58:16,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 12:58:16,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 12:58:16,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 12:58:16,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:58:17,437 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 12:58:17,440 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.66000008583. 2022-02-08 12:58:17,443 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:58:17,460 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 12:58:18,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:58:18,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 12:58:19,453 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:58:19,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 12:58:19,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 12:58:19,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 12:58:19,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 12:58:19,624 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 12:58:19,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 12:58:19,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 12:58:19,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:58:20,691 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 12:58:20,697 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.25499987602. 2022-02-08 12:58:20,700 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:58:20,721 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 12:58:21,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:58:21,760 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 12:58:21,924 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:58:21,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 12:58:21,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 12:58:21,994 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 12:58:22,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 12:58:22,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 12:58:22,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 12:58:22,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 12:58:22,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:58:23,203 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 12:58:23,207 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.50699996948. 2022-02-08 12:58:23,209 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:58:23,219 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 12:58:24,308 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:58:24,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 12:58:25,108 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:58:25,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 12:58:25,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 12:58:25,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 12:58:25,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 12:58:25,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 12:58:25,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 12:58:25,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 12:58:25,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:58:25,857 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 12:58:25,859 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.64999985695. 2022-02-08 12:58:25,861 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:58:25,885 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 12:58:26,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:58:27,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 12:58:27,496 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:58:27,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 12:58:27,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 12:58:27,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 12:58:27,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 12:58:27,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 12:58:27,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 12:58:27,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 12:58:27,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:58:28,743 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 12:58:28,750 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.88899993896. 2022-02-08 12:58:28,753 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 12:58:28,769 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 12:58:29,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 12:58:30,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 12:58:30,509 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 12:58:30,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 12:58:30,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 12:58:30,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 12:58:30,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 12:58:30,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 12:58:30,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 12:58:30,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 12:58:30,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 12:58:31,733 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 12:58:31,737 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.9849998951. ############################################################ # # RAM FootPrint: 4687.09765625 # ############################################################ 2022-02-08 12:58:31,933 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 12:58:31,937 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 110 seconds. Reason: Iteration:10 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 16458 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 16480 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:00:21,980 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 13:00:21,982 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 13:00:27,033 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:00:27,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:00:28,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:00:28,871 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:00:28,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:00:28,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:00:29,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:00:29,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:00:29,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:00:29,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:00:29,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:00:29,292 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:00:29,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:00:29,357 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['74452 memcached\n'], []) 2022-02-08 13:00:29,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 13:00:29,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:00:29,451 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['74452 memcached\n'], []) 2022-02-08 13:00:29,454 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:00:29,463 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:00:30,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:00:30,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:00:31,061 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:00:31,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:00:31,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:00:31,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:00:31,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:00:31,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:00:31,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:00:31,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:00:31,415 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:00:31,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:00:31,505 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['131062 memcached\n'], []) 2022-02-08 13:00:31,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 13:00:31,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:00:31,592 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['131062 memcached\n'], []) 2022-02-08 13:00:31,598 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:00:31,622 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:00:32,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:00:32,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:00:32,782 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:00:32,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:00:32,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:00:32,918 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:00:32,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:00:32,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:00:32,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:00:33,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:00:33,128 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:00:33,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:00:33,220 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['114822 memcached\n'], []) 2022-02-08 13:00:33,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 13:00:33,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:00:33,309 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['114822 memcached\n'], []) 2022-02-08 13:00:33,315 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:00:33,346 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:00:34,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:00:34,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:00:35,131 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:00:35,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:00:35,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:00:35,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:00:35,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:00:35,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:00:35,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:00:35,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:00:35,348 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:00:35,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:00:35,420 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['8567 memcached\n'], []) 2022-02-08 13:00:35,423 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 13:00:35,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:00:35,490 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['8567 memcached\n'], []) 2022-02-08 13:00:35,493 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:00:35,516 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:00:36,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:00:37,515 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:00:37,759 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:00:37,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:00:37,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:00:37,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:00:37,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:00:37,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:00:37,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:00:37,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:00:37,977 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:00:37,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:00:38,045 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['2448 memcached\n'], []) 2022-02-08 13:00:38,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 13:00:38,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:00:38,111 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['2448 memcached\n'], []) 2022-02-08 13:00:38,114 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:00:38,124 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:00:39,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:00:39,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:00:40,030 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:00:40,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:00:40,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:00:40,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:00:40,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:00:40,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:00:40,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:00:40,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:00:40,240 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:00:40,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:00:40,346 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['37636 memcached\n'], []) 2022-02-08 13:00:40,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 13:00:40,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:00:40,427 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['37636 memcached\n'], []) 2022-02-08 13:00:40,430 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:00:40,443 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:00:41,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:00:41,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:00:42,234 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:00:42,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:00:42,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:00:42,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:00:42,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:00:42,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:00:42,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:00:42,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:00:42,683 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:00:42,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:00:42,752 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['101172 memcached\n'], []) 2022-02-08 13:00:42,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 13:00:42,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:00:42,855 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['101172 memcached\n'], []) 2022-02-08 13:00:42,859 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:00:42,901 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:00:43,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:00:43,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:00:44,085 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:00:44,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:00:44,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:00:44,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:00:44,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:00:44,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:00:44,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:00:44,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:00:44,292 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:00:44,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:00:44,365 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['116368 memcached\n'], []) 2022-02-08 13:00:44,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 13:00:44,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:00:44,456 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['116368 memcached\n'], []) 2022-02-08 13:00:44,459 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:00:44,819 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:00:45,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:00:45,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:00:45,984 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:00:45,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:00:46,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:00:46,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:00:46,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:00:46,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:00:46,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:00:46,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:00:46,204 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:00:46,209 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:00:46,289 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['53550 memcached\n'], []) 2022-02-08 13:00:46,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 13:00:46,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:00:46,371 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['53550 memcached\n'], []) 2022-02-08 13:00:46,372 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:00:46,391 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:00:46,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:00:46,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:00:47,005 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:00:47,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:00:47,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:00:47,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:00:47,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:00:47,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:00:47,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:00:47,183 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:00:47,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:00:47,237 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 13:00:47,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:00:47,464 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 16501 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:00:54,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:00:54,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:00:55,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:00:55,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:00:57,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:00:57,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:00:58,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:00:58,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:00:58,276 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 13:00:58,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:00:58,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:00:58,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:00:58,391 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 13:00:58,394 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:00:58,401 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:00:58,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:00:58,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:00:58,859 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:00:58,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:00:58,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:00:58,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:00:58,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:00:58,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:00:58,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:00:58,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:00:58,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:00:59,006 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 13:00:59,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:00:59,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:01:06,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:01:07,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:01:08,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:01:08,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:01:10,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:01:10,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:01:10,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:01:10,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:01:10,619 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 13:01:10,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:01:10,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:01:10,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:01:10,750 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 13:01:10,753 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:01:10,782 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 13:01:10,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:01:11,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 13:01:11,253 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:01:11,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 13:01:11,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 13:01:11,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 13:01:11,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 13:01:11,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 13:01:11,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 13:01:11,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 13:01:11,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:01:11,424 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 13:01:11,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:01:11,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:01:11,701 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:01:11,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:01:11,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:01:11,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:01:11,822 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 13:01:11,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:01:11,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:01:11,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:01:11,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:11,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:01:12,010 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:12,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:12,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:12,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:12,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:12,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:12,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:12,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:12,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:12,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:01:12,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:12,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:12,466 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:12,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:12,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:12,624 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:12,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:12,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:12,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:12,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:01:12,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:12,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:12,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:12,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:12,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:12,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:12,915 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:12,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:12,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:13,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:01:13,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:13,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:13,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:13,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:13,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:13,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:13,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:13,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:13,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:13,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:01:13,288 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:13,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:13,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:13,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:13,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:13,448 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:13,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:13,499 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:13,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:13,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:01:13,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:13,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:13,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:13,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:13,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:13,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:13,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:13,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:13,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:13,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:01:13,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:13,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:13,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:13,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:13,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:13,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:14,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:14,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:14,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:14,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:01:14,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:14,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:14,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:14,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:14,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:14,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:14,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:14,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:14,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:14,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:01:14,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:14,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:14,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:14,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:14,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:14,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:14,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:14,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:14,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:14,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:01:14,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:14,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:14,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:14,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:14,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:14,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:14,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:14,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:14,895 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 13:01:14,911 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:01:15,086 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:01:15,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:01:15,836 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:01:15,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:01:15,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:01:15,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:01:15,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:01:15,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:01:15,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:01:16,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:01:16,084 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:01:16,135 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 13:01:16,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:01:16,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:01:22,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:01:24,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:01:24,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:01:25,183 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:01:25,237 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 13:01:25,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:01:25,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:01:25,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:01:25,548 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 13:01:25,552 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:01:25,569 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 13:01:25,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:01:25,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 13:01:26,062 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:01:26,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 13:01:26,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 13:01:26,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 13:01:26,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 13:01:26,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 13:01:26,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 13:01:26,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 13:01:26,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:01:26,247 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 13:01:26,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:01:26,418 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:01:26,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:01:26,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:01:26,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:01:26,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:01:26,533 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 13:01:26,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:01:26,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:01:26,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:01:26,627 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:26,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:01:26,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:26,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:26,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:26,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:26,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:26,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:26,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:27,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:27,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:27,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:01:27,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:27,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:27,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:27,183 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:27,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:27,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:27,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:27,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:27,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:27,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:01:27,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:27,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:27,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:27,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:27,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:27,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:27,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:27,528 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:27,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:27,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:01:27,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:27,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:27,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:27,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:27,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:27,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:27,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:27,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:27,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:27,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:01:27,854 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:27,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:27,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:27,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:27,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:27,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:28,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:28,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:28,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:28,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:01:28,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:28,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:28,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:28,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:28,201 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:28,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:28,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:28,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:28,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:28,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:01:28,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:28,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:28,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:28,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:28,464 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:28,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:28,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:28,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:28,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:28,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:01:28,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:28,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:28,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:28,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:28,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:28,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:28,760 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:28,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:28,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:28,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:01:28,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:28,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:28,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:28,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:28,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:28,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:28,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:29,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:29,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:29,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:01:29,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:29,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:29,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:29,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:29,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:29,214 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:29,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:29,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:29,298 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 13:01:29,315 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:01:29,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:01:29,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:01:30,079 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:01:30,094 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:01:30,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:01:30,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:01:30,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:01:30,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:01:30,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:01:30,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:01:30,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:01:30,371 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 13:01:30,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:01:30,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:01:36,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:01:37,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:01:38,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:01:38,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:01:40,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:01:40,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:01:41,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:01:41,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:01:41,388 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 13:01:41,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:01:41,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:01:41,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:01:41,540 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 13:01:41,548 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:01:41,558 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:01:41,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:01:42,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:01:42,157 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:01:42,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:01:42,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:01:42,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:01:42,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:01:42,308 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:01:42,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:01:42,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:01:42,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:01:42,433 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 13:01:42,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:01:42,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 16523 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:01:50,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:01:51,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:01:51,900 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:01:52,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:01:54,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:01:54,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:01:54,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:01:54,584 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:01:54,612 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 13:01:54,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:01:54,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:01:54,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:01:54,782 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 13:01:54,786 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:01:54,802 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 13:01:54,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:01:55,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 13:01:55,368 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:01:55,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 13:01:55,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 13:01:55,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 13:01:55,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 13:01:55,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 13:01:55,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 13:01:55,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 13:01:55,584 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:01:55,615 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 13:01:55,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:01:55,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:01:55,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:01:55,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:01:55,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:01:55,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:01:55,877 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 13:01:55,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:01:55,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:01:55,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:01:55,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:56,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:01:56,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:56,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:56,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:56,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:56,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:56,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:56,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:56,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:01:56,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:56,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:01:56,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:56,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:56,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:56,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:56,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:56,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:56,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:56,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:01:56,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:56,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:01:56,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:56,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:56,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:57,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:57,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:57,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:57,122 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:57,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:01:57,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:57,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:01:57,255 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:57,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:57,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:57,341 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:57,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:57,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:57,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:57,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:01:57,520 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:57,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:01:57,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:57,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:57,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:57,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:57,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:57,757 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:57,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:57,836 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:01:57,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:57,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:01:57,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:57,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:58,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:58,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:58,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:58,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:58,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:58,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:01:58,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:58,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:01:58,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:58,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:58,303 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:58,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:58,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:58,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:58,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:58,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:01:58,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:58,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:01:58,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:58,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:58,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:58,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:58,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:58,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:58,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:58,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:01:58,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:58,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:01:58,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:58,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:58,869 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:58,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:58,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:58,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:58,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:59,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:01:59,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:59,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:01:59,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:59,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:59,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:59,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:59,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:59,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:59,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:59,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:01:59,358 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 13:01:59,375 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 13:01:59,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:01:59,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 13:01:59,921 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:01:59,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 13:01:59,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 13:01:59,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 13:02:00,017 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 13:02:00,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 13:02:00,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 13:02:00,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 13:02:00,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:02:00,164 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 13:02:00,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:02:00,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:02:00,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:02:00,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:02:00,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:02:00,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:02:00,392 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 13:02:00,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:02:00,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:02:00,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:02:00,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:02:00,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:02:00,566 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:02:00,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:02:00,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:02:00,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:02:00,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:02:00,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:02:00,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:02:00,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:02:00,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:02:00,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:02:01,002 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:02:01,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:02:01,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:02:01,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:02:01,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:02:01,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:02:01,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:02:01,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:02:01,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:02:01,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:02:01,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:02:01,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:02:01,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:02:01,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:02:01,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:02:01,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:02:01,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:02:01,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:02:01,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:02:01,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:02:01,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:02:01,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:02:01,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:02:01,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:02:01,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:02:01,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:02:01,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:02:01,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:02:01,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:02:01,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:02:01,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:02:01,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:02:01,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:02:01,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:02:01,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:02:01,897 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:02:01,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:02:01,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:02:01,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:02:02,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:02:02,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:02:02,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:02:02,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:02:02,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:02:02,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:02:02,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:02:02,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:02:02,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:02:02,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:02:02,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:02:02,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:02:02,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:02:02,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:02:02,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:02:02,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:02:02,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:02:02,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:02:02,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:02:02,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:02:02,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:02:02,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:02:02,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:02:02,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:02:02,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:02:02,627 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:02:02,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:02:02,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:02:02,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:02:02,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:02:02,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:02:02,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:02:02,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:02:02,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:02:02,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:02:02,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:02:02,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:02:02,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:02:02,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:02:03,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:02:03,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:02:03,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:02:03,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:02:03,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:02:03,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:02:03,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:02:03,201 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:02:03,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:02:03,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:02:03,282 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 13:02:03,302 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:02:03,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:02:03,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:02:04,122 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:02:04,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:02:04,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:02:04,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:02:04,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:02:04,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:02:04,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:02:04,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:02:04,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:02:04,838 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 13:02:04,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:02:04,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:02:12,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:02:13,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:02:14,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:02:14,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:02:16,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:02:16,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:02:16,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:02:16,492 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:02:16,529 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 13:02:16,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:02:16,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:02:16,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:02:16,687 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 13:02:16,690 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:02:16,737 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:02:17,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:02:18,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:02:18,328 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:02:18,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:02:18,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:02:18,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:02:18,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:02:18,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:02:18,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:02:18,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:02:18,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:02:18,858 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 13:02:18,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:02:19,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:02:26,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:02:27,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:02:27,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:02:28,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:02:28,525 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 13:02:28,528 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:02:28,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:02:28,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:02:28,667 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 13:02:28,671 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:02:28,733 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:02:29,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:02:29,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:02:29,941 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:02:29,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:02:30,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:02:30,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:02:30,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:02:30,122 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:02:30,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:02:30,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:02:30,201 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:02:30,223 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 13:02:30,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:02:30,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:02:36,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:02:37,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:02:38,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:02:38,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:02:38,717 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 13:02:38,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:02:38,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:02:38,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:02:38,834 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 13:02:38,835 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:02:38,842 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:02:39,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:02:40,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:02:40,766 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:02:40,772 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:02:40,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:02:40,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:02:41,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:02:41,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:02:41,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:02:41,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:02:41,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:02:41,348 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 13:02:41,464 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:02:41,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 16543 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:02:50,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:02:56,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:02:57,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:02:57,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:03:01,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:03:01,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:03:01,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:03:02,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:03:02,183 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 13:03:02,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:03:02,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:03:02,316 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:03:02,354 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 13:03:02,355 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:03:02,368 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 13:03:02,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:03:02,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 13:03:02,961 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:03:02,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 13:03:03,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 13:03:03,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 13:03:03,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 13:03:03,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 13:03:03,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 13:03:03,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 13:03:03,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:03:03,171 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 13:03:03,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:03:03,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:03:03,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:03:03,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:03:03,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:03:03,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:03:03,391 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 13:03:03,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:03:03,418 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:03:03,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:03:03,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:03:03,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:03:03,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:03:03,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:03:03,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:03:03,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:03:03,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:03:03,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:03:03,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:03:03,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:03:03,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:03:03,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:03:04,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:03:04,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:03:04,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:03:04,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:03:04,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:03:04,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:03:04,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:03:04,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:03:04,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:03:04,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:03:04,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:03:04,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:03:04,341 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:03:04,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:03:04,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:03:04,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:03:04,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:03:04,499 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:03:04,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:03:04,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:03:04,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:03:04,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:03:04,624 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:03:04,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:03:04,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:03:04,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:03:04,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:03:04,757 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:03:04,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:03:04,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:03:04,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:03:04,869 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:03:04,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:03:04,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:03:04,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:03:04,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:03:05,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:03:05,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:03:05,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:03:05,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:03:05,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:03:05,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:03:05,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:03:05,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:03:05,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:03:05,239 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:03:05,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:03:05,316 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:03:05,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:03:05,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:03:05,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:03:05,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:03:05,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:03:05,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:03:05,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:03:05,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:03:05,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:03:05,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:03:05,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:03:05,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:03:05,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:03:05,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:03:05,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:03:05,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:03:05,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:03:05,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:03:05,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:03:05,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:03:05,918 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:03:05,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:03:06,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:03:06,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:03:06,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:03:06,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:03:06,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:03:06,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:03:06,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:03:06,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:03:06,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:03:06,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:03:06,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:03:06,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:03:06,448 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:03:06,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:03:06,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:03:06,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:03:06,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:03:06,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:03:06,621 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 13:03:06,627 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:03:06,651 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:03:07,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:03:07,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:03:07,834 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:03:07,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:03:07,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:03:07,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:03:08,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:03:08,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:03:08,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:03:08,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:03:08,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:03:08,651 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:03:08,654 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.02800011635. 2022-02-08 13:03:08,654 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:03:08,660 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:03:09,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:03:10,094 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:03:10,309 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:03:10,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:03:10,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:03:10,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:03:10,423 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:03:10,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:03:10,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:03:10,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:03:10,584 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:03:11,289 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:03:11,295 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.63999986649. 2022-02-08 13:03:11,299 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:03:11,326 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:03:12,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:03:12,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:03:13,188 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:03:13,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:03:13,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:03:13,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:03:13,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:03:13,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:03:13,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:03:13,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:03:13,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:03:14,190 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:03:14,196 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.89599990845. 2022-02-08 13:03:14,197 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:03:14,210 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:03:15,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:03:15,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:03:16,108 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:03:16,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:03:16,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:03:16,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:03:16,204 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:03:16,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:03:16,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:03:16,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:03:16,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:03:17,671 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:03:17,674 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.47600007057. 2022-02-08 13:03:17,674 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:03:17,680 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:03:18,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:03:19,227 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:03:19,598 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:03:19,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:03:19,627 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:03:19,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:03:19,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:03:19,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:03:19,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:03:19,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:03:19,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:03:20,519 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:03:20,520 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.84599995613. 2022-02-08 13:03:20,522 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:03:20,529 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:03:21,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:03:21,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:03:21,983 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:03:21,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:03:22,017 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:03:22,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:03:22,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:03:22,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:03:22,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:03:22,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:03:22,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:03:22,913 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:03:22,917 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.39499998093. 2022-02-08 13:03:22,918 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:03:22,933 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:03:23,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:03:24,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:03:24,641 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:03:24,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:03:24,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:03:24,701 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:03:24,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:03:24,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:03:24,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:03:24,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:03:24,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:03:25,417 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:03:25,423 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.50300002098. 2022-02-08 13:03:25,424 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:03:25,441 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:03:26,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:03:26,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:03:27,259 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:03:27,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:03:27,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:03:27,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:03:27,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:03:27,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:03:27,548 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:03:27,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:03:27,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:03:28,536 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:03:28,538 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.1130001545. 2022-02-08 13:03:28,539 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:03:28,540 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:03:29,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:03:30,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:03:30,641 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:03:30,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:03:30,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:03:30,897 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:03:30,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:03:31,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:03:31,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:03:31,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:03:31,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:03:32,101 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:03:32,102 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.56399989128. ############################################################ # # RAM FootPrint: 4984.1796875 # ############################################################ 2022-02-08 13:03:32,773 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 13:03:32,776 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 86 seconds. Reason: Iteration:11 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 16570 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 16593 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:04:58,832 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 13:04:58,835 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 13:05:03,855 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:05:04,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:05:05,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:05:05,696 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:05:05,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:05:05,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:05:05,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:05:05,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:05:05,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:05:05,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:05:06,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:05:06,128 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:05:06,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:05:06,194 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['75256 memcached\n'], []) 2022-02-08 13:05:06,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 13:05:06,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:05:06,263 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['75256 memcached\n'], []) 2022-02-08 13:05:06,269 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:05:06,282 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:05:07,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:05:07,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:05:07,849 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:05:07,854 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:05:07,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:05:07,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:05:07,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:05:07,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:05:07,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:05:08,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:05:08,056 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:05:08,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:05:08,125 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['1163 memcached\n'], []) 2022-02-08 13:05:08,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 13:05:08,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:05:08,210 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['1163 memcached\n'], []) 2022-02-08 13:05:08,211 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:05:08,217 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:05:09,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:05:09,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:05:09,923 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:05:09,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:05:09,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:05:10,010 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:05:10,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:05:10,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:05:10,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:05:10,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:05:10,180 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:05:10,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:05:10,279 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['115729 memcached\n'], []) 2022-02-08 13:05:10,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 13:05:10,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:05:10,372 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['115729 memcached\n'], []) 2022-02-08 13:05:10,375 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:05:10,384 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:05:11,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:05:12,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:05:12,246 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:05:12,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:05:12,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:05:12,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:05:12,431 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:05:12,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:05:12,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:05:12,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:05:12,551 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:05:12,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:05:12,641 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['9411 memcached\n'], []) 2022-02-08 13:05:12,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 13:05:12,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:05:12,717 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['9411 memcached\n'], []) 2022-02-08 13:05:12,720 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:05:12,736 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:05:13,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:05:14,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:05:15,098 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:05:15,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:05:15,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:05:15,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:05:15,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:05:15,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:05:15,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:05:15,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:05:15,315 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:05:15,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:05:15,397 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['3281 memcached\n'], []) 2022-02-08 13:05:15,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 13:05:15,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:05:15,470 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['3281 memcached\n'], []) 2022-02-08 13:05:15,473 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:05:15,482 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:05:16,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:05:17,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:05:17,822 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:05:17,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:05:17,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:05:17,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:05:17,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:05:17,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:05:18,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:05:18,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:05:18,114 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:05:18,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:05:18,213 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['38435 memcached\n'], []) 2022-02-08 13:05:18,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 13:05:18,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:05:18,283 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['38435 memcached\n'], []) 2022-02-08 13:05:18,286 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:05:18,290 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:05:19,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:05:20,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:05:20,328 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:05:20,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:05:20,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:05:20,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:05:20,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:05:20,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:05:20,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:05:20,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:05:20,562 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:05:20,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:05:20,631 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['101980 memcached\n'], []) 2022-02-08 13:05:20,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 13:05:20,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:05:20,704 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['101980 memcached\n'], []) 2022-02-08 13:05:20,707 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:05:20,717 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:05:21,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:05:21,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:05:22,203 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:05:22,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:05:22,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:05:22,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:05:22,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:05:22,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:05:22,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:05:22,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:05:22,408 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:05:22,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:05:22,480 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['117184 memcached\n'], []) 2022-02-08 13:05:22,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 13:05:22,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:05:22,545 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['117184 memcached\n'], []) 2022-02-08 13:05:22,549 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:05:22,559 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:05:23,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:05:23,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:05:24,078 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:05:24,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:05:24,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:05:24,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:05:24,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:05:24,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:05:24,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:05:24,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:05:24,316 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:05:24,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:05:24,404 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['54358 memcached\n'], []) 2022-02-08 13:05:24,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 13:05:24,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:05:24,470 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['54358 memcached\n'], []) 2022-02-08 13:05:24,473 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:05:24,483 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:05:24,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:05:24,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:05:25,009 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:05:25,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:05:25,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:05:25,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:05:25,086 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:05:25,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:05:25,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:05:25,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:05:25,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:05:25,233 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 13:05:25,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:05:25,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:05:32,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:05:33,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:05:33,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:05:34,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:05:35,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:05:36,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:05:36,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:05:36,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:05:36,499 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 13:05:36,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:05:36,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:05:36,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:05:36,644 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 13:05:36,650 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:05:36,661 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:05:36,854 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:05:37,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:05:37,161 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:05:37,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:05:37,181 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:05:37,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:05:37,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:05:37,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:05:37,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:05:37,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:05:37,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:05:37,352 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 13:05:37,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:05:37,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:05:44,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:05:45,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:05:46,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:05:46,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:05:48,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:05:48,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:05:48,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:05:48,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:05:48,700 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 13:05:48,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:05:48,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:05:48,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:05:48,848 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 13:05:48,851 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:05:48,861 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 13:05:49,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:05:49,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 13:05:49,311 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:05:49,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 13:05:49,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 13:05:49,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 13:05:49,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 13:05:49,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 13:05:49,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 13:05:49,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 13:05:49,456 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:05:49,479 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 13:05:49,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:05:49,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:05:49,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:05:49,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:05:49,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:05:49,709 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:05:49,727 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 13:05:49,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:05:49,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:05:49,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:05:49,793 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:05:49,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:05:49,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:05:49,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:05:49,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:05:50,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:05:50,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 5 queries submitted, 0 failed, 16614 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:05:50,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:05:50,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:05:50,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:05:50,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:05:50,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:05:50,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:05:50,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:05:50,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:05:50,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:05:50,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:05:50,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:05:50,456 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:05:50,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:05:50,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:05:50,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:05:50,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:05:50,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:05:50,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:05:50,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:05:50,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:05:50,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:05:50,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:05:50,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:05:50,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:05:50,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:05:50,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:05:50,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:05:50,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:05:50,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:05:50,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:05:50,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:05:51,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:05:51,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:05:51,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:05:51,122 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:05:51,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:05:51,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:05:51,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:05:51,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:05:51,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:05:51,308 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:05:51,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:05:51,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:05:51,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:05:51,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:05:51,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:05:51,456 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:05:51,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:05:51,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:05:51,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:05:51,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:05:51,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:05:51,584 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:05:51,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:05:51,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:05:51,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:05:51,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:05:51,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:05:51,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:05:51,757 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:05:51,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:05:51,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:05:51,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:05:51,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:05:51,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:05:51,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:05:51,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:05:51,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:05:51,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:05:52,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:05:52,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:05:52,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:05:52,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:05:52,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:05:52,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:05:52,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:05:52,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:05:52,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:05:52,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:05:52,280 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:05:52,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:05:52,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:05:52,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:05:52,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:05:52,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:05:52,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:05:52,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:05:52,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:05:52,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:05:52,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:05:52,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:05:52,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:05:52,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:05:52,655 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 13:05:52,667 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:05:52,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:05:53,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:05:53,174 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:05:53,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:05:53,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:05:53,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:05:53,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:05:53,255 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:05:53,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:05:53,293 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:05:53,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:05:53,344 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 13:05:53,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:05:53,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:06:00,010 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:06:01,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:06:01,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:06:02,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:06:02,588 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 13:06:02,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:06:02,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:06:02,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:06:02,997 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 13:06:03,003 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:06:03,019 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 13:06:03,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:06:03,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 13:06:03,471 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:06:03,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 13:06:03,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 13:06:03,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 13:06:03,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 13:06:03,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 13:06:03,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 13:06:03,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 13:06:03,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:06:03,661 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 13:06:03,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:06:03,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:06:03,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:06:03,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:06:03,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:06:03,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:06:03,869 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 13:06:03,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:06:03,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:06:03,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:06:03,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:03,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:06:04,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:04,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:04,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:04,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:04,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:04,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:04,323 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:04,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:04,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:04,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:06:04,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:04,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:04,528 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:04,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:04,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:04,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:04,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:04,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:04,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:04,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:06:04,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:04,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:04,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:04,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:04,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:04,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:04,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:04,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:04,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:04,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:06:04,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:04,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:05,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:05,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:05,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:05,124 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:05,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:05,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:05,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:05,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:06:05,270 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:05,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:05,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:05,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:05,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:05,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:05,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:05,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:05,494 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:05,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:06:05,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:05,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:05,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:05,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:05,627 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:05,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:05,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:05,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:05,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:05,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:06:05,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:05,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:05,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:05,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:05,854 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:05,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:05,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:05,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:05,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:06,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:06:06,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:06,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:06,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:06,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:06,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:06,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:06,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:06,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:06,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:06,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:06:06,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:06,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:06,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:06,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:06,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:06,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:06,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:06,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:06,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:06,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:06:06,566 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:06,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:06,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:06,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:06,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:06,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:06,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:06,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:06,756 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 13:06:06,766 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:06:07,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:06:07,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:06:07,420 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:06:07,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:06:07,459 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:06:07,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:06:07,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:06:07,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:06:07,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:06:07,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:06:07,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:06:07,707 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 13:06:07,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:06:07,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:06:14,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:06:15,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:06:15,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:06:16,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:06:18,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:06:18,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:06:18,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:06:18,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:06:18,711 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 13:06:18,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:06:18,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:06:18,854 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:06:18,904 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 13:06:18,908 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:06:18,937 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:06:19,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:06:19,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:06:19,471 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:06:19,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:06:19,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:06:19,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:06:19,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:06:19,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:06:19,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:06:19,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:06:19,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:06:19,711 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 13:06:19,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:06:19,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:06:27,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:06:28,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:06:29,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:06:29,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:06:31,499 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:06:31,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:06:31,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:06:32,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:06:32,256 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 13:06:32,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:06:32,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:06:32,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:06:32,421 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 13:06:32,424 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:06:32,433 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 13:06:32,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:06:32,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 13:06:32,924 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:06:32,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 13:06:32,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 13:06:32,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 13:06:33,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 13:06:33,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 13:06:33,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 13:06:33,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 13:06:33,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:06:33,119 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 13:06:33,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:06:33,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:06:33,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:06:33,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:06:33,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:06:33,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:06:33,381 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 13:06:33,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:06:33,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:06:33,443 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:06:33,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:33,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:06:33,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:33,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:33,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:33,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:33,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:33,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:33,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:33,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:33,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:33,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:06:34,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:34,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:34,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:34,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:34,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:34,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:34,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:34,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:34,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:34,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:06:34,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:34,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:34,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:34,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:34,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:34,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:34,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:34,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:34,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:34,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:06:34,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:34,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:34,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:34,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:34,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:34,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:34,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:34,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:34,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:34,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:06:34,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:34,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:34,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:34,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:34,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:34,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:35,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:35,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:35,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:35,102 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:06:35,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:35,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:35,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:35,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:35,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:35,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:35,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:35,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:35,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:35,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:06:35,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:35,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:35,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:35,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:35,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:35,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:35,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:35,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:35,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:35,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:06:35,640 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:35,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:35,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:35,709 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:35,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:35,772 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:35,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:35,823 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:35,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:35,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:06:35,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:35,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:35,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:35,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:36,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:36,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:36,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:36,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:36,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:36,150 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:06:36,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:36,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:36,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:36,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:36,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:36,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:36,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:36,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:36,463 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 13:06:36,477 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 13:06:36,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:06:36,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 13:06:36,950 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:06:36,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 13:06:36,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 13:06:37,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 13:06:37,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 13:06:37,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 13:06:37,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 13:06:37,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 13:06:37,102 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:06:37,128 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 13:06:37,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:06:37,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:06:37,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:06:37,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:06:37,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:06:37,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:06:37,371 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 13:06:37,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:06:37,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:06:37,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:06:37,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:37,464 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:06:37,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:37,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:37,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:37,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:37,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:37,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:37,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:37,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:06:37,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:37,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:06:37,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:37,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:37,772 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:37,793 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:37,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:37,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:37,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:37,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:06:37,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:37,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:06:37,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:37,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:38,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:38,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:38,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:38,084 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:38,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:38,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:06:38,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:38,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:06:38,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:38,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:38,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:38,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:38,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:38,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:38,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:38,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:06:38,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:38,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:06:38,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:38,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:38,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:38,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:38,579 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:38,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:38,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:38,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:06:38,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:38,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:06:38,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:38,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:38,808 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:38,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:38,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:38,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:38,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:38,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:06:38,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:38,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:06:38,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:39,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:39,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:39,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:39,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:39,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:39,124 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:39,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:06:39,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:39,201 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:06:39,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:39,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:39,280 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:39,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:39,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:39,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:39,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:39,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:06:39,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:39,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:06:39,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:39,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:39,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:39,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:39,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:39,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:39,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:39,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:06:39,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:39,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:06:39,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:39,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:39,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:39,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:39,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:39,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:39,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:39,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:06:39,930 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 13:06:39,938 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:06:40,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:06:41,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:06:41,709 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:06:41,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:06:41,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:06:41,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:06:41,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:06:41,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:06:41,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:06:42,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:06:42,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:06:42,089 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 13:06:42,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:06:42,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 16636 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:06:50,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:06:51,624 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:06:52,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:06:52,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:06:54,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:06:54,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:06:54,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:06:54,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:06:54,839 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 13:06:54,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:06:54,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:06:54,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:06:55,007 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 13:06:55,009 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:06:55,012 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:06:55,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:06:55,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:06:55,703 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:06:55,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:06:55,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:06:55,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:06:55,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:06:55,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:06:55,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:06:55,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:06:55,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:06:56,016 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 13:06:56,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:06:56,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:07:03,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:07:04,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:07:05,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:07:05,823 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:07:05,907 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 13:07:05,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:07:05,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:07:06,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:07:06,108 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 13:07:06,111 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:07:06,121 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:07:06,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:07:06,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:07:06,805 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:07:06,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:07:06,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:07:06,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:07:06,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:07:06,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:07:06,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:07:06,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:07:06,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:07:06,982 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 13:07:07,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:07:07,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:07:13,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:07:14,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:07:15,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:07:15,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:07:15,757 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 13:07:15,760 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:07:15,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:07:15,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:07:15,869 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 13:07:15,871 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:07:15,875 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:07:16,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:07:17,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:07:17,503 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:07:17,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:07:17,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:07:17,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:07:17,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:07:17,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:07:17,640 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:07:17,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:07:17,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:07:17,779 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 13:07:17,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:07:17,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:07:26,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:07:27,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:07:27,698 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:07:28,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:07:31,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:07:31,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:07:31,793 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:07:32,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:07:32,052 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 13:07:32,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:07:32,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:07:32,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:07:32,184 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 13:07:32,187 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:07:32,197 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 13:07:32,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:07:32,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 13:07:32,642 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:07:32,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 13:07:32,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 13:07:32,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 13:07:32,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 13:07:32,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 13:07:32,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 13:07:32,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 13:07:32,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:07:32,834 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 13:07:32,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:07:32,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:07:32,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:07:32,987 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:07:33,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:07:33,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:07:33,055 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 13:07:33,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:07:33,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:07:33,102 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:07:33,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:07:33,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:07:33,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:07:33,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:07:33,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:07:33,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:07:33,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:07:33,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:07:33,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:07:33,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:07:33,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:07:33,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:07:33,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:07:33,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:07:33,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:07:33,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:07:33,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:07:33,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:07:33,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:07:33,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:07:33,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:07:33,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:07:33,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:07:33,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:07:33,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:07:33,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:07:33,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:07:33,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:07:34,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:07:34,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:07:34,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:07:34,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:07:34,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:07:34,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:07:34,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:07:34,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:07:34,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:07:34,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:07:34,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:07:34,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:07:34,293 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:07:34,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:07:34,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:07:34,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:07:34,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:07:34,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:07:34,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:07:34,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:07:34,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:07:34,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:07:34,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:07:34,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:07:34,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:07:34,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:07:34,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:07:34,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:07:34,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:07:34,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:07:34,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:07:34,808 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:07:34,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:07:34,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:07:34,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:07:34,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:07:34,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:07:34,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:07:34,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:07:35,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:07:35,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:07:35,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:07:35,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:07:35,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:07:35,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:07:35,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:07:35,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:07:35,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:07:35,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:07:35,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:07:35,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:07:35,308 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:07:35,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:07:35,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:07:35,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:07:35,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:07:35,431 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:07:35,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:07:35,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:07:35,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:07:35,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:07:35,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:07:35,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:07:35,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:07:35,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:07:35,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:07:35,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:07:35,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:07:35,709 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:07:35,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:07:35,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:07:35,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:07:35,803 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 13:07:35,805 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:07:35,812 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:07:36,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:07:36,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:07:36,798 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:07:36,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:07:36,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:07:36,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:07:36,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:07:36,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:07:36,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:07:36,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:07:37,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:07:37,611 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:07:37,614 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.80900001526. 2022-02-08 13:07:37,615 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:07:37,624 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:07:37,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:07:38,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:07:38,489 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:07:38,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:07:38,520 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:07:38,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:07:38,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:07:38,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:07:38,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:07:38,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:07:38,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:07:39,408 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:07:39,414 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.79800009727. 2022-02-08 13:07:39,415 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:07:39,427 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:07:40,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:07:41,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:07:41,302 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:07:41,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:07:41,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:07:41,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:07:41,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:07:41,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:07:41,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:07:41,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:07:41,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:07:42,250 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:07:42,255 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.83900022507. 2022-02-08 13:07:42,256 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:07:42,266 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:07:43,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:07:44,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:07:44,434 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:07:44,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:07:44,464 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:07:44,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:07:44,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:07:44,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:07:44,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:07:44,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:07:44,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:07:45,415 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:07:45,417 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.16100001335. 2022-02-08 13:07:45,418 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:07:45,424 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:07:46,341 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:07:46,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:07:47,187 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:07:47,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:07:47,214 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:07:47,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:07:47,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:07:47,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:07:47,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:07:47,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:07:47,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:07:48,186 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:07:48,187 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.76900005341. 2022-02-08 13:07:48,188 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:07:48,194 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:07:48,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:07:49,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:07:49,500 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:07:49,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:07:49,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:07:49,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:07:49,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:07:49,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:07:49,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:07:49,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:07:49,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 16661 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:07:50,799 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:07:50,802 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.61400008202. 2022-02-08 13:07:50,803 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:07:50,812 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:07:51,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:07:52,275 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:07:52,509 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:07:52,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:07:52,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:07:52,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:07:52,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:07:52,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:07:52,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:07:52,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:07:52,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:07:53,381 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:07:53,381 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.57800006866. 2022-02-08 13:07:53,382 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:07:53,388 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:07:54,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:07:54,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:07:54,957 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:07:54,964 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:07:55,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:07:55,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:07:55,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:07:55,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:07:55,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:07:55,201 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:07:55,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:07:56,049 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:07:56,052 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.66900014877. 2022-02-08 13:07:56,053 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:07:56,071 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:07:56,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:07:57,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:07:57,819 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:07:57,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:07:57,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:07:57,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:07:57,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:07:57,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:07:58,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:07:58,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:07:58,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:07:58,841 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:07:58,844 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.79000020027. ############################################################ # # RAM FootPrint: 4982.4375 # ############################################################ 2022-02-08 13:07:59,009 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 13:07:59,012 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 62 seconds. Reason: Iteration:12 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 16685 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:09:01,048 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 13:09:01,049 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill Loader_GleamBookUsers0__default_VolumeCollection13_7_1644308333.41 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection16_0_1644308340.5 2022-02-08 13:09:06,061 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:09:06,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:09:07,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:09:07,369 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:09:07,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:09:07,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:09:07,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:09:07,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:09:07,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:09:07,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:09:07,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:09:07,740 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:09:07,742 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:09:07,842 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['76033 memcached\n'], []) 2022-02-08 13:09:07,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 13:09:07,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:09:07,931 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['76033 memcached\n'], []) 2022-02-08 13:09:07,933 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:09:07,941 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:09:08,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:09:09,346 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:09:09,529 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:09:09,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:09:09,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:09:09,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:09:09,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:09:09,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:09:09,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:09:09,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:09:09,749 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:09:09,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:09:09,875 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['2018 memcached\n'], []) 2022-02-08 13:09:09,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 13:09:09,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:09:09,964 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['2018 memcached\n'], []) 2022-02-08 13:09:09,971 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:09:10,002 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:09:11,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:09:11,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:09:12,275 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:09:12,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:09:12,303 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:09:12,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:09:12,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:09:12,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:09:12,459 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:09:12,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:09:12,536 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:09:12,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:09:12,628 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['116675 memcached\n'], []) 2022-02-08 13:09:12,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 13:09:12,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:09:12,710 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['116675 memcached\n'], []) 2022-02-08 13:09:12,713 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:09:12,723 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:09:13,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:09:14,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:09:14,594 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:09:14,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:09:14,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:09:14,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:09:14,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:09:14,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:09:14,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:09:14,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:09:14,842 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:09:14,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:09:14,917 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['10241 memcached\n'], []) 2022-02-08 13:09:14,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 13:09:14,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:09:14,983 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['10241 memcached\n'], []) 2022-02-08 13:09:14,983 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:09:14,993 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:09:15,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:09:16,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:09:16,648 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:09:16,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:09:16,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:09:16,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:09:16,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:09:16,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:09:16,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:09:16,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:09:16,884 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:09:16,890 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:09:16,960 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['4126 memcached\n'], []) 2022-02-08 13:09:16,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 13:09:16,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:09:17,046 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['4126 memcached\n'], []) 2022-02-08 13:09:17,048 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:09:17,069 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:09:17,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:09:17,900 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:09:18,045 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:09:18,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:09:18,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:09:18,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:09:18,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:09:18,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:09:18,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:09:18,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:09:18,230 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:09:18,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:09:18,328 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['39282 memcached\n'], []) 2022-02-08 13:09:18,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 13:09:18,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:09:18,391 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['39282 memcached\n'], []) 2022-02-08 13:09:18,394 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:09:18,404 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:09:19,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:09:19,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:09:19,898 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:09:19,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:09:19,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:09:19,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:09:19,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:09:19,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:09:20,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:09:20,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:09:20,072 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:09:20,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:09:20,161 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['102754 memcached\n'], []) 2022-02-08 13:09:20,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 13:09:20,214 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:09:20,243 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['102754 memcached\n'], []) 2022-02-08 13:09:20,246 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:09:20,253 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:09:21,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:09:21,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:09:21,572 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:09:21,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:09:21,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:09:21,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:09:21,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:09:21,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:09:21,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:09:21,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:09:21,776 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:09:21,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:09:21,851 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['117990 memcached\n'], []) 2022-02-08 13:09:21,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 13:09:21,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:09:21,914 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['117990 memcached\n'], []) 2022-02-08 13:09:21,917 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:09:21,927 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:09:22,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:09:23,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:09:23,355 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:09:23,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:09:23,377 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:09:23,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:09:23,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:09:23,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:09:23,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:09:23,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:09:23,538 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:09:23,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:09:23,615 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['55174 memcached\n'], []) 2022-02-08 13:09:23,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 13:09:23,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:09:23,703 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['55174 memcached\n'], []) 2022-02-08 13:09:23,709 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:09:23,723 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:09:23,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:09:24,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:09:24,171 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:09:24,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:09:24,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:09:24,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:09:24,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:09:24,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:09:24,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:09:24,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:09:24,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:09:24,420 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 13:09:24,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:09:24,760 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:09:31,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:09:32,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:09:32,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:09:33,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:09:34,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:09:35,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:09:35,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:09:35,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:09:35,467 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 13:09:35,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:09:35,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:09:35,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:09:35,605 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 13:09:35,608 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:09:35,618 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:09:35,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:09:35,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:09:36,036 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:09:36,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:09:36,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:09:36,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:09:36,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:09:36,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:09:36,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:09:36,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:09:36,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:09:36,221 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 13:09:36,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:09:36,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:09:43,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:09:44,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:09:45,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:09:45,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:09:46,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:09:47,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:09:47,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:09:47,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:09:47,348 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 13:09:47,349 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:09:47,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:09:47,459 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:09:47,483 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 13:09:47,487 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:09:47,503 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 13:09:47,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:09:47,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 13:09:47,971 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:09:47,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 13:09:47,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 13:09:48,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 13:09:48,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 13:09:48,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 13:09:48,094 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 13:09:48,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 13:09:48,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:09:48,173 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 13:09:48,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:09:48,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:09:48,288 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:09:48,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:09:48,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:09:48,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:09:48,374 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 13:09:48,377 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:09:48,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:09:48,418 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:09:48,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:09:48,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:09:48,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:09:48,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:09:48,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:09:48,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:09:48,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:09:48,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:09:48,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:09:48,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:09:48,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:09:48,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:09:48,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:09:48,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:09:48,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:09:48,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:09:48,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:09:48,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:09:48,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:09:48,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:09:48,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:09:48,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:09:48,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:09:49,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:09:49,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:09:49,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:09:49,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:09:49,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:09:49,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:09:49,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:09:49,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:09:49,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:09:49,293 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:09:49,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:09:49,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:09:49,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:09:49,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:09:49,418 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:09:49,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:09:49,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:09:49,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:09:49,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:09:49,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:09:49,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:09:49,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:09:49,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:09:49,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:09:49,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:09:49,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:09:49,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:09:49,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:09:49,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:09:49,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:09:49,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:09:49,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:09:49,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:09:49,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:09:49,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:09:49,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:09:49,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:09:49,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:09:50,002 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:09:50,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:09:50,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:09:50,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:09:50,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 5 queries submitted, 0 failed, 16705 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:09:50,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:09:50,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:09:50,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:09:50,186 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:09:50,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:09:50,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:09:50,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:09:50,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:09:50,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:09:50,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:09:50,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:09:50,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:09:50,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:09:50,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:09:50,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:09:50,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:09:50,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:09:50,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:09:50,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:09:50,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:09:50,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:09:50,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:09:50,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:09:50,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:09:50,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:09:50,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:09:50,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:09:50,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:09:50,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:09:50,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:09:50,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:09:50,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:09:50,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:09:50,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:09:50,986 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 13:09:51,006 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:09:51,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:09:51,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:09:51,903 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:09:51,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:09:52,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:09:52,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:09:52,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:09:52,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:09:52,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:09:52,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:09:52,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:09:52,407 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 13:09:52,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:09:52,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:09:59,124 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:10:00,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:10:01,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:10:01,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:10:01,661 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 13:10:01,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:10:01,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:10:02,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:10:02,150 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 13:10:02,154 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:10:02,180 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 13:10:02,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:10:02,538 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 13:10:02,625 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:10:02,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 13:10:02,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 13:10:02,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 13:10:02,693 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 13:10:02,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 13:10:02,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 13:10:02,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 13:10:02,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:10:02,812 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 13:10:02,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:10:02,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:10:02,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:10:02,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:10:03,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:10:03,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:10:03,049 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 13:10:03,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:10:03,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:10:03,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:10:03,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:03,150 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:10:03,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:03,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:03,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:03,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:03,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:03,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:03,288 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:03,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:03,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:03,349 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:10:03,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:03,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:03,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:03,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:03,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:03,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:03,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:03,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:03,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:03,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:10:03,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:03,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:03,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:03,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:03,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:03,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:03,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:03,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:03,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:03,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:10:03,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:03,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:03,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:03,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:03,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:03,994 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:04,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:04,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:04,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:04,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:10:04,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:04,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:04,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:04,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:04,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:04,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:04,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:04,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:04,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:04,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:10:04,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:04,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:04,477 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:04,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:04,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:04,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:04,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:04,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:04,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:04,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:10:04,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:04,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:04,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:04,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:04,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:04,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:04,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:04,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:04,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:04,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:10:04,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:04,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:05,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:05,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:05,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:05,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:05,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:05,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:05,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:05,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:10:05,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:05,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:05,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:05,303 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:05,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:05,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:05,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:05,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:05,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:05,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:10:05,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:05,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:05,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:05,545 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:05,566 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:05,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:05,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:05,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:05,694 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 13:10:05,707 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:10:05,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:10:06,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:10:06,236 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:10:06,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:10:06,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:10:06,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:10:06,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:10:06,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:10:06,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:10:06,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:10:06,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:10:06,451 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 13:10:06,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:10:06,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:10:13,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:10:13,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:10:14,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:10:15,084 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:10:17,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:10:17,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:10:17,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:10:17,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:10:17,721 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 13:10:17,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:10:17,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:10:17,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:10:17,832 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 13:10:17,835 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:10:17,845 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:10:17,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:10:18,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:10:18,431 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:10:18,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:10:18,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:10:18,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:10:18,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:10:18,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:10:18,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:10:18,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:10:18,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:10:18,671 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 13:10:18,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:10:18,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:10:26,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:10:27,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:10:28,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:10:28,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:10:30,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:10:30,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:10:31,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:10:31,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:10:31,250 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 13:10:31,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:10:31,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:10:31,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:10:31,440 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 13:10:31,443 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:10:31,451 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 13:10:31,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:10:31,831 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 13:10:31,947 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:10:31,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 13:10:31,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 13:10:32,010 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 13:10:32,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 13:10:32,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 13:10:32,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 13:10:32,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 13:10:32,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:10:32,150 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 13:10:32,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:10:32,303 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:10:32,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:10:32,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:10:32,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:10:32,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:10:32,446 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 13:10:32,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:10:32,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:10:32,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:10:32,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:32,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:10:32,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:32,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:32,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:32,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:32,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:32,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:32,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:32,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:32,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:32,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:10:32,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:32,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:32,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:32,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:32,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:32,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:33,002 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:33,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:33,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:33,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:10:33,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:33,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:33,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:33,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:33,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:33,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:33,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:33,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:33,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:33,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:10:33,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:33,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:33,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:33,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:33,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:33,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:33,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:33,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:33,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:33,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:10:33,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:33,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:33,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:33,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:33,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:33,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:33,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:33,897 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:33,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:33,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:10:34,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:34,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:34,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:34,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:34,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:34,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:34,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:34,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:34,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:34,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:10:34,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:34,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:34,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:34,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:34,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:34,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:34,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:34,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:34,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:34,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:10:34,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:34,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:34,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:34,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:34,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:34,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:34,793 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:34,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:34,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:34,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:10:34,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:34,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:34,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:34,994 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:35,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:35,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:35,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:35,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:35,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:35,186 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:10:35,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:35,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:35,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:35,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:35,346 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:35,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:35,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:35,431 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:35,469 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 13:10:35,492 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 13:10:35,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:10:35,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 13:10:35,973 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:10:35,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 13:10:36,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 13:10:36,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 13:10:36,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 13:10:36,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 13:10:36,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 13:10:36,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 13:10:36,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:10:36,204 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 13:10:36,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:10:36,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:10:36,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:10:36,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:10:36,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:10:36,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:10:36,450 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 13:10:36,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:10:36,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:10:36,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:10:36,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:36,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:10:36,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:36,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:36,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:36,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:36,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:36,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:36,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:36,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:10:36,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:36,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:10:36,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:36,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:36,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:36,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:36,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:36,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:36,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:36,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:10:37,010 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:37,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:10:37,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:37,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:37,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:37,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:37,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:37,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:37,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:37,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:10:37,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:37,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:10:37,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:37,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:37,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:37,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:37,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:37,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:37,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:37,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:10:37,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:37,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:10:37,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:37,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:37,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:37,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:37,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:37,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:37,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:37,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:10:37,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:37,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:10:37,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:37,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:37,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:37,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:37,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:37,964 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:37,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:38,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:10:38,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:38,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:10:38,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:38,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:38,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:38,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:38,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:38,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:38,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:38,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:10:38,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:38,323 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:10:38,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:38,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:38,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:38,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:38,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:38,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:38,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:38,528 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:10:38,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:38,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:10:38,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:38,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:38,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:38,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:38,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:38,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:38,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:38,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:10:38,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:38,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:10:38,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:38,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:38,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:38,897 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:38,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:38,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:38,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:38,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:10:39,009 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 13:10:39,022 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:10:39,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:10:40,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:10:40,529 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:10:40,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:10:40,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:10:40,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:10:40,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:10:40,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:10:40,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:10:40,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:10:40,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:10:40,803 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 13:10:40,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:10:40,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:10:48,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:10:49,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 16728 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:10:50,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:10:50,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:10:52,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:10:53,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:10:53,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:10:53,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:10:53,349 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 13:10:53,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:10:53,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:10:53,494 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:10:53,536 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 13:10:53,540 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:10:53,565 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:10:53,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:10:54,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:10:54,615 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:10:54,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:10:54,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:10:54,698 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:10:54,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:10:54,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:10:54,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:10:54,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:10:54,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:10:54,875 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 13:10:55,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:10:55,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:11:02,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:11:03,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:11:04,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:11:04,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:11:04,888 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 13:11:04,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:11:04,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:11:05,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:11:05,216 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 13:11:05,221 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:11:05,252 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:11:05,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:11:06,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:11:06,607 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:11:06,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:11:06,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:11:06,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:11:06,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:11:06,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:11:06,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:11:06,788 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:11:06,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:11:06,836 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 13:11:07,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:11:07,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:11:13,464 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:11:14,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:11:14,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:11:15,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:11:15,365 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 13:11:15,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:11:15,392 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:11:15,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:11:15,486 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 13:11:15,490 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:11:15,507 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:11:16,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:11:16,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:11:17,194 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:11:17,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:11:17,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:11:17,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:11:17,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:11:17,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:11:17,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:11:17,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:11:17,499 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:11:17,536 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 13:11:17,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:11:17,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:11:26,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:11:27,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:11:28,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:11:29,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:11:32,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:11:32,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:11:33,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:11:33,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:11:33,733 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 13:11:33,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:11:33,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:11:33,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:11:33,891 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 13:11:33,892 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:11:33,898 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 13:11:34,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:11:34,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 13:11:34,348 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:11:34,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 13:11:34,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 13:11:34,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 13:11:34,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 13:11:34,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 13:11:34,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 13:11:34,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 13:11:34,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:11:34,555 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 13:11:34,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:11:34,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:11:34,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:11:34,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:11:34,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:11:34,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:11:34,779 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 13:11:34,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:11:34,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:11:34,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:11:34,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:11:34,869 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:11:34,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:11:34,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:11:34,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:11:34,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:11:34,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:11:35,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:11:35,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:11:35,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:11:35,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:11:35,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:11:35,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:11:35,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:11:35,181 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:11:35,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:11:35,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:11:35,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:11:35,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:11:35,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:11:35,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:11:35,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:11:35,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:11:35,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:11:35,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:11:35,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:11:35,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:11:35,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:11:35,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:11:35,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:11:35,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:11:35,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:11:35,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:11:35,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:11:35,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:11:35,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:11:35,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:11:35,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:11:35,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:11:35,900 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:11:35,918 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:11:35,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:11:35,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:11:35,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:11:36,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:11:36,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:11:36,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:11:36,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:11:36,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:11:36,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:11:36,204 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:11:36,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:11:36,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:11:36,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:11:36,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:11:36,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:11:36,349 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:11:36,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:11:36,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:11:36,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:11:36,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:11:36,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:11:36,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:11:36,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:11:36,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:11:36,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:11:36,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:11:36,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:11:36,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:11:36,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:11:36,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:11:36,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:11:36,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:11:36,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:11:36,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:11:36,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:11:36,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:11:36,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:11:36,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:11:36,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:11:36,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:11:37,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:11:37,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:11:37,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:11:37,086 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:11:37,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:11:37,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:11:37,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:11:37,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:11:37,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:11:37,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:11:37,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:11:37,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:11:37,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:11:37,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:11:37,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:11:37,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:11:37,448 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:11:37,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:11:37,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:11:37,529 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 13:11:37,532 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:11:37,540 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:11:38,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:11:38,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:11:38,529 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:11:38,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:11:38,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:11:38,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:11:38,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:11:38,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:11:38,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:11:38,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:11:38,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:11:39,279 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:11:39,283 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.75099992752. 2022-02-08 13:11:39,285 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:11:39,295 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:11:39,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:11:40,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:11:40,168 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:11:40,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:11:40,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:11:40,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:11:40,255 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:11:40,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:11:40,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:11:40,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:11:40,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:11:41,625 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:11:41,628 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.34299993515. 2022-02-08 13:11:41,630 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:11:41,637 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:11:42,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:11:43,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:11:43,311 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:11:43,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:11:43,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:11:43,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:11:43,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:11:43,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:11:43,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:11:43,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:11:43,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:11:44,375 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:11:44,381 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.75200009346. 2022-02-08 13:11:44,382 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:11:44,392 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:11:45,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:11:46,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:11:46,545 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:11:46,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:11:46,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:11:46,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:11:46,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:11:46,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:11:46,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:11:46,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:11:46,869 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:11:47,831 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:11:47,835 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.45200014114. 2022-02-08 13:11:47,835 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:11:47,846 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:11:48,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:11:49,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:11:49,880 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:11:49,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:11:49,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:11:49,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:11:49,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:11:50,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:11:50,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:11:50,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 5 queries submitted, 0 failed, 16753 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:11:50,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:11:50,990 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:11:50,990 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.15499997139. 2022-02-08 13:11:50,990 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:11:51,000 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:11:51,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:11:52,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:11:52,559 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:11:52,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:11:52,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:11:52,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:11:52,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:11:52,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:11:52,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:11:52,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:11:52,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:11:56,155 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:11:56,161 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.16900014877. 2022-02-08 13:11:56,164 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:11:56,177 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:11:57,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:11:57,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:11:57,769 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:11:57,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:11:57,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:11:57,831 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:11:57,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:11:57,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:11:57,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:11:57,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:11:57,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:11:58,575 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:11:58,585 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.42100000381. 2022-02-08 13:11:58,592 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:11:58,625 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:11:59,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:11:59,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:11:59,881 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:11:59,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:11:59,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:11:59,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:12:00,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:12:00,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:12:00,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:12:00,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:12:00,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:12:00,668 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:12:00,671 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.07999992371. 2022-02-08 13:12:00,671 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:12:00,678 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:12:01,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:12:02,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:12:02,586 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:12:02,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:12:02,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:12:02,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:12:02,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:12:02,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:12:02,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:12:02,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:12:02,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:12:03,950 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:12:03,953 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.28200006485. ############################################################ # # RAM FootPrint: 5015.15234375 # ############################################################ 2022-02-08 13:12:04,161 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 13:12:04,161 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 79 seconds. Reason: Iteration:13 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 16778 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:13:23,197 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 13:13:23,198 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 13:13:28,206 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:13:28,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:13:29,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:13:30,053 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:13:30,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:13:30,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:13:30,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:13:30,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:13:30,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:13:30,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:13:30,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:13:30,617 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:13:30,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:13:30,698 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['76759 memcached\n'], []) 2022-02-08 13:13:30,701 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 13:13:30,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:13:30,880 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['76759 memcached\n'], []) 2022-02-08 13:13:30,881 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:13:30,894 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:13:31,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:13:32,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:13:32,588 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:13:32,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:13:32,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:13:32,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:13:32,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:13:32,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:13:32,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:13:32,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:13:33,023 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:13:33,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:13:33,101 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['2776 memcached\n'], []) 2022-02-08 13:13:33,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 13:13:33,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:13:33,176 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['2776 memcached\n'], []) 2022-02-08 13:13:33,180 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:13:33,201 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:13:34,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:13:34,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:13:35,141 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:13:35,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:13:35,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:13:35,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:13:35,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:13:35,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:13:35,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:13:35,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:13:35,450 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:13:35,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:13:35,528 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['117535 memcached\n'], []) 2022-02-08 13:13:35,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 13:13:35,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:13:35,612 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['117535 memcached\n'], []) 2022-02-08 13:13:35,615 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:13:35,631 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:13:36,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:13:37,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:13:37,315 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:13:37,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:13:37,341 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:13:37,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:13:37,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:13:37,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:13:37,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:13:37,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:13:37,543 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:13:37,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:13:37,622 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['11000 memcached\n'], []) 2022-02-08 13:13:37,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 13:13:37,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:13:37,709 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['11000 memcached\n'], []) 2022-02-08 13:13:37,711 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:13:37,720 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:13:38,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:13:39,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:13:39,355 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:13:39,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:13:39,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:13:39,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:13:39,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:13:39,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:13:39,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:13:39,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:13:39,553 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:13:39,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:13:39,627 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['4885 memcached\n'], []) 2022-02-08 13:13:39,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 13:13:39,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:13:39,864 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['4885 memcached\n'], []) 2022-02-08 13:13:39,868 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:13:39,885 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:13:41,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:13:41,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:13:42,098 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:13:42,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:13:42,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:13:42,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:13:42,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:13:42,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:13:42,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:13:42,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:13:42,293 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:13:42,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:13:42,368 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['40036 memcached\n'], []) 2022-02-08 13:13:42,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 13:13:42,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:13:42,430 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['40036 memcached\n'], []) 2022-02-08 13:13:42,431 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:13:42,437 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:13:43,341 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:13:43,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:13:44,107 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:13:44,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:13:44,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:13:44,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:13:44,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:13:44,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:13:44,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:13:44,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:13:44,308 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:13:44,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:13:44,359 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['103477 memcached\n'], []) 2022-02-08 13:13:44,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 13:13:44,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:13:44,430 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['103477 memcached\n'], []) 2022-02-08 13:13:44,433 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:13:44,444 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:13:44,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:13:45,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:13:45,486 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:13:45,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:13:45,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:13:45,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:13:45,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:13:45,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:13:45,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:13:45,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:13:45,631 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:13:45,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:13:45,744 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['118713 memcached\n'], []) 2022-02-08 13:13:45,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 13:13:45,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:13:45,808 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['118713 memcached\n'], []) 2022-02-08 13:13:45,809 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:13:45,812 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:13:46,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:13:47,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:13:47,382 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:13:47,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:13:47,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:13:47,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:13:47,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:13:47,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:13:47,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:13:47,566 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:13:47,598 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:13:47,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:13:47,664 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['55925 memcached\n'], []) 2022-02-08 13:13:47,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 13:13:47,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:13:47,733 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['55925 memcached\n'], []) 2022-02-08 13:13:47,737 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:13:47,766 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:13:47,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:13:48,094 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:13:48,188 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:13:48,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:13:48,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:13:48,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:13:48,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:13:48,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:13:48,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:13:48,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:13:48,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:13:48,368 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 13:13:48,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:13:48,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 16801 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:13:55,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:13:56,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:13:56,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:13:57,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:13:58,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:13:58,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:13:59,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:13:59,293 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:13:59,321 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 13:13:59,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:13:59,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:13:59,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:13:59,453 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 13:13:59,459 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:13:59,477 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:13:59,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:13:59,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:14:00,005 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:14:00,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:14:00,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:14:00,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:14:00,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:14:00,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:14:00,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:14:00,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:14:00,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:14:00,221 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 13:14:00,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:14:00,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:14:07,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:14:08,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:14:09,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:14:09,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:14:11,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:14:11,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:14:11,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:14:11,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:14:11,665 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 13:14:11,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:14:11,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:14:11,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:14:11,802 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 13:14:11,805 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:14:11,815 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 13:14:11,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:14:12,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 13:14:12,267 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:14:12,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 13:14:12,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 13:14:12,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 13:14:12,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 13:14:12,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 13:14:12,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 13:14:12,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 13:14:12,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:14:12,447 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 13:14:12,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:14:12,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:14:12,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:14:12,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:14:12,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:14:12,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:14:12,778 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 13:14:12,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:14:12,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:14:12,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:14:12,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:12,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:14:12,915 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:12,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:12,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:13,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:13,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:13,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:13,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:13,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:13,183 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:13,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:14:13,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:13,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:13,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:13,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:13,341 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:13,364 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:13,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:13,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:13,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:13,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:14:13,492 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:13,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:13,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:13,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:13,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:13,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:13,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:13,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:13,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:13,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:14:13,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:13,788 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:13,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:13,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:13,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:13,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:13,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:13,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:13,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:13,994 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:14:14,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:14,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:14,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:14,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:14,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:14,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:14,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:14,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:14,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:14,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:14:14,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:14,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:14,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:14,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:14,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:14,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:14,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:14,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:14,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:14,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:14:14,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:14,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:14,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:14,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:14,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:14,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:14,772 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:14,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:14,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:14,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:14:14,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:14,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:14,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:14,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:14,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:15,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:15,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:15,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:15,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:15,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:14:15,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:15,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:15,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:15,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:15,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:15,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:15,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:15,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:15,364 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:15,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:14:15,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:15,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:15,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:15,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:15,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:15,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:15,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:15,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:15,607 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 13:14:15,618 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:14:15,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:14:16,028 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:14:16,369 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:14:16,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:14:16,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:14:16,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:14:16,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:14:16,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:14:16,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:14:16,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:14:16,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:14:16,844 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 13:14:16,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:14:17,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:14:23,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:14:24,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:14:25,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:14:25,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:14:26,107 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 13:14:26,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:14:26,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:14:26,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:14:26,523 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 13:14:26,526 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:14:26,536 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 13:14:26,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:14:26,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 13:14:27,051 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:14:27,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 13:14:27,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 13:14:27,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 13:14:27,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 13:14:27,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 13:14:27,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 13:14:27,186 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 13:14:27,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:14:27,233 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 13:14:27,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:14:27,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:14:27,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:14:27,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:14:27,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:14:27,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:14:27,423 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 13:14:27,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:14:27,459 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:14:27,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:14:27,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:27,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:14:27,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:27,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:27,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:27,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:27,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:27,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:27,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:27,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:27,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:27,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:14:27,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:27,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:27,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:27,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:27,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:27,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:28,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:28,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:28,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:28,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:14:28,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:28,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:28,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:28,201 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:28,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:28,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:28,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:28,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:28,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:28,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:14:28,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:28,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:28,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:28,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:28,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:28,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:28,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:28,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:28,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:28,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:14:28,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:28,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:28,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:28,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:28,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:28,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:28,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:28,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:28,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:28,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:14:28,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:28,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:28,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:28,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:29,017 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:29,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:29,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:29,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:29,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:29,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:14:29,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:29,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:29,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:29,303 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:29,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:29,346 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:29,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:29,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:29,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:29,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:14:29,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:29,522 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:29,545 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:29,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:29,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:29,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:29,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:29,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:29,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:29,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:14:29,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:29,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:29,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:29,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:29,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:29,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:29,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:29,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:30,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:30,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:14:30,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:30,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:30,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:30,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:30,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:30,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:30,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:30,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:30,302 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 13:14:30,308 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:14:30,448 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:14:30,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:14:30,855 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:14:30,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:14:30,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:14:30,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:14:30,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:14:30,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:14:30,964 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:14:30,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:14:31,020 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:14:31,046 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 13:14:31,181 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:14:31,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:14:37,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:14:38,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:14:38,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:14:39,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:14:41,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:14:41,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:14:41,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:14:41,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:14:42,013 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 13:14:42,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:14:42,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:14:42,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:14:42,128 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 13:14:42,132 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:14:42,151 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:14:42,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:14:42,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:14:42,706 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:14:42,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:14:42,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:14:42,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:14:42,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:14:42,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:14:42,854 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:14:42,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:14:42,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:14:42,931 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 13:14:43,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:14:43,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 16826 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:14:51,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:14:51,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:14:52,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:14:52,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:14:54,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:14:54,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:14:55,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:14:55,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:14:55,219 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 13:14:55,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:14:55,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:14:55,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:14:55,381 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 13:14:55,384 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:14:55,401 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 13:14:55,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:14:55,808 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 13:14:55,940 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:14:55,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 13:14:55,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 13:14:56,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 13:14:56,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 13:14:56,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 13:14:56,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 13:14:56,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 13:14:56,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:14:56,197 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 13:14:56,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:14:56,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:14:56,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:14:56,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:14:56,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:14:56,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:14:56,486 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 13:14:56,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:14:56,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:14:56,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:14:56,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:56,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:14:56,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:56,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:56,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:56,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:56,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:56,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:56,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:56,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:14:56,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:56,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:14:56,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:57,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:57,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:57,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:57,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:57,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:57,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:57,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:14:57,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:57,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:14:57,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:57,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:57,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:57,443 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:57,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:57,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:57,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:57,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:14:57,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:57,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:14:57,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:57,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:57,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:57,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:57,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:57,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:57,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:57,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:14:57,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:57,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:14:57,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:58,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:58,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:58,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:58,094 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:58,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:58,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:58,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:14:58,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:58,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:14:58,275 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:58,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:58,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:58,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:58,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:58,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:58,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:58,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:14:58,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:58,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:14:58,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:58,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:58,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:58,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:58,701 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:58,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:58,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:58,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:14:58,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:58,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:14:58,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:58,918 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:58,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:58,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:59,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:59,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:59,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:59,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:14:59,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:59,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:14:59,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:59,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:59,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:59,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:59,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:59,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:59,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:59,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:14:59,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:59,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:14:59,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:59,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:59,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:59,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:59,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:59,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:59,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:59,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:14:59,744 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 13:14:59,763 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 13:14:59,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:15:00,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 13:15:00,285 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:15:00,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 13:15:00,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 13:15:00,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 13:15:00,341 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 13:15:00,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 13:15:00,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 13:15:00,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 13:15:00,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:15:00,459 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 13:15:00,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:15:00,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:15:00,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:15:00,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:15:00,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:15:00,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:15:00,727 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 13:15:00,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:15:00,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:15:00,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:15:00,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:00,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:15:00,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:00,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:00,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:00,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:00,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:00,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:00,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:00,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:01,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:01,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:15:01,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:01,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:01,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:01,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:01,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:01,150 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:01,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:01,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:01,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:01,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:15:01,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:01,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:01,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:01,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:01,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:01,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:01,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:01,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:01,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:01,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:15:01,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:01,538 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:01,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:01,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:01,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:01,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:01,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:01,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:01,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:01,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:15:01,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:01,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:01,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:01,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:01,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:01,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:01,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:01,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:01,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:01,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:15:01,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:02,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:02,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:02,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:02,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:02,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:02,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:02,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:02,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:02,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:15:02,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:02,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:02,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:02,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:02,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:02,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:02,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:02,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:02,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:02,459 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:15:02,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:02,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:02,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:02,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:02,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:02,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:02,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:02,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:02,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:02,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:15:02,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:02,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:02,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:02,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:02,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:02,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:02,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:02,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:02,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:02,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:15:02,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:02,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:03,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:03,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:03,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:03,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:03,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:03,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:03,164 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 13:15:03,171 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:15:03,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:15:03,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:15:03,934 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:15:03,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:15:03,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:15:03,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:15:04,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:15:04,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:15:04,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:15:04,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:15:04,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:15:04,164 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 13:15:04,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:15:04,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:15:11,793 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:15:12,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:15:13,528 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:15:14,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:15:15,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:15:15,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:15:16,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:15:16,122 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:15:16,206 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 13:15:16,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:15:16,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:15:16,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:15:16,394 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 13:15:16,401 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:15:16,424 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:15:17,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:15:17,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:15:18,111 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:15:18,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:15:18,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:15:18,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:15:18,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:15:18,255 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:15:18,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:15:18,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:15:18,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:15:18,387 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 13:15:18,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:15:18,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:15:25,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:15:27,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:15:27,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:15:28,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:15:28,122 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 13:15:28,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:15:28,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:15:28,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:15:28,246 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 13:15:28,250 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:15:28,266 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:15:28,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:15:29,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:15:29,516 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:15:29,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:15:29,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:15:29,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:15:29,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:15:29,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:15:29,624 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:15:29,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:15:29,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:15:29,729 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 13:15:29,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:15:29,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:15:36,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:15:37,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:15:38,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:15:38,538 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:15:38,573 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 13:15:38,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:15:38,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:15:38,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:15:38,687 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 13:15:38,690 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:15:38,703 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:15:40,456 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:15:40,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:15:41,053 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:15:41,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:15:41,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:15:41,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:15:41,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:15:41,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:15:41,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:15:41,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:15:41,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:15:41,328 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 13:15:41,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:15:41,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:15:49,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 5 queries submitted, 0 failed, 16850 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:15:50,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:15:51,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:15:51,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:15:55,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:15:55,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:15:56,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:15:56,308 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:15:56,358 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 13:15:56,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:15:56,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:15:56,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:15:56,506 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 13:15:56,509 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:15:56,536 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 13:15:56,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:15:56,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 13:15:56,980 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:15:56,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 13:15:57,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 13:15:57,020 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 13:15:57,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 13:15:57,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 13:15:57,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 13:15:57,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 13:15:57,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:15:57,157 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 13:15:57,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:15:57,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:15:57,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:15:57,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:15:57,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:15:57,349 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:15:57,374 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 13:15:57,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:15:57,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:15:57,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:15:57,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:57,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:15:57,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:57,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:57,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:57,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:57,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:57,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:57,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:57,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:15:57,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:57,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:15:57,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:57,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:57,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:57,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:57,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:57,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:57,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:57,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:15:57,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:57,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:15:57,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:58,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:58,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:58,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:58,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:58,102 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:58,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:58,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:15:58,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:58,201 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:15:58,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:58,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:58,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:58,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:58,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:58,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:58,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:58,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:15:58,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:58,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:15:58,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:58,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:58,515 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:58,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:58,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:58,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:58,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:58,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:15:58,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:58,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:15:58,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:58,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:58,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:58,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:58,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:58,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:58,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:58,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:15:58,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:58,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:15:58,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:59,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:59,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:59,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:59,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:59,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:59,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:59,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:15:59,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:59,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:15:59,214 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:59,234 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:59,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:59,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:59,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:59,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:59,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:59,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:15:59,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:59,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:15:59,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:59,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:59,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:59,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:59,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:59,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:59,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:59,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:15:59,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:59,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:15:59,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:59,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:59,757 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:59,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:59,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:59,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:59,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:59,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:15:59,905 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 13:15:59,911 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:15:59,924 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:16:00,492 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:16:00,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:16:00,976 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:16:00,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:16:01,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:16:01,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:16:01,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:16:01,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:16:01,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:16:01,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:16:01,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:16:01,691 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:16:01,694 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.78399991989. 2022-02-08 13:16:01,694 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:16:01,700 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:16:03,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:16:03,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:16:03,641 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:16:03,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:16:03,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:16:03,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:16:03,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:16:03,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:16:03,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:16:03,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:16:03,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:16:04,921 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:16:04,924 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.22800016403. 2022-02-08 13:16:04,924 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:16:04,937 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:16:06,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:16:06,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:16:06,798 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:16:06,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:16:06,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:16:06,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:16:06,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:16:06,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:16:06,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:16:07,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:16:07,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:16:07,711 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:16:07,713 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.78900003433. 2022-02-08 13:16:07,713 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:16:07,723 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:16:08,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:16:09,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:16:09,756 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:16:09,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:16:09,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:16:09,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:16:09,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:16:09,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:16:09,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:16:09,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:16:10,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:16:10,842 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:16:10,848 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.13400006294. 2022-02-08 13:16:10,857 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:16:10,881 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:16:11,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:16:12,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:16:12,546 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:16:12,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:16:12,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:16:12,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:16:12,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:16:12,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:16:12,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:16:12,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:16:12,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:16:13,477 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:16:13,483 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.62600016594. 2022-02-08 13:16:13,483 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:16:13,496 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:16:14,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:16:14,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:16:14,881 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:16:14,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:16:14,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:16:14,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:16:14,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:16:15,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:16:15,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:16:15,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:16:15,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:16:16,832 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:16:16,838 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.35199999809. 2022-02-08 13:16:16,841 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:16:16,849 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:16:17,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:16:18,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:16:18,690 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:16:18,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:16:18,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:16:18,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:16:18,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:16:18,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:16:18,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:16:18,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:16:18,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:16:19,565 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:16:19,575 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.73399996758. 2022-02-08 13:16:19,578 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:16:19,588 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:16:20,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:16:20,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:16:20,704 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:16:20,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:16:20,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:16:20,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:16:20,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:16:20,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:16:20,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:16:20,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:16:20,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:16:22,039 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:16:22,049 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.47099995613. 2022-02-08 13:16:22,052 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:16:22,065 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:16:23,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:16:23,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:16:23,944 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:16:23,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:16:23,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:16:24,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:16:24,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:16:24,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:16:24,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:16:24,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:16:24,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:16:25,058 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:16:25,061 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.00900006294. ############################################################ # # RAM FootPrint: 5001.4140625 # ############################################################ 2022-02-08 13:16:25,201 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 13:16:25,204 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 84 seconds. Reason: Iteration:14 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 16877 passed, 0 rejected, 0 cancelled, 0 timeout Loader_GleamBookUsers0__default_VolumeCollection15_1_1644308339.49 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection7_-1_1644308341.51 2022-02-08 13:17:49,236 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 13:17:49,239 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 5 queries submitted, 0 failed, 16899 passed, 0 rejected, 0 cancelled, 0 timeout Loader_GleamBookUsers0__default_VolumeCollection17_3_1644308337.46 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection6_-2_1644308342.52 2022-02-08 13:17:54,247 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:17:55,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:17:55,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:17:56,223 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:17:56,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:17:56,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:17:56,341 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:17:56,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:17:56,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:17:56,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:17:56,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:17:56,601 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:17:56,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:17:56,687 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['77551 memcached\n'], []) 2022-02-08 13:17:56,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 13:17:56,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:17:56,773 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['77551 memcached\n'], []) 2022-02-08 13:17:56,776 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:17:56,783 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:17:57,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:17:58,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:17:58,319 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:17:58,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:17:58,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:17:58,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:17:58,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:17:58,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:17:58,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:17:58,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:17:58,549 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:17:58,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:17:58,648 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['3553 memcached\n'], []) 2022-02-08 13:17:58,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 13:17:58,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:17:58,733 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['3553 memcached\n'], []) 2022-02-08 13:17:58,736 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:17:58,746 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:17:59,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:18:00,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:18:00,941 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:18:00,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:18:00,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:18:01,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:18:01,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:18:01,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:18:01,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:18:01,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:18:01,155 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:18:01,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:18:01,232 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['118414 memcached\n'], []) 2022-02-08 13:18:01,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 13:18:01,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:18:01,301 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['118414 memcached\n'], []) 2022-02-08 13:18:01,302 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:18:01,305 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:18:02,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:18:02,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:18:02,926 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:18:02,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:18:02,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:18:02,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:18:03,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:18:03,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:18:03,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:18:03,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:18:03,118 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:18:03,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:18:03,196 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['11773 memcached\n'], []) 2022-02-08 13:18:03,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 13:18:03,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:18:03,263 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['11773 memcached\n'], []) 2022-02-08 13:18:03,266 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:18:03,273 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:18:04,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:18:05,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:18:05,345 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:18:05,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:18:05,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:18:05,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:18:05,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:18:05,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:18:05,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:18:05,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:18:05,539 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:18:05,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:18:05,605 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['5659 memcached\n'], []) 2022-02-08 13:18:05,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 13:18:05,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:18:05,671 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['5659 memcached\n'], []) 2022-02-08 13:18:05,673 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:18:05,678 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:18:06,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:18:07,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:18:07,322 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:18:07,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:18:07,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:18:07,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:18:07,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:18:07,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:18:07,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:18:07,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:18:07,480 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:18:07,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:18:07,543 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['40810 memcached\n'], []) 2022-02-08 13:18:07,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 13:18:07,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:18:07,604 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['40810 memcached\n'], []) 2022-02-08 13:18:07,605 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:18:07,608 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:18:08,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:18:09,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:18:09,316 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:18:09,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:18:09,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:18:09,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:18:09,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:18:09,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:18:09,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:18:09,443 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:18:09,463 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:18:09,464 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:18:09,516 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['104276 memcached\n'], []) 2022-02-08 13:18:09,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 13:18:09,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:18:09,576 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['104276 memcached\n'], []) 2022-02-08 13:18:09,578 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:18:09,585 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:18:10,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:18:10,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:18:11,082 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:18:11,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:18:11,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:18:11,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:18:11,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:18:11,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:18:11,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:18:11,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:18:11,240 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:18:11,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:18:11,299 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['119460 memcached\n'], []) 2022-02-08 13:18:11,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 13:18:11,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:18:11,357 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['119460 memcached\n'], []) 2022-02-08 13:18:11,358 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:18:11,365 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:18:12,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:18:12,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:18:12,835 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:18:12,836 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:18:12,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:18:12,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:18:12,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:18:13,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:18:13,020 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:18:13,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:18:13,082 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:18:13,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:18:13,183 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['56773 memcached\n'], []) 2022-02-08 13:18:13,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 13:18:13,219 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:18:13,244 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['56773 memcached\n'], []) 2022-02-08 13:18:13,246 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:18:13,253 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:18:13,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:18:13,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:18:13,690 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:18:13,693 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:18:13,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:18:13,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:18:13,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:18:13,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:18:13,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:18:13,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:18:13,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:18:13,859 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 13:18:13,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:18:13,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:18:20,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:18:21,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:18:21,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:18:22,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:18:23,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:18:24,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:18:24,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:18:24,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:18:24,538 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 13:18:24,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:18:24,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:18:24,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:18:24,655 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 13:18:24,658 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:18:24,668 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:18:24,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:18:24,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:18:25,063 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:18:25,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:18:25,084 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:18:25,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:18:25,124 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:18:25,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:18:25,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:18:25,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:18:25,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:18:25,213 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 13:18:25,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:18:25,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:18:32,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:18:33,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:18:34,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:18:34,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:18:36,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:18:36,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:18:36,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:18:36,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:18:36,750 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 13:18:36,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:18:36,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:18:36,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:18:36,891 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 13:18:36,894 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:18:36,903 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 13:18:36,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:18:37,234 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 13:18:37,322 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:18:37,323 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 13:18:37,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 13:18:37,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 13:18:37,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 13:18:37,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 13:18:37,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 13:18:37,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 13:18:37,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:18:37,479 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 13:18:37,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:18:37,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:18:37,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:18:37,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:18:37,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:18:37,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:18:37,710 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 13:18:37,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:18:37,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:18:37,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:18:37,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:37,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:18:37,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:37,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:37,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:37,897 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:37,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:37,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:37,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:37,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:38,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:38,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:18:38,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:38,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:38,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:38,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:38,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:38,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:38,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:38,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:38,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:38,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:18:38,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:38,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:38,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:38,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:38,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:38,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:38,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:38,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:38,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:38,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:18:38,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:38,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:38,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:38,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:38,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:38,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:38,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:38,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:38,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:38,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:18:38,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:38,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:38,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:38,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:38,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:38,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:38,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:38,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:38,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:38,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:18:38,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:38,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:39,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:39,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:39,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:39,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:39,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:39,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:39,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:39,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:18:39,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:39,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:39,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:39,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:39,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:39,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:39,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:39,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:39,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:39,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:18:39,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:39,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:39,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:39,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:39,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:39,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:39,566 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:39,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:39,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:39,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:18:39,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:39,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:39,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:39,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:39,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:39,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:39,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:39,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:39,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:39,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:18:39,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:39,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:39,900 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:39,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:39,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:39,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:39,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:40,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:40,023 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 13:18:40,029 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:18:40,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:18:40,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:18:41,334 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:18:41,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:18:41,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:18:41,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:18:41,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:18:41,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:18:41,831 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:18:41,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:18:42,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:18:42,153 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 13:18:42,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:18:42,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:18:48,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:18:49,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 16920 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:18:50,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:18:50,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:18:50,832 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 13:18:50,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:18:50,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:18:51,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:18:51,181 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 13:18:51,184 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:18:51,191 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 13:18:51,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:18:51,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 13:18:51,645 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:18:51,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 13:18:51,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 13:18:51,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 13:18:51,698 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 13:18:51,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 13:18:51,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 13:18:51,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 13:18:51,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:18:51,801 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 13:18:51,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:18:51,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:18:51,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:18:51,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:18:51,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:18:51,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:18:52,013 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 13:18:52,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:18:52,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:18:52,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:18:52,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:52,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:18:52,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:52,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:52,186 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:52,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:52,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:52,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:52,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:52,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:18:52,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:52,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:18:52,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:52,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:52,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:52,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:52,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:52,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:52,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:52,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:18:52,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:52,566 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:18:52,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:52,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:52,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:52,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:52,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:52,693 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:52,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:52,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:18:52,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:52,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:18:52,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:52,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:52,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:52,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:52,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:52,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:52,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:52,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:18:52,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:52,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:18:53,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:53,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:53,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:53,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:53,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:53,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:53,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:53,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:18:53,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:53,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:18:53,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:53,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:53,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:53,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:53,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:53,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:53,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:53,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:18:53,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:53,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:18:53,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:53,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:53,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:53,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:53,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:53,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:53,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:53,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:18:53,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:53,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:18:53,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:53,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:53,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:53,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:53,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:53,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:53,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:53,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:18:53,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:53,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:18:53,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:53,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:53,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:53,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:53,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:53,994 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:54,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:54,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:18:54,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:54,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:18:54,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:54,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:54,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:54,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:54,186 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:54,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:54,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:54,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:18:54,266 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 13:18:54,269 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:18:54,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:18:54,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:18:54,726 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:18:54,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:18:54,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:18:54,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:18:54,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:18:54,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:18:54,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:18:54,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:18:54,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:18:54,903 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 13:18:54,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:18:54,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:19:01,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:19:02,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:19:03,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:19:03,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:19:06,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:19:06,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:19:06,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:19:06,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:19:06,974 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 13:19:06,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:19:07,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:19:07,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:19:07,095 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 13:19:07,095 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:19:07,098 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:19:07,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:19:07,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:19:07,788 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:19:07,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:19:07,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:19:07,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:19:07,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:19:07,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:19:07,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:19:07,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:19:08,102 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:19:08,165 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 13:19:08,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:19:08,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:19:16,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:19:17,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:19:17,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:19:18,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:19:20,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:19:20,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:19:20,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:19:20,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:19:20,815 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 13:19:20,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:19:20,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:19:21,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:19:21,036 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 13:19:21,039 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:19:21,045 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 13:19:21,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:19:21,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 13:19:21,510 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:19:21,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 13:19:21,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 13:19:21,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 13:19:21,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 13:19:21,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 13:19:21,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 13:19:21,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 13:19:21,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:19:21,664 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 13:19:21,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:19:21,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:19:21,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:19:21,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:19:21,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:19:21,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:19:21,861 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 13:19:21,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:19:21,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:19:21,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:19:21,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:21,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:19:21,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:22,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:22,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:22,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:22,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:22,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:22,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:22,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:22,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:22,209 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:19:22,234 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:22,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:22,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:22,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:22,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:22,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:22,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:22,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:22,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:22,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:19:22,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:22,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:22,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:22,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:22,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:22,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:22,624 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:22,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:22,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:22,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:19:22,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:22,742 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:22,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:22,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:22,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:22,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:22,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:22,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:22,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:22,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:19:22,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:22,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:23,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:23,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:23,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:23,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:23,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:23,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:23,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:23,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:19:23,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:23,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:23,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:23,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:23,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:23,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:23,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:23,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:23,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:23,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:19:23,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:23,499 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:23,520 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:23,548 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:23,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:23,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:23,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:23,640 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:23,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:23,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:19:23,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:23,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:23,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:23,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:23,808 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:23,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:23,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:23,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:23,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:23,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:19:23,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:23,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:24,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:24,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:24,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:24,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:24,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:24,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:24,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:24,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:19:24,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:24,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:24,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:24,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:24,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:24,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:24,323 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:24,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:24,365 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 13:19:24,377 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 13:19:24,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:19:24,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 13:19:24,835 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:19:24,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 13:19:24,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 13:19:24,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 13:19:24,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 13:19:24,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 13:19:24,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 13:19:24,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 13:19:24,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:19:24,992 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 13:19:25,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:19:25,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:19:25,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:19:25,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:19:25,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:19:25,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:19:25,155 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 13:19:25,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:19:25,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:19:25,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:19:25,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:25,234 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:19:25,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:25,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:25,316 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:25,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:25,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:25,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:25,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:25,443 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:19:25,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:25,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:19:25,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:25,528 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:25,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:25,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:25,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:25,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:25,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:25,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:19:25,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:25,693 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:19:25,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:25,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:25,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:25,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:25,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:25,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:25,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:25,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:19:25,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:25,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:19:25,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:25,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:25,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:26,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:26,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:26,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:26,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:26,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:19:26,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:26,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:19:26,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:26,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:26,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:26,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:26,227 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:26,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:26,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:26,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:19:26,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:26,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:19:26,364 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:26,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:26,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:26,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:26,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:26,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:26,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:26,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:19:26,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:26,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:19:26,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:26,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:26,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:26,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:26,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:26,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:26,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:26,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:19:26,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:26,849 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:19:26,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:26,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:26,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:26,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:26,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:26,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:26,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:27,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:19:27,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:27,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:19:27,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:27,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:27,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:27,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:27,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:27,181 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:27,201 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:27,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:19:27,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:27,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:19:27,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:27,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:27,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:27,346 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:27,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:27,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:27,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:27,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:19:27,437 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 13:19:27,447 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:19:28,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:19:28,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:19:29,095 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:19:29,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:19:29,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:19:29,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:19:29,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:19:29,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:19:29,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:19:29,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:19:29,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:19:29,316 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 13:19:29,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:19:29,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:19:37,897 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:19:38,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:19:39,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:19:40,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:19:41,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:19:41,841 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:19:41,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:19:42,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:19:42,140 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 13:19:42,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:19:42,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:19:42,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:19:42,270 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 13:19:42,273 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:19:42,278 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:19:42,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:19:42,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:19:42,849 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:19:42,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:19:42,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:19:43,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:19:43,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:19:43,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:19:43,094 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:19:43,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:19:43,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:19:43,197 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 13:19:43,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:19:43,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 16945 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:19:50,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:19:51,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:19:52,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:19:52,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:19:52,790 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 13:19:52,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:19:52,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:19:52,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:19:53,088 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 13:19:53,092 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:19:53,098 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:19:53,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:19:54,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:19:54,303 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:19:54,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:19:54,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:19:54,346 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:19:54,364 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:19:54,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:19:54,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:19:54,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:19:54,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:19:54,474 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 13:19:54,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:19:54,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:20:01,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:20:02,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:20:02,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:20:03,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:20:03,237 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 13:20:03,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:20:03,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:20:03,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:20:03,397 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 13:20:03,398 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:20:03,405 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:20:04,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:20:05,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:20:05,321 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:20:05,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:20:05,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:20:05,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:20:05,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:20:05,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:20:05,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:20:05,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:20:05,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:20:05,834 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 13:20:05,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:20:05,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:20:14,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:20:19,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:20:20,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:20:21,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:20:23,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:20:24,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:20:24,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:20:24,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:20:24,697 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 13:20:24,698 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:20:24,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:20:24,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:20:24,855 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 13:20:24,857 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:20:24,859 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 13:20:24,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:20:25,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 13:20:25,280 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:20:25,280 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 13:20:25,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 13:20:25,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 13:20:25,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 13:20:25,349 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 13:20:25,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 13:20:25,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 13:20:25,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:20:25,430 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 13:20:25,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:20:25,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:20:25,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:20:25,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:20:25,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:20:25,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:20:25,625 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 13:20:25,627 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:20:25,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:20:25,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:20:25,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:20:25,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:20:25,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:20:25,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:20:25,793 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:20:25,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:20:25,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:20:25,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:20:25,900 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:20:25,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:20:25,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:20:25,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:20:26,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:20:26,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:20:26,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:20:26,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:20:26,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:20:26,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:20:26,117 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:20:26,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:20:26,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:20:26,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:20:26,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:20:26,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:20:26,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:20:26,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:20:26,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:20:26,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:20:26,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:20:26,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:20:26,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:20:26,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:20:26,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:20:26,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:20:26,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:20:26,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:20:26,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:20:26,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:20:26,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:20:26,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:20:26,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:20:26,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:20:26,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:20:26,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:20:26,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:20:26,701 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:20:26,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:20:26,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:20:26,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:20:26,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:20:26,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:20:26,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:20:26,869 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:20:26,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:20:26,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:20:26,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:20:26,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:20:26,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:20:27,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:20:27,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:20:27,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:20:27,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:20:27,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:20:27,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:20:27,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:20:27,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:20:27,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:20:27,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:20:27,239 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:20:27,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:20:27,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:20:27,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:20:27,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:20:27,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:20:27,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:20:27,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:20:27,423 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:20:27,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:20:27,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:20:27,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:20:27,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:20:27,528 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:20:27,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:20:27,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:20:27,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:20:27,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:20:27,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:20:27,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:20:27,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:20:27,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:20:27,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:20:27,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:20:27,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:20:27,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:20:27,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:20:27,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:20:27,854 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:20:27,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:20:27,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:20:27,915 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:20:27,934 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 13:20:27,937 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:20:27,944 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:20:28,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:20:28,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:20:29,096 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:20:29,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:20:29,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:20:29,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:20:29,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:20:29,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:20:29,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:20:29,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:20:29,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:20:29,812 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:20:29,813 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.87699985504. 2022-02-08 13:20:29,815 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:20:29,821 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:20:30,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:20:31,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:20:31,359 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:20:31,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:20:31,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:20:31,418 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:20:31,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:20:31,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:20:31,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:20:31,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:20:31,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:20:32,174 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:20:32,177 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.36100006104. 2022-02-08 13:20:32,177 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:20:32,184 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:20:32,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:20:33,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:20:33,296 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:20:33,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:20:33,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:20:33,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:20:33,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:20:33,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:20:33,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:20:33,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:20:33,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:20:34,134 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:20:34,135 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.95899987221. 2022-02-08 13:20:34,137 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:20:34,141 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:20:35,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:20:35,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:20:35,740 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:20:35,742 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:20:35,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:20:35,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:20:35,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:20:35,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:20:35,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:20:35,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:20:35,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:20:36,779 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:20:36,785 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.64800000191. 2022-02-08 13:20:36,786 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:20:36,792 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:20:37,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:20:38,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:20:38,477 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:20:38,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:20:38,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:20:38,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:20:38,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:20:38,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:20:38,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:20:38,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:20:38,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:20:39,447 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:20:39,447 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.66100001335. 2022-02-08 13:20:39,447 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:20:39,448 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:20:39,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:20:40,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:20:40,223 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:20:40,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:20:40,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:20:40,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:20:40,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:20:40,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:20:40,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:20:40,377 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:20:40,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:20:41,549 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:20:41,552 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.10500001907. 2022-02-08 13:20:41,553 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:20:41,563 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:20:42,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:20:42,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:20:43,020 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:20:43,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:20:43,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:20:43,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:20:43,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:20:43,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:20:43,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:20:43,150 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:20:43,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:20:43,796 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:20:43,796 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.24300003052. 2022-02-08 13:20:43,796 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:20:43,799 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:20:44,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:20:44,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:20:44,888 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:20:44,890 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:20:44,918 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:20:44,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:20:44,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:20:45,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:20:45,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:20:45,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:20:45,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:20:47,832 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:20:47,836 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.03700017929. 2022-02-08 13:20:47,838 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:20:47,851 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:20:48,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:20:49,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:20:49,622 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:20:49,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:20:49,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:20:49,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:20:49,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:20:49,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:20:49,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:20:49,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:20:49,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 16966 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:20:52,971 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:20:52,974 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.13499999046. ############################################################ # # RAM FootPrint: 4953.80859375 # ############################################################ 2022-02-08 13:20:53,055 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 13:20:53,055 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 96 seconds. Reason: Iteration:15 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 16991 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:22:29,075 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 13:22:29,076 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill Loader_GleamBookUsers0__default_VolumeCollection12_4_1644308336.45 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection5_-3_1644308343.54 2022-02-08 13:22:34,091 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:22:35,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:22:35,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:22:36,051 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:22:36,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:22:36,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:22:36,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:22:36,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:22:36,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:22:36,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:22:36,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:22:36,380 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:22:36,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:22:36,434 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['78302 memcached\n'], []) 2022-02-08 13:22:36,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 13:22:36,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:22:36,559 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['78302 memcached\n'], []) 2022-02-08 13:22:36,562 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:22:36,569 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:22:37,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:22:38,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:22:38,354 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:22:38,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:22:38,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:22:38,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:22:38,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:22:38,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:22:38,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:22:38,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:22:38,598 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:22:38,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:22:38,683 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['4364 memcached\n'], []) 2022-02-08 13:22:38,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 13:22:38,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:22:38,757 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['4364 memcached\n'], []) 2022-02-08 13:22:38,759 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:22:38,767 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:22:39,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:22:40,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:22:40,642 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:22:40,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:22:40,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:22:40,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:22:40,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:22:40,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:22:40,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:22:40,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:22:40,957 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:22:40,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:22:41,042 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['119328 memcached\n'], []) 2022-02-08 13:22:41,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 13:22:41,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:22:41,115 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['119328 memcached\n'], []) 2022-02-08 13:22:41,117 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:22:41,122 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:22:41,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:22:42,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:22:42,756 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:22:42,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:22:42,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:22:42,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:22:42,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:22:42,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:22:42,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:22:42,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:22:42,980 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:22:42,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:22:43,075 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['12550 memcached\n'], []) 2022-02-08 13:22:43,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 13:22:43,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:22:43,134 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['12550 memcached\n'], []) 2022-02-08 13:22:43,137 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:22:43,142 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:22:43,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:22:44,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:22:45,137 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:22:45,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:22:45,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:22:45,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:22:45,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:22:45,234 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:22:45,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:22:45,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:22:45,334 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:22:45,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:22:45,404 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['6472 memcached\n'], []) 2022-02-08 13:22:45,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 13:22:45,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:22:45,466 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['6472 memcached\n'], []) 2022-02-08 13:22:45,467 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:22:45,476 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:22:46,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:22:47,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:22:47,453 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:22:47,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:22:47,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:22:47,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:22:47,515 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:22:47,538 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:22:47,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:22:47,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:22:47,611 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:22:47,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:22:47,668 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['41620 memcached\n'], []) 2022-02-08 13:22:47,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 13:22:47,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:22:47,721 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['41620 memcached\n'], []) 2022-02-08 13:22:47,723 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:22:47,729 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:22:48,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:22:49,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:22:49,457 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:22:49,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:22:49,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:22:49,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:22:49,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:22:49,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:22:49,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:22:49,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:22:49,753 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:22:49,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:22:49,799 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['105025 memcached\n'], []) 2022-02-08 13:22:49,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 13:22:49,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:22:49,852 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['105025 memcached\n'], []) 2022-02-08 13:22:49,852 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:22:49,855 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 5 queries submitted, 0 failed, 17020 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:22:50,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:22:50,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:22:51,092 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:22:51,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:22:51,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:22:51,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:22:51,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:22:51,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:22:51,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:22:51,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:22:51,292 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:22:51,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:22:51,354 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['120256 memcached\n'], []) 2022-02-08 13:22:51,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 13:22:51,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:22:51,421 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['120256 memcached\n'], []) 2022-02-08 13:22:51,423 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:22:51,430 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:22:51,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:22:52,010 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:22:52,088 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:22:52,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:22:52,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:22:52,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:22:52,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:22:52,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:22:52,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:22:52,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:22:52,227 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:22:52,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:22:52,348 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['57553 memcached\n'], []) 2022-02-08 13:22:52,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 13:22:52,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:22:52,418 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['57553 memcached\n'], []) 2022-02-08 13:22:52,421 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:22:52,427 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:22:52,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:22:52,757 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:22:52,861 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:22:52,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:22:52,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:22:52,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:22:52,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:22:52,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:22:52,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:22:53,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:22:53,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:22:53,049 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 13:22:53,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:22:53,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:23:00,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:23:01,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:23:02,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:23:02,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:23:04,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:23:04,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:23:04,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:23:04,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:23:04,868 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 13:23:04,869 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:23:04,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:23:04,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:23:04,987 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 13:23:04,992 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:23:05,013 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:23:05,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:23:05,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:23:05,427 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:23:05,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:23:05,448 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:23:05,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:23:05,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:23:05,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:23:05,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:23:05,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:23:05,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:23:05,571 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 13:23:05,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:23:05,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:23:13,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:23:14,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:23:14,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:23:15,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:23:16,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:23:16,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:23:16,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:23:16,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:23:16,936 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 13:23:16,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:23:16,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:23:17,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:23:17,096 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 13:23:17,098 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:23:17,107 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 13:23:17,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:23:17,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 13:23:17,523 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:23:17,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 13:23:17,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 13:23:17,566 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 13:23:17,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 13:23:17,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 13:23:17,627 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 13:23:17,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 13:23:17,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:23:17,684 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 13:23:17,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:23:17,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:23:17,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:23:17,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:23:17,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:23:17,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:23:17,874 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 13:23:17,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:23:17,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:23:17,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:23:17,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:17,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:23:17,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:18,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:18,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:18,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:18,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:18,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:18,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:18,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:18,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:18,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:23:18,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:18,293 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:18,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:18,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:18,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:18,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:18,392 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:18,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:18,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:18,459 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:23:18,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:18,499 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:18,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:18,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:18,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:18,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:18,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:18,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:18,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:18,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:23:18,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:18,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:18,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:18,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:18,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:18,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:18,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:18,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:18,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:18,890 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:23:18,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:18,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:18,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:18,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:19,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:19,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:19,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:19,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:19,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:19,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:23:19,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:19,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:19,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:19,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:19,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:19,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:19,270 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:19,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:19,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:19,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:23:19,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:19,377 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:19,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:19,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:19,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:19,459 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:19,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:19,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:19,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:19,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:23:19,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:19,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:19,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:19,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:19,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:19,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:19,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:19,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:19,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:19,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:23:19,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:19,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:19,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:19,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:19,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:19,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:19,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:19,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:19,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:19,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:23:20,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:20,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:20,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:20,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:20,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:20,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:20,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:20,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:20,414 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 13:23:20,426 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:23:20,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:23:20,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:23:21,206 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:23:21,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:23:21,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:23:21,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:23:21,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:23:21,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:23:21,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:23:21,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:23:21,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:23:21,658 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 13:23:21,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:23:21,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:23:28,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:23:29,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:23:30,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:23:30,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:23:30,799 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 13:23:30,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:23:30,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:23:30,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:23:30,947 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 13:23:30,953 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:23:30,964 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 13:23:31,084 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:23:31,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 13:23:31,391 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:23:31,392 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 13:23:31,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 13:23:31,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 13:23:31,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 13:23:31,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 13:23:31,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 13:23:31,520 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 13:23:31,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:23:31,559 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 13:23:31,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:23:31,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:23:31,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:23:31,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:23:31,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:23:31,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:23:31,734 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 13:23:31,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:23:31,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:23:31,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:23:31,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:31,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:23:31,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:31,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:31,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:31,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:31,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:31,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:32,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:32,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:23:32,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:32,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:23:32,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:32,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:32,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:32,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:32,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:32,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:32,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:32,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:23:32,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:32,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:23:32,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:32,280 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:32,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:32,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:32,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:32,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:32,377 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:32,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:23:32,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:32,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:23:32,466 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:32,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:32,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:32,528 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:32,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:32,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:32,584 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:32,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:23:32,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:32,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:23:32,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:32,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:32,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:32,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:32,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:32,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:32,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:32,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:23:32,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:32,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:23:32,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:32,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:32,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:32,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:32,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:32,999 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:33,020 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:33,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:23:33,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:33,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:23:33,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:33,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:33,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:33,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:33,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:33,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:33,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:33,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:23:33,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:33,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:23:33,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:33,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:33,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:33,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:33,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:33,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:33,466 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:33,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:23:33,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:33,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:23:33,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:33,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:33,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:33,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:33,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:33,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:33,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:33,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:23:33,709 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:33,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:23:33,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:33,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:33,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:33,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:33,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:33,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:33,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:33,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:23:33,900 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 13:23:33,908 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:23:34,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:23:34,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:23:34,634 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:23:34,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:23:34,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:23:34,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:23:34,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:23:34,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:23:34,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:23:34,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:23:34,918 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:23:34,953 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 13:23:35,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:23:35,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:23:41,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:23:42,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:23:43,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:23:43,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:23:45,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:23:45,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:23:46,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:23:46,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:23:46,285 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 13:23:46,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:23:46,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:23:46,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:23:46,391 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 13:23:46,394 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:23:46,401 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:23:46,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:23:46,918 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:23:47,055 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:23:47,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:23:47,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:23:47,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:23:47,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:23:47,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:23:47,275 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:23:47,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:23:47,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:23:47,375 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 13:23:47,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:23:47,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 17040 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:23:55,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:23:56,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:23:57,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:23:57,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:23:59,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:23:59,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:23:59,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:23:59,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:23:59,841 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 13:23:59,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:23:59,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:23:59,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:23:59,993 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 13:23:59,994 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:24:00,002 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 13:24:00,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:24:00,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 13:24:00,490 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:24:00,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 13:24:00,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 13:24:00,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 13:24:00,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 13:24:00,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 13:24:00,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 13:24:00,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 13:24:00,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:24:00,658 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 13:24:00,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:24:00,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:24:00,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:24:00,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:24:00,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:24:00,841 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:24:00,865 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 13:24:00,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:24:00,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:24:00,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:24:00,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:00,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:24:01,002 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:01,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:01,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:01,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:01,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:01,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:01,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:01,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:01,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:01,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:24:01,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:01,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:01,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:01,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:01,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:01,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:01,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:01,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:01,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:01,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:24:01,545 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:01,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:01,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:01,624 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:01,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:01,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:01,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:01,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:01,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:01,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:24:01,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:01,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:01,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:01,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:01,900 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:01,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:01,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:01,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:02,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:02,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:24:02,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:02,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:02,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:02,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:02,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:02,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:02,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:02,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:02,239 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:02,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:24:02,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:02,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:02,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:02,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:02,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:02,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:02,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:02,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:02,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:02,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:24:02,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:02,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:02,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:02,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:02,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:02,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:02,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:02,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:02,709 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:02,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:24:02,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:02,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:02,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:02,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:02,841 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:02,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:02,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:02,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:02,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:02,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:24:02,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:02,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:03,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:03,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:03,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:03,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:03,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:03,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:03,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:03,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:24:03,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:03,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:03,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:03,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:03,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:03,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:03,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:03,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:03,401 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 13:24:03,407 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 13:24:03,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:24:03,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 13:24:03,815 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:24:03,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 13:24:03,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 13:24:03,849 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 13:24:03,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 13:24:03,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 13:24:03,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 13:24:03,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 13:24:03,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:24:03,963 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 13:24:04,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:24:04,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:24:04,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:24:04,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:24:04,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:24:04,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:24:04,151 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 13:24:04,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:24:04,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:24:04,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:24:04,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:04,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:24:04,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:04,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:04,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:04,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:04,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:04,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:04,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:04,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:24:04,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:04,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:24:04,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:04,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:04,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:04,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:04,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:04,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:04,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:04,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:24:04,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:04,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:24:04,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:04,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:04,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:04,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:04,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:04,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:04,831 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:04,854 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:24:04,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:04,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:24:04,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:04,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:04,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:04,987 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:05,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:05,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:05,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:05,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:24:05,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:05,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:24:05,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:05,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:05,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:05,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:05,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:05,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:05,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:05,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:24:05,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:05,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:24:05,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:05,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:05,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:05,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:05,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:05,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:05,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:05,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:24:05,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:05,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:24:05,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:05,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:05,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:05,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:05,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:05,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:05,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:05,709 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:24:05,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:05,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:24:05,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:05,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:05,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:05,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:05,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:05,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:05,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:05,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:24:05,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:05,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:24:05,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:05,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:06,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:06,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:06,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:06,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:06,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:06,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:24:06,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:06,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:24:06,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:06,204 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:06,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:06,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:06,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:06,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:06,303 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:06,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:24:06,341 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 13:24:06,345 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:24:07,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:24:07,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:24:07,997 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:24:08,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:24:08,017 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:24:08,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:24:08,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:24:08,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:24:08,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:24:08,155 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:24:08,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:24:08,200 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 13:24:08,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:24:08,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:24:16,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:24:17,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:24:17,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:24:18,489 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:24:19,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:24:20,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:24:20,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:24:20,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:24:20,540 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 13:24:20,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:24:20,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:24:20,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:24:20,698 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 13:24:20,700 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:24:20,706 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:24:21,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:24:21,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:24:22,168 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:24:22,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:24:22,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:24:22,270 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:24:22,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:24:22,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:24:22,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:24:22,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:24:22,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:24:22,486 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 13:24:22,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:24:22,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:24:29,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:24:30,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:24:31,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:24:31,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:24:31,891 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 13:24:31,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:24:31,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:24:31,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:24:31,999 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 13:24:32,000 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:24:32,003 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:24:32,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:24:33,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:24:33,190 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:24:33,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:24:33,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:24:33,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:24:33,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:24:33,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:24:33,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:24:33,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:24:33,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:24:33,348 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 13:24:33,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:24:33,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:24:39,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:24:40,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:24:41,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:24:41,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:24:41,688 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 13:24:41,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:24:41,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:24:41,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:24:41,779 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 13:24:41,780 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:24:41,785 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:24:42,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:24:43,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:24:43,631 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:24:43,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:24:43,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:24:43,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:24:43,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:24:43,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:24:43,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:24:44,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:24:44,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:24:44,092 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 13:24:44,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:24:44,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 17063 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:24:53,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:24:54,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:24:55,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:24:55,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:24:58,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:24:59,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:24:59,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:24:59,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:24:59,845 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 13:24:59,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:24:59,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:24:59,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:24:59,979 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 13:24:59,980 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:24:59,989 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 13:25:00,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:25:00,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 13:25:00,427 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:25:00,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 13:25:00,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 13:25:00,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 13:25:00,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 13:25:00,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 13:25:00,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 13:25:00,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 13:25:00,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:25:00,582 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 13:25:00,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:25:00,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:25:00,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:25:00,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:25:00,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:25:00,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:25:00,770 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 13:25:00,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:25:00,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:25:00,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:25:00,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:25:00,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:25:00,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:25:00,915 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:25:00,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:25:00,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:25:01,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:25:01,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:25:01,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:25:01,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:25:01,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:25:01,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:25:01,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:25:01,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:25:01,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:25:01,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:25:01,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:25:01,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:25:01,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:25:01,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:25:01,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:25:01,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:25:01,377 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:25:01,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:25:01,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:25:01,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:25:01,464 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:25:01,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:25:01,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:25:01,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:25:01,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:25:01,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:25:01,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:25:01,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:25:01,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:25:01,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:25:01,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:25:01,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:25:01,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:25:01,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:25:01,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:25:01,793 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:25:01,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:25:01,841 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:25:01,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:25:01,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:25:01,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:25:01,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:25:01,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:25:01,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:25:02,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:25:02,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:25:02,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:25:02,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:25:02,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:25:02,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:25:02,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:25:02,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:25:02,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:25:02,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:25:02,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:25:02,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:25:02,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:25:02,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:25:02,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:25:02,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:25:02,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:25:02,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:25:02,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:25:02,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:25:02,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:25:02,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:25:02,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:25:02,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:25:02,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:25:02,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:25:02,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:25:02,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:25:02,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:25:02,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:25:02,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:25:02,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:25:02,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:25:02,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:25:02,709 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:25:02,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:25:02,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:25:02,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:25:02,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:25:02,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:25:02,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:25:02,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:25:02,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:25:02,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:25:02,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:25:02,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:25:02,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:25:03,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:25:03,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:25:03,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:25:03,069 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 13:25:03,078 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:25:03,118 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:25:03,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:25:04,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:25:04,315 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:25:04,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:25:04,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:25:04,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:25:04,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:25:04,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:25:04,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:25:04,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:25:04,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:25:05,058 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:25:05,061 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.98300004005. 2022-02-08 13:25:05,062 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:25:05,072 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:25:05,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:25:06,492 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:25:06,664 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:25:06,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:25:06,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:25:06,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:25:06,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:25:06,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:25:06,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:25:06,836 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:25:06,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:25:07,881 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:25:07,884 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.8220000267. 2022-02-08 13:25:07,884 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:25:07,891 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:25:09,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:25:09,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:25:09,792 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:25:09,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:25:09,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:25:09,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:25:09,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:25:09,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:25:09,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:25:09,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:25:10,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:25:10,625 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:25:10,628 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.74300003052. 2022-02-08 13:25:10,628 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:25:10,634 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:25:11,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:25:12,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:25:12,289 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:25:12,293 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:25:12,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:25:12,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:25:12,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:25:12,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:25:12,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:25:12,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:25:12,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:25:13,384 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:25:13,388 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.75900006294. 2022-02-08 13:25:13,388 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:25:13,398 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:25:14,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:25:15,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:25:15,357 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:25:15,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:25:15,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:25:15,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:25:15,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:25:15,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:25:15,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:25:15,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:25:15,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:25:16,167 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:25:16,168 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.77900004387. 2022-02-08 13:25:16,170 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:25:16,174 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:25:16,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:25:17,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:25:17,267 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:25:17,270 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:25:17,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:25:17,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:25:17,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:25:17,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:25:17,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:25:17,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:25:17,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:25:18,072 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:25:18,073 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.90499997139. 2022-02-08 13:25:18,075 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:25:18,079 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:25:18,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:25:19,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:25:19,792 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:25:19,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:25:19,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:25:19,841 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:25:19,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:25:19,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:25:19,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:25:19,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:25:19,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:25:20,720 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:25:20,723 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.64900016785. 2022-02-08 13:25:20,724 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:25:20,733 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:25:21,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:25:21,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:25:21,884 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:25:21,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:25:21,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:25:21,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:25:21,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:25:22,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:25:22,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:25:22,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:25:22,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:25:23,232 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:25:23,233 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.50900006294. 2022-02-08 13:25:23,236 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:25:23,243 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:25:24,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:25:25,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:25:25,203 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:25:25,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:25:25,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:25:25,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:25:25,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:25:25,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:25:25,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:25:25,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:25:25,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:25:27,246 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:25:27,250 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.0150001049. ############################################################ # # RAM FootPrint: 4994.8203125 # ############################################################ 2022-02-08 13:25:27,348 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 13:25:27,351 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 97 seconds. Reason: Iteration:16 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 17086 passed, 0 rejected, 0 cancelled, 0 timeout Loader_GleamBookUsers0__default_VolumeCollection9_10_1644308330.38 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection4_-4_1644308344.55 5 queries submitted, 0 failed, 17110 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:27:04,368 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 13:27:04,369 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 13:27:09,378 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:27:10,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:27:10,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:27:11,154 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:27:11,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:27:11,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:27:11,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:27:11,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:27:11,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:27:11,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:27:11,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:27:11,423 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:27:11,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:27:11,519 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['79090 memcached\n'], []) 2022-02-08 13:27:11,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 13:27:11,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:27:11,641 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['79090 memcached\n'], []) 2022-02-08 13:27:11,642 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:27:11,654 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:27:12,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:27:13,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:27:13,173 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:27:13,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:27:13,209 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:27:13,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:27:13,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:27:13,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:27:13,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:27:13,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:27:13,428 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:27:13,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:27:13,604 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['5150 memcached\n'], []) 2022-02-08 13:27:13,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 13:27:13,640 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:27:13,673 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['5150 memcached\n'], []) 2022-02-08 13:27:13,676 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:27:13,688 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:27:14,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:27:15,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:27:16,046 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:27:16,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:27:16,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:27:16,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:27:16,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:27:16,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:27:16,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:27:16,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:27:16,269 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:27:16,270 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:27:16,342 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['120220 memcached\n'], []) 2022-02-08 13:27:16,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 13:27:16,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:27:16,411 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['120220 memcached\n'], []) 2022-02-08 13:27:16,411 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:27:16,418 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:27:17,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:27:17,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:27:18,101 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:27:18,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:27:18,124 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:27:18,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:27:18,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:27:18,214 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:27:18,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:27:18,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:27:18,289 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:27:18,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:27:18,358 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['13384 memcached\n'], []) 2022-02-08 13:27:18,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 13:27:18,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:27:18,431 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['13384 memcached\n'], []) 2022-02-08 13:27:18,433 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:27:18,451 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:27:19,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:27:20,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:27:20,398 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:27:20,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:27:20,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:27:20,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:27:20,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:27:20,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:27:20,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:27:20,548 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:27:20,572 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:27:20,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:27:20,648 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['7258 memcached\n'], []) 2022-02-08 13:27:20,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 13:27:20,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:27:20,713 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['7258 memcached\n'], []) 2022-02-08 13:27:20,714 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:27:20,719 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:27:21,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:27:22,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:27:22,898 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:27:22,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:27:22,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:27:22,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:27:22,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:27:22,994 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:27:23,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:27:23,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:27:23,075 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:27:23,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:27:23,141 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['42408 memcached\n'], []) 2022-02-08 13:27:23,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 13:27:23,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:27:23,203 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['42408 memcached\n'], []) 2022-02-08 13:27:23,203 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:27:23,211 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:27:24,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:27:24,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:27:25,066 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:27:25,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:27:25,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:27:25,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:27:25,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:27:25,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:27:25,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:27:25,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:27:25,259 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:27:25,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:27:25,319 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['105820 memcached\n'], []) 2022-02-08 13:27:25,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 13:27:25,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:27:25,382 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['105820 memcached\n'], []) 2022-02-08 13:27:25,384 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:27:25,391 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:27:26,204 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:27:26,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:27:26,875 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:27:26,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:27:26,900 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:27:26,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:27:26,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:27:26,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:27:26,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:27:27,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:27:27,045 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:27:27,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:27:27,102 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['121014 memcached\n'], []) 2022-02-08 13:27:27,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 13:27:27,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:27:27,171 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['121014 memcached\n'], []) 2022-02-08 13:27:27,174 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:27:27,180 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:27:27,793 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:27:28,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:27:28,433 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:27:28,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:27:28,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:27:28,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:27:28,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:27:28,520 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:27:28,545 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:27:28,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:27:28,596 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:27:28,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:27:28,663 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['58377 memcached\n'], []) 2022-02-08 13:27:28,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 13:27:28,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:27:28,721 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['58377 memcached\n'], []) 2022-02-08 13:27:28,723 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:27:28,733 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:27:28,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:27:29,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:27:29,157 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:27:29,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:27:29,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:27:29,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:27:29,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:27:29,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:27:29,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:27:29,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:27:29,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:27:29,323 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 13:27:29,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:27:29,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:27:36,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:27:36,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:27:37,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:27:38,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:27:39,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:27:40,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:27:40,308 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:27:40,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:27:40,473 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 13:27:40,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:27:40,499 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:27:40,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:27:40,792 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 13:27:40,792 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:27:40,796 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:27:40,897 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:27:41,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:27:41,204 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:27:41,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:27:41,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:27:41,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:27:41,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:27:41,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:27:41,303 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:27:41,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:27:41,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:27:41,361 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 13:27:41,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:27:41,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:27:48,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:27:49,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:27:50,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 17136 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:27:50,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:27:52,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:27:52,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:27:52,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:27:52,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:27:52,553 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 13:27:52,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:27:52,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:27:52,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:27:52,670 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 13:27:52,671 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:27:52,676 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 13:27:52,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:27:53,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 13:27:53,098 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:27:53,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 13:27:53,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 13:27:53,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 13:27:53,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 13:27:53,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 13:27:53,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 13:27:53,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 13:27:53,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:27:53,250 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 13:27:53,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:27:53,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:27:53,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:27:53,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:27:53,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:27:53,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:27:53,447 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 13:27:53,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:27:53,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:27:53,494 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:27:53,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:27:53,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:27:53,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:27:53,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:27:53,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:27:53,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:27:53,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:27:53,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:27:53,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:27:53,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:27:53,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:27:53,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:27:53,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:27:53,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:27:53,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:27:53,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:27:53,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:27:53,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:27:53,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:27:53,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:27:54,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:27:54,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:27:54,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:27:54,084 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:27:54,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:27:54,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:27:54,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:27:54,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:27:54,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:27:54,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:27:54,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:27:54,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:27:54,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:27:54,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:27:54,308 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:27:54,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:27:54,349 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:27:54,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:27:54,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:27:54,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:27:54,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:27:54,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:27:54,466 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:27:54,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:27:54,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:27:54,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:27:54,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:27:54,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:27:54,584 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:27:54,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:27:54,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:27:54,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:27:54,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:27:54,693 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:27:54,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:27:54,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:27:54,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:27:54,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:27:54,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:27:54,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:27:54,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:27:54,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:27:54,897 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:27:54,918 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:27:54,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:27:54,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:27:54,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:27:55,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:27:55,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:27:55,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:27:55,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:27:55,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:27:55,124 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:27:55,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:27:55,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:27:55,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:27:55,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:27:55,239 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:27:55,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:27:55,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:27:55,303 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:27:55,323 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:27:55,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:27:55,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:27:55,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:27:55,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:27:55,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:27:55,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:27:55,466 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:27:55,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:27:55,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:27:55,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:27:55,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:27:55,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:27:55,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:27:55,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:27:55,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:27:55,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:27:55,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:27:55,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:27:55,711 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 13:27:55,717 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:27:55,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:27:56,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:27:56,924 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:27:56,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:27:56,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:27:56,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:27:57,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:27:57,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:27:57,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:27:57,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:27:57,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:27:57,960 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 13:27:58,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:27:58,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:28:04,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:28:05,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:28:06,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:28:07,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:28:07,355 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 13:28:07,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:28:07,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:28:07,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:28:07,665 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 13:28:07,668 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:28:07,677 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 13:28:07,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:28:08,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 13:28:08,109 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:28:08,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 13:28:08,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 13:28:08,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 13:28:08,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 13:28:08,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 13:28:08,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 13:28:08,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 13:28:08,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:28:08,266 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 13:28:08,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:28:08,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:28:08,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:28:08,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:28:08,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:28:08,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:28:08,453 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 13:28:08,456 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:28:08,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:28:08,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:28:08,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:08,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:28:08,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:08,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:08,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:08,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:08,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:08,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:08,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:08,793 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:08,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:08,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:28:08,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:08,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:08,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:08,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:08,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:08,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:08,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:09,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:09,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:09,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:28:09,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:09,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:09,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:09,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:09,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:09,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:09,209 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:09,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:09,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:09,270 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:28:09,293 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:09,316 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:09,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:09,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:09,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:09,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:09,418 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:09,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:09,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:09,477 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:28:09,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:09,520 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:09,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:09,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:09,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:09,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:09,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:09,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:09,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:09,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:28:09,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:09,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:09,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:09,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:09,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:09,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:09,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:09,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:09,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:09,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:28:09,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:09,915 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:09,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:09,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:09,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:10,002 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:10,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:10,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:10,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:10,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:28:10,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:10,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:10,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:10,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:10,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:10,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:10,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:10,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:10,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:10,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:28:10,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:10,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:10,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:10,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:10,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:10,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:10,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:10,499 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:10,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:10,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:28:10,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:10,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:10,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:10,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:10,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:10,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:10,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:10,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:10,740 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 13:28:10,743 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:28:10,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:28:11,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:28:11,233 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:28:11,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:28:11,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:28:11,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:28:11,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:28:11,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:28:11,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:28:11,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:28:11,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:28:11,388 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 13:28:11,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:28:11,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:28:17,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:28:18,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:28:19,239 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:28:19,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:28:21,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:28:22,186 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:28:22,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:28:22,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:28:22,529 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 13:28:22,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:28:22,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:28:22,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:28:22,640 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 13:28:22,641 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:28:22,651 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:28:22,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:28:23,102 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:28:23,365 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:28:23,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:28:23,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:28:23,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:28:23,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:28:23,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:28:23,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:28:23,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:28:23,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:28:23,749 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 13:28:23,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:28:23,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:28:31,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:28:33,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:28:33,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:28:34,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:28:36,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:28:36,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:28:36,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:28:36,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:28:36,967 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 13:28:36,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:28:37,020 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:28:37,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:28:37,161 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 13:28:37,163 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:28:37,170 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 13:28:37,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:28:37,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 13:28:37,644 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:28:37,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 13:28:37,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 13:28:37,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 13:28:37,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 13:28:37,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 13:28:37,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 13:28:37,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 13:28:37,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:28:37,821 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 13:28:37,890 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:28:37,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:28:37,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:28:37,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:28:37,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:28:37,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:28:38,017 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 13:28:38,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:28:38,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:28:38,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:28:38,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:38,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:28:38,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:38,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:38,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:38,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:38,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:38,341 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:38,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:38,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:38,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:38,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:28:38,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:38,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:38,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:38,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:38,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:38,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:38,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:38,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:38,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:38,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:28:38,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:38,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:38,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:38,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:38,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:38,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:38,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:38,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:38,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:38,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:28:38,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:39,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:39,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:39,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:39,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:39,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:39,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:39,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:39,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:39,219 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:28:39,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:39,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:39,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:39,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:39,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:39,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:39,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:39,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:39,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:39,464 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:28:39,489 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:39,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:39,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:39,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:39,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:39,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:39,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:39,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:39,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:39,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:28:39,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:39,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:39,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:39,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:39,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:39,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:39,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:39,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:39,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:40,002 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:28:40,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:40,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:40,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:40,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:40,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:40,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:40,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:40,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:40,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:40,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:28:40,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:40,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:40,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:40,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:40,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:40,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:40,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:40,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:40,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:40,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:28:40,499 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:40,522 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:40,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:40,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:40,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:40,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:40,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:40,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:40,677 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 13:28:40,686 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 13:28:40,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:28:41,010 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 13:28:41,096 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:28:41,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 13:28:41,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 13:28:41,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 13:28:41,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 13:28:41,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 13:28:41,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 13:28:41,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 13:28:41,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:28:41,250 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 13:28:41,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:28:41,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:28:41,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:28:41,423 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:28:41,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:28:41,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:28:41,473 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 13:28:41,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:28:41,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:28:41,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:28:41,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:41,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:28:41,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:41,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:41,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:41,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:41,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:41,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:41,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:41,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:28:41,831 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:41,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:28:41,869 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:41,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:41,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:41,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:41,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:41,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:41,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:42,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:28:42,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:42,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:28:42,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:42,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:42,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:42,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:42,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:42,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:42,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:42,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:28:42,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:42,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:28:42,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:42,316 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:42,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:42,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:42,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:42,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:42,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:42,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:28:42,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:42,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:28:42,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:42,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:42,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:42,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:42,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:42,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:42,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:42,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:28:42,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:42,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:28:42,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:42,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:42,788 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:42,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:42,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:42,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:42,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:42,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:28:42,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:42,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:28:42,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:42,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:43,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:43,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:43,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:43,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:43,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:43,122 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:28:43,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:43,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:28:43,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:43,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:43,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:43,255 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:43,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:43,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:43,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:43,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:28:43,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:43,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:28:43,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:43,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:43,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:43,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:43,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:43,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:43,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:43,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:28:43,579 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:43,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:28:43,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:43,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:43,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:43,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:43,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:43,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:43,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:43,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:28:43,796 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 13:28:43,802 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:28:44,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:28:45,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:28:45,256 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:28:45,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:28:45,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:28:45,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:28:45,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:28:45,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:28:45,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:28:45,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:28:45,520 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:28:45,545 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 13:28:45,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:28:45,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 17161 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:28:53,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:28:59,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:28:59,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:29:00,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:29:02,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:29:02,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:29:02,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:29:02,900 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:29:02,940 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 13:29:02,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:29:02,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:29:03,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:29:03,065 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 13:29:03,069 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:29:03,088 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:29:03,538 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:29:03,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:29:04,154 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:29:04,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:29:04,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:29:04,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:29:04,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:29:04,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:29:04,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:29:04,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:29:04,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:29:04,457 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 13:29:04,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:29:04,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:29:12,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:29:21,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:29:21,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:29:22,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:29:22,339 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 13:29:22,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:29:22,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:29:22,431 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:29:22,460 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 13:29:22,463 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:29:22,474 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:29:23,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:29:23,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:29:23,845 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:29:23,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:29:23,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:29:23,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:29:23,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:29:23,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:29:24,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:29:24,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:29:24,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:29:24,203 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 13:29:24,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:29:24,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:29:30,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:29:31,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:29:32,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:29:32,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:29:32,638 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 13:29:32,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:29:32,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:29:32,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:29:32,739 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 13:29:32,740 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:29:32,749 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:29:33,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:29:34,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:29:34,509 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:29:34,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:29:34,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:29:34,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:29:34,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:29:34,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:29:34,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:29:34,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:29:34,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:29:34,782 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 13:29:34,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:29:34,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:29:42,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:29:43,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:29:44,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:29:45,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:29:47,915 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:29:48,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:29:48,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:29:48,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:29:48,835 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 13:29:48,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:29:48,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:29:48,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:29:48,967 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 13:29:48,969 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:29:48,976 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 13:29:49,084 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:29:49,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 13:29:49,414 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:29:49,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 13:29:49,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 13:29:49,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 13:29:49,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 13:29:49,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 13:29:49,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 13:29:49,528 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 13:29:49,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:29:49,569 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 13:29:49,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:29:49,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:29:49,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:29:49,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:29:49,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:29:49,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:29:49,742 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 13:29:49,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:29:49,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:29:49,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:29:49,808 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:29:49,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:29:49,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:29:49,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:29:49,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:29:49,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:29:49,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:29:50,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:29:50,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:29:50,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:29:50,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:29:50,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:29:50,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:29:50,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 5 queries submitted, 0 failed, 17183 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:29:50,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:29:50,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:29:50,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:29:50,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:29:50,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:29:50,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:29:50,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:29:50,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:29:50,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:29:50,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:29:50,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:29:50,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:29:50,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:29:50,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:29:50,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:29:50,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:29:50,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:29:50,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:29:50,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:29:50,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:29:50,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:29:50,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:29:50,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:29:50,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:29:50,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:29:50,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:29:50,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:29:50,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:29:50,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:29:50,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:29:50,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:29:50,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:29:50,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:29:50,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:29:50,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:29:50,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:29:50,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:29:50,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:29:50,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:29:50,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:29:51,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:29:51,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:29:51,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:29:51,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:29:51,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:29:51,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:29:51,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:29:51,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:29:51,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:29:51,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:29:51,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:29:51,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:29:51,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:29:51,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:29:51,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:29:51,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:29:51,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:29:51,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:29:51,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:29:51,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:29:51,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:29:51,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:29:51,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:29:51,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:29:51,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:29:51,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:29:51,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:29:51,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:29:51,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:29:51,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:29:51,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:29:51,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:29:51,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:29:51,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:29:51,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:29:51,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:29:51,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:29:51,808 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:29:51,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:29:51,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:29:51,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:29:51,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:29:51,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:29:51,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:29:51,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:29:51,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:29:51,990 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 13:29:51,992 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:29:51,994 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:29:52,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:29:52,849 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:29:53,009 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:29:53,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:29:53,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:29:53,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:29:53,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:29:53,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:29:53,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:29:53,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:29:53,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:29:53,858 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:29:53,861 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.86999988556. 2022-02-08 13:29:53,865 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:29:53,874 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:29:54,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:29:55,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:29:55,717 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:29:55,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:29:55,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:29:55,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:29:55,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:29:55,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:29:55,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:29:55,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:29:55,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:29:56,723 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:29:56,723 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.8599998951. 2022-02-08 13:29:56,723 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:29:56,726 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:29:57,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:29:58,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:29:58,730 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:29:58,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:29:58,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:29:58,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:29:58,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:29:58,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:29:58,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:29:59,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:29:59,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:30:01,674 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:30:01,678 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 4.95200014114. 2022-02-08 13:30:01,680 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:30:01,686 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:30:03,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:30:03,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:30:03,947 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:30:03,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:30:03,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:30:04,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:30:04,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:30:04,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:30:04,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:30:04,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:30:04,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:30:08,542 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:30:08,545 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.86500000954. 2022-02-08 13:30:08,546 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:30:08,549 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:30:09,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:30:10,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:30:10,400 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:30:10,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:30:10,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:30:10,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:30:10,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:30:10,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:30:10,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:30:10,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:30:10,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:30:11,369 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:30:11,371 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.8259999752. 2022-02-08 13:30:11,371 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:30:11,377 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:30:12,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:30:12,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:30:12,619 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:30:12,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:30:12,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:30:12,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:30:12,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:30:12,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:30:12,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:30:12,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:30:12,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:30:15,312 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:30:15,315 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.9430000782. 2022-02-08 13:30:15,315 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:30:15,322 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:30:16,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:30:16,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:30:16,750 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:30:16,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:30:16,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:30:16,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:30:16,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:30:16,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:30:16,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:30:16,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:30:16,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:30:17,456 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:30:17,459 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.14300012589. 2022-02-08 13:30:17,460 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:30:17,466 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:30:18,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:30:18,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:30:18,540 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:30:18,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:30:18,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:30:18,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:30:18,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:30:18,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:30:18,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:30:18,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:30:18,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:30:22,551 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:30:22,552 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 5.09200000763. 2022-02-08 13:30:22,552 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:30:22,553 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:30:23,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:30:23,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:30:23,818 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:30:23,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:30:23,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:30:23,869 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:30:23,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:30:23,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:30:23,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:30:23,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:30:24,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:30:25,144 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:30:25,147 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.59500002861. ############################################################ # # RAM FootPrint: 4996.14453125 # ############################################################ 2022-02-08 13:30:25,209 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 13:30:25,209 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 91 seconds. Reason: Iteration:17 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 17205 passed, 0 rejected, 0 cancelled, 0 timeout Loader_GleamBookUsers0__default_VolumeCollection19_8_1644308332.4 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection3_-5_1644308345.56 Loader_GleamBookUsers0__default_VolumeCollection14_6_1644308334.42 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection2_-6_1644308346.57 Loader_GleamBookUsers0__default_VolumeCollection8_9_1644308331.39 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection10_-7_1644308347.58 Loader_GleamBookUsers0__default_VolumeCollection18_2_1644308338.48 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection1_-8_1644308348.59 5 queries submitted, 0 failed, 17228 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:31:56,246 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 13:31:56,250 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 13:32:01,263 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:32:02,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:32:02,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:32:03,119 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:32:03,122 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:32:03,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:32:03,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:32:03,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:32:03,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:32:03,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:32:03,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:32:03,443 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:32:03,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:32:03,529 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['79847 memcached\n'], []) 2022-02-08 13:32:03,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 13:32:03,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:32:03,608 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['79847 memcached\n'], []) 2022-02-08 13:32:03,611 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:32:03,619 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:32:04,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:32:05,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:32:05,328 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:32:05,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:32:05,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:32:05,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:32:05,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:32:05,456 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:32:05,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:32:05,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:32:05,535 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:32:05,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:32:05,614 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['5981 memcached\n'], []) 2022-02-08 13:32:05,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 13:32:05,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:32:05,691 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['5981 memcached\n'], []) 2022-02-08 13:32:05,691 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:32:05,696 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:32:06,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:32:07,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:32:07,790 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:32:07,793 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:32:07,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:32:07,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:32:07,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:32:07,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:32:07,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:32:08,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:32:08,040 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:32:08,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:32:08,118 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['121156 memcached\n'], []) 2022-02-08 13:32:08,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 13:32:08,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:32:08,191 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['121156 memcached\n'], []) 2022-02-08 13:32:08,194 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:32:08,207 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:32:09,227 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:32:09,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:32:10,033 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:32:10,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:32:10,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:32:10,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:32:10,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:32:10,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:32:10,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:32:10,239 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:32:10,263 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:32:10,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:32:10,397 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['14201 memcached\n'], []) 2022-02-08 13:32:10,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 13:32:10,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:32:10,467 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['14201 memcached\n'], []) 2022-02-08 13:32:10,470 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:32:10,476 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:32:11,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:32:11,987 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:32:12,246 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:32:12,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:32:12,280 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:32:12,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:32:12,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:32:12,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:32:12,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:32:12,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:32:12,464 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:32:12,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:32:12,543 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['8086 memcached\n'], []) 2022-02-08 13:32:12,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 13:32:12,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:32:12,631 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['8086 memcached\n'], []) 2022-02-08 13:32:12,631 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:32:12,634 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:32:13,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:32:14,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:32:14,882 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:32:14,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:32:14,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:32:14,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:32:14,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:32:14,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:32:14,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:32:15,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:32:15,046 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:32:15,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:32:15,108 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['43236 memcached\n'], []) 2022-02-08 13:32:15,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 13:32:15,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:32:15,164 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['43236 memcached\n'], []) 2022-02-08 13:32:15,165 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:32:15,173 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:32:16,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:32:16,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:32:16,869 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:32:16,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:32:16,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:32:16,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:32:16,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:32:16,977 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:32:16,999 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:32:17,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:32:17,065 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:32:17,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:32:17,130 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['106576 memcached\n'], []) 2022-02-08 13:32:17,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 13:32:17,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:32:17,200 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['106576 memcached\n'], []) 2022-02-08 13:32:17,203 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:32:17,213 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:32:18,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:32:18,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:32:18,778 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:32:18,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:32:18,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:32:18,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:32:18,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:32:18,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:32:18,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:32:18,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:32:18,963 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:32:18,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:32:19,052 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['121801 memcached\n'], []) 2022-02-08 13:32:19,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 13:32:19,086 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:32:19,115 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['121801 memcached\n'], []) 2022-02-08 13:32:19,115 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:32:19,119 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:32:19,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:32:20,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:32:20,901 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:32:20,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:32:20,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:32:20,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:32:21,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:32:21,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:32:21,094 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:32:21,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:32:21,140 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:32:21,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:32:21,213 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['59167 memcached\n'], []) 2022-02-08 13:32:21,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 13:32:21,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:32:21,288 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['59167 memcached\n'], []) 2022-02-08 13:32:21,292 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:32:21,305 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:32:21,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:32:21,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:32:21,710 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:32:21,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:32:21,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:32:21,757 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:32:21,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:32:21,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:32:21,808 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:32:21,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:32:21,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:32:21,888 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 13:32:21,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:32:21,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:32:29,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:32:30,183 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:32:30,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:32:31,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:32:33,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:32:33,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:32:33,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:32:33,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:32:33,687 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 13:32:33,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:32:33,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:32:33,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:32:33,798 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 13:32:33,801 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:32:33,811 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:32:33,994 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:32:34,219 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:32:34,312 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:32:34,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:32:34,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:32:34,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:32:34,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:32:34,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:32:34,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:32:34,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:32:34,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:32:34,483 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 13:32:34,548 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:32:34,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:32:41,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:32:42,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:32:43,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:32:43,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:32:45,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:32:45,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:32:45,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:32:45,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:32:45,874 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 13:32:45,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:32:45,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:32:45,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:32:46,003 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 13:32:46,005 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:32:46,012 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 13:32:46,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:32:46,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 13:32:46,430 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:32:46,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 13:32:46,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 13:32:46,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 13:32:46,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 13:32:46,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 13:32:46,528 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 13:32:46,545 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 13:32:46,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:32:46,599 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 13:32:46,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:32:46,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:32:46,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:32:46,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:32:46,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:32:46,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:32:46,846 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 13:32:46,849 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:32:46,869 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:32:46,900 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:32:46,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:32:46,951 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:32:46,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:32:47,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:32:47,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:32:47,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:32:47,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:32:47,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:32:47,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:32:47,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:32:47,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:32:47,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:32:47,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:32:47,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:32:47,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:32:47,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:32:47,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:32:47,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:32:47,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:32:47,448 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:32:47,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:32:47,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:32:47,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:32:47,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:32:47,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:32:47,579 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:32:47,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:32:47,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:32:47,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:32:47,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:32:47,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:32:47,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:32:47,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:32:47,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:32:47,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:32:47,836 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:32:47,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:32:47,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:32:47,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:32:47,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:32:47,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:32:47,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:32:47,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:32:48,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:32:48,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:32:48,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:32:48,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:32:48,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:32:48,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:32:48,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:32:48,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:32:48,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:32:48,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:32:48,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:32:48,303 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:32:48,323 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:32:48,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:32:48,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:32:48,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:32:48,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:32:48,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:32:48,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:32:48,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:32:48,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:32:48,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:32:48,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:32:48,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:32:48,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:32:48,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:32:48,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:32:48,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:32:48,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:32:48,757 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:32:48,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:32:48,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:32:48,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:32:48,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:32:48,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:32:48,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:32:48,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:32:48,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:32:48,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:32:48,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:32:49,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:32:49,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:32:49,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:32:49,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:32:49,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:32:49,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:32:49,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:32:49,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:32:49,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:32:49,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:32:49,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:32:49,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:32:49,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:32:49,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:32:49,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:32:49,456 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:32:49,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:32:49,496 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 13:32:49,505 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:32:49,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:32:49,994 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 5 queries submitted, 0 failed, 17253 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:32:50,194 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:32:50,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:32:50,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:32:50,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:32:50,466 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:32:50,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:32:50,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:32:50,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:32:50,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:32:50,710 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 13:32:50,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:32:50,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:32:57,854 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:32:59,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:32:59,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:33:00,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:33:00,144 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 13:33:00,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:33:00,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:33:00,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:33:00,270 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 13:33:00,273 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:33:00,283 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 13:33:00,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:33:00,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 13:33:00,714 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:33:00,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 13:33:00,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 13:33:00,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 13:33:00,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 13:33:00,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 13:33:00,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 13:33:00,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 13:33:00,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:33:00,871 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 13:33:00,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:33:00,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:33:00,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:33:01,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:33:01,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:33:01,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:33:01,078 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 13:33:01,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:33:01,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:33:01,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:33:01,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:01,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:33:01,227 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:01,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:01,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:01,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:01,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:01,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:01,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:01,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:01,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:01,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:33:01,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:01,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:01,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:01,624 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:01,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:01,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:01,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:01,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:01,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:01,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:33:01,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:01,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:01,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:01,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:01,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:01,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:01,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:01,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:01,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:02,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:33:02,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:02,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:02,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:02,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:02,122 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:02,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:02,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:02,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:02,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:02,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:33:02,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:02,303 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:02,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:02,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:02,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:02,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:02,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:02,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:02,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:02,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:33:02,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:02,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:02,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:02,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:02,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:02,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:02,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:02,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:02,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:02,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:33:02,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:02,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:02,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:02,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:02,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:02,890 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:02,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:02,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:02,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:02,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:33:03,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:03,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:03,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:03,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:03,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:03,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:03,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:03,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:03,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:03,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:33:03,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:03,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:03,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:03,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:03,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:03,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:03,459 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:03,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:03,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:03,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:33:03,579 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:03,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:03,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:03,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:03,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:03,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:03,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:03,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:03,759 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 13:33:03,769 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:33:03,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:33:04,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:33:04,197 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:33:04,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:33:04,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:33:04,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:33:04,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:33:04,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:33:04,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:33:04,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:33:04,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:33:04,348 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 13:33:04,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:33:04,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:33:10,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:33:11,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:33:12,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:33:12,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:33:15,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:33:15,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:33:15,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:33:15,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:33:15,710 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 13:33:15,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:33:15,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:33:15,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:33:15,822 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 13:33:15,823 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:33:15,828 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:33:16,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:33:17,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:33:17,289 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:33:17,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:33:17,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:33:17,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:33:17,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:33:17,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:33:17,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:33:17,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:33:17,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:33:17,957 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 13:33:18,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:33:18,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:33:25,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:33:26,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:33:27,349 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:33:27,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:33:29,849 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:33:30,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:33:30,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:33:30,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:33:30,530 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 13:33:30,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:33:30,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:33:30,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:33:30,825 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 13:33:30,828 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:33:30,838 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 13:33:30,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:33:31,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 13:33:31,301 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:33:31,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 13:33:31,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 13:33:31,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 13:33:31,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 13:33:31,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 13:33:31,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 13:33:31,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 13:33:31,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:33:31,490 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 13:33:31,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:33:31,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:33:31,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:33:31,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:33:31,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:33:31,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:33:31,690 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 13:33:31,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:33:31,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:33:31,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:33:31,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:31,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:33:31,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:31,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:31,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:31,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:31,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:32,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:32,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:32,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:32,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:32,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:33:32,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:32,181 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:32,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:32,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:32,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:32,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:32,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:32,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:32,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:32,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:33:32,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:32,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:32,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:32,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:32,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:32,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:32,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:32,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:32,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:32,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:33:32,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:32,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:32,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:32,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:32,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:32,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:32,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:32,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:32,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:32,897 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:33:32,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:32,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:32,987 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:33,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:33,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:33,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:33,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:33,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:33,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:33,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:33:33,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:33,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:33,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:33,349 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:33,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:33,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:33,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:33,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:33,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:33,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:33:33,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:33,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:33,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:33,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:33,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:33,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:33,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:33,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:33,757 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:33,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:33:33,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:33,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:33,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:33,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:33,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:33,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:33,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:34,002 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:34,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:34,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:33:34,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:34,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:34,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:34,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:34,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:34,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:34,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:34,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:34,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:34,308 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:33:34,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:34,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:34,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:34,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:34,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:34,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:34,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:34,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:34,533 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 13:33:34,551 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 13:33:34,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:33:34,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 13:33:34,970 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:33:34,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 13:33:34,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 13:33:35,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 13:33:35,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 13:33:35,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 13:33:35,084 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 13:33:35,102 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 13:33:35,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:33:35,137 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 13:33:35,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:33:35,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:33:35,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:33:35,288 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:33:35,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:33:35,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:33:35,342 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 13:33:35,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:33:35,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:33:35,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:33:35,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:35,431 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:33:35,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:35,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:35,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:35,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:35,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:35,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:35,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:35,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:33:35,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:35,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:33:35,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:35,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:35,831 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:35,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:35,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:35,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:35,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:35,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:33:35,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:35,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:33:35,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:36,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:36,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:36,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:36,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:36,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:36,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:36,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:33:36,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:36,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:33:36,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:36,255 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:36,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:36,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:36,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:36,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:36,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:36,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:33:36,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:36,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:33:36,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:36,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:36,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:36,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:36,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:36,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:36,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:36,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:33:36,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:36,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:33:36,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:36,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:36,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:36,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:36,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:36,841 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:36,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:36,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:33:36,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:36,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:33:36,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:37,002 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:37,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:37,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:37,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:37,102 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:37,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:37,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:33:37,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:37,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:33:37,239 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:37,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:37,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:37,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:37,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:37,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:37,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:37,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:33:37,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:37,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:33:37,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:37,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:37,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:37,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:37,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:37,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:37,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:37,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:33:37,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:37,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:33:37,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:37,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:37,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:37,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:37,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:37,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:37,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:37,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:33:37,953 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 13:33:37,963 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:33:38,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:33:39,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:33:39,677 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:33:39,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:33:39,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:33:39,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:33:39,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:33:39,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:33:39,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:33:39,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:33:39,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:33:39,966 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 13:33:40,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:33:40,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:33:48,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:33:50,122 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 5 queries submitted, 0 failed, 17278 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:33:50,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:33:51,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:33:53,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:33:53,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:33:53,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:33:53,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:33:53,628 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 13:33:53,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:33:53,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:33:53,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:33:53,756 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 13:33:53,759 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:33:53,763 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:33:53,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:33:54,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:33:54,348 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:33:54,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:33:54,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:33:54,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:33:54,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:33:54,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:33:54,477 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:33:54,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:33:54,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:33:54,585 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 13:33:54,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:33:54,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:34:02,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:34:03,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:34:03,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:34:04,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:34:04,196 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 13:34:04,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:34:04,227 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:34:04,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:34:04,315 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 13:34:04,319 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:34:04,326 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:34:05,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:34:05,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:34:05,724 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:34:05,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:34:05,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:34:05,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:34:05,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:34:05,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:34:05,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:34:05,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:34:05,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:34:05,921 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 13:34:06,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:34:06,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:34:12,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:34:13,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:34:13,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:34:14,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:34:14,447 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 13:34:14,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:34:14,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:34:14,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:34:14,565 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 13:34:14,568 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:34:14,579 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:34:15,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:34:15,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:34:15,963 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:34:15,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:34:15,987 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:34:16,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:34:16,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:34:16,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:34:16,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:34:16,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:34:16,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:34:16,223 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 13:34:16,275 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:34:16,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:34:24,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:34:25,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:34:26,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:34:27,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:34:29,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:34:30,443 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:34:30,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:34:30,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:34:30,993 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 13:34:30,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:34:31,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:34:31,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:34:31,155 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 13:34:31,158 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:34:31,171 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 13:34:31,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:34:31,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 13:34:31,631 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:34:31,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 13:34:31,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 13:34:31,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 13:34:31,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 13:34:31,709 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 13:34:31,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 13:34:31,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 13:34:31,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:34:31,789 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 13:34:31,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:34:31,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:34:31,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:34:31,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:34:31,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:34:31,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:34:31,987 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 13:34:31,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:34:32,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:34:32,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:34:32,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:34:32,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:34:32,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:34:32,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:34:32,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:34:32,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:34:32,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:34:32,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:34:32,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:34:32,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:34:32,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:34:32,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:34:32,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:34:32,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:34:32,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:34:32,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:34:32,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:34:32,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:34:32,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:34:32,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:34:32,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:34:32,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:34:32,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:34:32,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:34:32,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:34:32,734 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:34:32,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:34:32,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:34:32,803 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:34:32,823 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:34:32,854 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:34:32,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:34:32,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:34:32,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:34:32,956 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:34:32,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:34:33,002 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:34:33,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:34:33,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:34:33,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:34:33,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:34:33,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:34:33,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:34:33,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:34:33,191 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:34:33,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:34:33,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:34:33,293 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:34:33,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:34:33,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:34:33,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:34:33,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:34:33,431 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:34:33,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:34:33,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:34:33,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:34:33,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:34:33,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:34:33,584 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:34:33,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:34:33,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:34:33,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:34:33,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:34:33,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:34:33,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:34:33,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:34:33,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:34:33,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:34:33,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:34:33,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:34:33,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:34:33,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:34:33,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:34:33,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:34:33,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:34:33,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:34:33,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:34:34,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:34:34,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:34:34,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:34:34,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:34:34,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:34:34,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:34:34,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:34:34,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:34:34,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:34:34,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:34:34,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:34:34,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:34:34,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:34:34,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:34:34,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:34:34,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:34:34,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:34:34,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:34:34,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:34:34,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:34:34,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:34:34,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:34:34,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:34:34,506 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 13:34:34,509 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:34:34,517 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:34:35,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:34:35,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:34:35,664 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:34:35,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:34:35,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:34:35,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:34:35,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:34:35,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:34:35,772 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:34:35,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:34:35,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:34:36,352 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:34:36,355 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.84599995613. 2022-02-08 13:34:36,355 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:34:36,358 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:34:36,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:34:37,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:34:37,635 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:34:37,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:34:37,693 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:34:37,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:34:37,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:34:37,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:34:37,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:34:37,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:34:37,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:34:38,667 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:34:38,668 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.31399989128. 2022-02-08 13:34:38,670 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:34:38,676 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:34:39,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:34:39,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:34:40,168 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:34:40,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:34:40,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:34:40,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:34:40,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:34:40,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:34:40,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:34:40,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:34:40,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:34:41,012 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:34:41,013 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.34299993515. 2022-02-08 13:34:41,013 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:34:41,016 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:34:41,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:34:42,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:34:42,710 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:34:42,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:34:42,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:34:42,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:34:42,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:34:42,849 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:34:42,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:34:42,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:34:42,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:34:43,657 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:34:43,660 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.64499998093. 2022-02-08 13:34:43,660 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:34:43,667 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:34:44,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:34:44,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:34:45,154 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:34:45,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:34:45,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:34:45,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:34:45,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:34:45,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:34:45,293 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:34:45,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:34:45,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:34:46,150 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:34:46,151 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.49199986458. 2022-02-08 13:34:46,153 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:34:46,161 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:34:47,086 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:34:47,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:34:47,644 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:34:47,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:34:47,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:34:47,709 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:34:47,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:34:47,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:34:47,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:34:47,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:34:47,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:34:48,427 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:34:48,427 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.27499985695. 2022-02-08 13:34:48,427 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:34:48,430 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:34:49,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:34:49,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:34:49,918 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:34:49,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:34:49,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:34:49,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:34:50,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:34:50,028 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:34:50,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:34:50,084 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:34:50,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 5 queries submitted, 0 failed, 17301 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:34:50,664 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:34:50,664 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.23699998856. 2022-02-08 13:34:50,664 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:34:50,670 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:34:51,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:34:51,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:34:52,002 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:34:52,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:34:52,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:34:52,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:34:52,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:34:52,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:34:52,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:34:52,209 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:34:52,239 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:34:52,901 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:34:52,904 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.23900008202. 2022-02-08 13:34:52,904 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:34:52,911 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:34:54,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:34:54,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:34:55,032 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:34:55,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:34:55,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:34:55,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:34:55,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:34:55,255 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:34:55,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:34:55,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:34:55,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:34:56,346 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:34:56,354 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 3.44900012016. ############################################################ # # RAM FootPrint: 4838.9921875 # ############################################################ 2022-02-08 13:34:56,447 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 13:34:56,448 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 61 seconds. Reason: Iteration:18 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 17323 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:35:57,479 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 13:35:57,480 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 13:36:02,490 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:36:02,999 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:36:03,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:36:03,678 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:36:03,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:36:03,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:36:03,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:36:03,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:36:03,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:36:04,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:36:04,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:36:04,119 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:36:04,122 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:36:04,246 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['80663 memcached\n'], []) 2022-02-08 13:36:04,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 13:36:04,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:36:04,355 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['80663 memcached\n'], []) 2022-02-08 13:36:04,355 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:36:04,358 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:36:05,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:36:05,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:36:06,150 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:36:06,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:36:06,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:36:06,209 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:36:06,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:36:06,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:36:06,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:36:06,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:36:06,401 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:36:06,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:36:06,532 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['6810 memcached\n'], []) 2022-02-08 13:36:06,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 13:36:06,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:36:06,611 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['6810 memcached\n'], []) 2022-02-08 13:36:06,611 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:36:06,615 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:36:07,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:36:08,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:36:08,256 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:36:08,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:36:08,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:36:08,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:36:08,374 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:36:08,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:36:08,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:36:08,466 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:36:08,502 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:36:08,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:36:08,579 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['122095 memcached\n'], []) 2022-02-08 13:36:08,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 13:36:08,640 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:36:08,671 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['122095 memcached\n'], []) 2022-02-08 13:36:08,673 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:36:08,684 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:36:09,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:36:10,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:36:10,351 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:36:10,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:36:10,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:36:10,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:36:10,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:36:10,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:36:10,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:36:10,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:36:10,568 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:36:10,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:36:10,641 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['14998 memcached\n'], []) 2022-02-08 13:36:10,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 13:36:10,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:36:10,716 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['14998 memcached\n'], []) 2022-02-08 13:36:10,717 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:36:10,723 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:36:11,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:36:12,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:36:12,538 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:36:12,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:36:12,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:36:12,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:36:12,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:36:12,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:36:12,693 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:36:12,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:36:12,765 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:36:12,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:36:12,828 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['8890 memcached\n'], []) 2022-02-08 13:36:12,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 13:36:12,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:36:12,891 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['8890 memcached\n'], []) 2022-02-08 13:36:12,892 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:36:12,905 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:36:13,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:36:14,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:36:14,805 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:36:14,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:36:14,823 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:36:14,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:36:14,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:36:14,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:36:14,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:36:14,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:36:14,927 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:36:14,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:36:15,003 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['44068 memcached\n'], []) 2022-02-08 13:36:15,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 13:36:15,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:36:15,063 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['44068 memcached\n'], []) 2022-02-08 13:36:15,065 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:36:15,069 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:36:15,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:36:16,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:36:16,780 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:36:16,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:36:16,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:36:16,841 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:36:16,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:36:16,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:36:16,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:36:16,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:36:16,960 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:36:16,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:36:17,015 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['107400 memcached\n'], []) 2022-02-08 13:36:17,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 13:36:17,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:36:17,062 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['107400 memcached\n'], []) 2022-02-08 13:36:17,063 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:36:17,065 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:36:17,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:36:18,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:36:18,299 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:36:18,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:36:18,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:36:18,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:36:18,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:36:18,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:36:18,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:36:18,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:36:18,421 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:36:18,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:36:18,476 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['122614 memcached\n'], []) 2022-02-08 13:36:18,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 13:36:18,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:36:18,530 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['122614 memcached\n'], []) 2022-02-08 13:36:18,532 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:36:18,532 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:36:19,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:36:19,849 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:36:20,066 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:36:20,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:36:20,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:36:20,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:36:20,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:36:20,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:36:20,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:36:20,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:36:20,187 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:36:20,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:36:20,243 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['60016 memcached\n'], []) 2022-02-08 13:36:20,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 13:36:20,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:36:20,295 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['60016 memcached\n'], []) 2022-02-08 13:36:20,296 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:36:20,299 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:36:20,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:36:20,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:36:20,670 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:36:20,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:36:20,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:36:20,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:36:20,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:36:20,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:36:20,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:36:20,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:36:20,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:36:20,796 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 13:36:20,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:36:20,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:36:27,349 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:36:28,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:36:28,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:36:29,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:36:31,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:36:31,448 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:36:31,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:36:31,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:36:31,792 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 13:36:31,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:36:31,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:36:31,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:36:31,890 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 13:36:31,890 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:36:31,892 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:36:31,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:36:32,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:36:32,262 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:36:32,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:36:32,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:36:32,293 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:36:32,308 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:36:32,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:36:32,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:36:32,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:36:32,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:36:32,384 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 13:36:32,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:36:32,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:36:39,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:36:40,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:36:41,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:36:41,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:36:43,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:36:43,392 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:36:43,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:36:43,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:36:43,658 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 13:36:43,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:36:43,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:36:43,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:36:43,776 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 13:36:43,776 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:36:43,779 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 13:36:43,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:36:44,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 13:36:44,167 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:36:44,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 13:36:44,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 13:36:44,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 13:36:44,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 13:36:44,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 13:36:44,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 13:36:44,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 13:36:44,275 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:36:44,290 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 13:36:44,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:36:44,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:36:44,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:36:44,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:36:44,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:36:44,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:36:44,413 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 13:36:44,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:36:44,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:36:44,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:36:44,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:44,487 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:36:44,522 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:44,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:44,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:44,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:44,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:44,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:44,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:44,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:44,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:44,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:36:44,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:44,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:44,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:44,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:44,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:44,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:44,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:44,915 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:44,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:44,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:36:44,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:44,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:45,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:45,017 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:45,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:45,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:45,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:45,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:45,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:45,122 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:36:45,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:45,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:45,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:45,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:45,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:45,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:45,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:45,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:45,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:45,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:36:45,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:45,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:45,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:45,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:45,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:45,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:45,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:45,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:45,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:45,522 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:36:45,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:45,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:45,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:45,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:45,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:45,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:45,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:45,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:45,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:45,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:36:45,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:45,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:45,760 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:45,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:45,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:45,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:45,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:45,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:45,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:45,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:36:45,905 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:45,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:45,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:45,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:45,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:45,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:46,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:46,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:46,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:46,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:36:46,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:46,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:46,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:46,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:46,150 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:46,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:46,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:46,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:46,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:46,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:36:46,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:46,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:46,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:46,295 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:46,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:46,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:46,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:46,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:46,372 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 13:36:46,375 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:36:46,477 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:36:46,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:36:46,779 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:36:46,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:36:46,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:36:46,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:36:46,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:36:46,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:36:46,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:36:46,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:36:46,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:36:46,908 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 13:36:46,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:36:46,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 17347 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:36:53,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:36:54,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:36:55,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:36:55,448 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:36:55,464 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 13:36:55,466 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:36:55,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:36:55,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:36:55,572 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 13:36:55,572 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:36:55,575 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 13:36:55,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:36:55,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 13:36:55,944 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:36:55,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 13:36:55,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 13:36:55,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 13:36:55,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 13:36:56,002 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 13:36:56,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 13:36:56,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 13:36:56,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:36:56,061 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 13:36:56,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:36:56,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:36:56,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:36:56,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:36:56,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:36:56,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:36:56,181 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 13:36:56,181 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:36:56,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:36:56,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:36:56,227 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:56,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:36:56,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:56,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:56,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:56,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:56,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:56,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:56,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:56,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:36:56,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:56,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:36:56,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:56,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:56,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:56,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:56,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:56,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:56,690 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:56,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:36:56,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:56,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:36:56,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:56,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:56,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:56,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:56,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:56,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:56,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:56,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:36:56,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:56,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:36:56,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:56,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:56,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:56,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:57,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:57,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:57,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:57,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:36:57,079 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:57,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:36:57,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:57,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:57,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:57,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:57,183 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:57,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:57,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:57,234 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:36:57,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:57,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:36:57,290 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:57,308 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:57,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:57,346 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:57,364 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:57,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:57,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:57,418 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:36:57,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:57,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:36:57,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:57,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:57,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:57,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:57,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:57,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:57,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:57,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:36:57,617 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:57,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:36:57,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:57,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:57,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:57,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:57,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:57,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:57,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:57,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:36:57,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:57,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:36:57,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:57,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:57,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:57,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:57,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:57,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:57,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:57,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:36:57,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:57,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:36:57,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:58,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:58,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:58,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:58,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:58,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:58,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:58,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:36:58,121 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 13:36:58,124 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:36:58,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:36:58,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:36:58,523 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:36:58,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:36:58,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:36:58,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:36:58,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:36:58,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:36:58,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:36:58,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:36:58,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:36:58,663 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 13:36:58,693 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:36:58,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:37:05,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:37:06,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:37:06,823 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:37:07,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:37:09,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:37:09,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:37:10,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:37:10,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:37:10,180 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 13:37:10,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:37:10,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:37:10,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:37:10,273 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 13:37:10,273 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:37:10,275 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:37:11,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:37:11,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:37:11,878 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:37:11,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:37:11,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:37:11,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:37:11,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:37:11,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:37:12,028 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:37:12,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:37:12,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:37:12,109 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 13:37:12,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:37:12,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:37:20,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:37:21,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:37:21,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:37:22,385 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:37:24,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:37:24,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:37:24,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:37:25,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:37:25,055 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 13:37:25,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:37:25,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:37:25,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:37:25,273 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 13:37:25,273 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:37:25,276 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 13:37:25,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:37:25,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 13:37:25,688 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:37:25,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 13:37:25,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 13:37:25,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 13:37:25,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 13:37:25,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 13:37:25,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 13:37:25,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 13:37:25,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:37:25,825 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 13:37:25,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:37:25,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:37:25,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:37:25,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:37:25,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:37:25,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:37:25,963 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 13:37:25,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:37:25,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:37:25,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:37:26,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:26,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:37:26,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:26,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:26,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:26,181 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:26,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:26,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:26,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:26,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:26,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:26,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:37:26,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:26,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:26,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:26,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:26,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:26,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:26,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:26,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:26,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:26,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:37:26,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:26,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:26,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:26,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:26,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:26,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:26,701 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:26,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:26,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:26,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:37:26,808 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:26,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:26,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:26,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:26,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:26,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:26,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:26,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:26,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:26,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:37:27,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:27,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:27,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:27,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:27,086 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:27,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:27,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:27,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:27,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:27,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:37:27,204 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:27,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:27,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:27,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:27,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:27,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:27,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:27,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:27,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:27,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:37:27,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:27,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:27,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:27,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:27,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:27,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:27,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:27,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:27,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:27,579 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:37:27,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:27,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:27,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:27,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:27,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:27,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:27,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:27,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:27,770 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:27,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:37:27,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:27,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:27,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:27,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:27,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:27,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:27,915 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:27,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:27,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:27,971 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:37:27,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:28,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:28,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:28,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:28,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:28,084 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:28,102 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:28,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:28,138 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 13:37:28,141 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 13:37:28,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:37:28,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 13:37:28,585 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:37:28,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 13:37:28,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 13:37:28,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 13:37:28,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 13:37:28,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 13:37:28,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 13:37:28,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 13:37:28,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:37:28,707 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 13:37:28,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:37:28,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:37:28,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:37:28,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:37:28,793 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:37:28,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:37:28,822 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 13:37:28,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:37:28,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:37:28,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:37:28,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:28,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:37:28,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:28,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:29,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:29,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:29,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:29,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:29,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:29,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:37:29,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:29,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:37:29,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:29,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:29,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:29,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:29,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:29,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:29,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:29,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:37:29,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:29,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:37:29,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:29,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:29,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:29,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:29,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:29,464 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:29,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:29,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:37:29,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:29,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:37:29,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:29,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:29,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:29,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:29,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:29,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:29,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:29,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:37:29,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:29,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:37:29,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:29,742 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:29,760 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:29,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:29,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:29,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:29,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:29,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:37:29,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:29,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:37:29,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:29,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:29,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:29,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:29,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:29,994 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:30,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:30,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:37:30,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:30,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:37:30,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:30,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:30,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:30,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:30,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:30,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:30,209 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:30,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:37:30,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:30,262 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:37:30,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:30,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:30,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:30,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:30,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:30,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:30,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:30,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:37:30,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:30,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:37:30,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:30,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:30,477 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:30,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:30,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:30,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:30,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:30,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:37:30,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:30,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:37:30,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:30,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:30,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:30,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:30,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:30,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:30,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:30,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:37:30,755 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 13:37:30,757 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:37:31,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:37:32,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:37:32,332 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:37:32,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:37:32,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:37:32,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:37:32,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:37:32,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:37:32,461 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:37:32,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:37:32,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:37:32,526 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 13:37:32,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:37:32,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:37:40,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:37:41,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:37:42,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:37:42,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:37:44,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:37:44,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:37:44,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:37:45,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:37:45,107 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 13:37:45,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:37:45,122 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:37:45,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:37:45,243 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 13:37:45,243 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:37:45,246 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:37:45,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:37:45,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:37:45,823 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:37:45,823 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:37:45,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:37:45,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:37:45,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:37:45,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:37:45,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:37:45,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:37:45,987 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:37:46,005 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 13:37:46,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:37:46,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 17370 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:37:53,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:37:54,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:37:55,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:37:55,836 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:37:55,946 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 13:37:55,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:37:55,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:37:56,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:37:56,098 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 13:37:56,098 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:37:56,101 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:37:56,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:37:57,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:37:57,299 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:37:57,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:37:57,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:37:57,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:37:57,349 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:37:57,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:37:57,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:37:57,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:37:57,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:37:57,444 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 13:37:57,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:37:57,503 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:38:04,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:38:05,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:38:05,499 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:38:05,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:38:05,959 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 13:38:05,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:38:05,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:38:06,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:38:06,045 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 13:38:06,046 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:38:06,049 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:38:07,236 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:38:07,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:38:08,040 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:38:08,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:38:08,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:38:08,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:38:08,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:38:08,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:38:08,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:38:08,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:38:08,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:38:08,825 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 13:38:08,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:38:08,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:38:18,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:38:21,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:38:21,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:38:22,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:38:25,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:38:25,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:38:26,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:38:26,308 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:38:26,342 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 13:38:26,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:38:26,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:38:26,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:38:26,446 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 13:38:26,447 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:38:26,448 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 13:38:26,556 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:38:26,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 13:38:26,871 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:38:26,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 13:38:26,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 13:38:26,900 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 13:38:26,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 13:38:26,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 13:38:26,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 13:38:26,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 13:38:26,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:38:26,990 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 13:38:27,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:38:27,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:38:27,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:38:27,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:38:27,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:38:27,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:38:27,154 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 13:38:27,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:38:27,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:38:27,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:38:27,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:38:27,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:38:27,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:38:27,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:38:27,316 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:38:27,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:38:27,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:38:27,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:38:27,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:38:27,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:38:27,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:38:27,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:38:27,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:38:27,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:38:27,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:38:27,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:38:27,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:38:27,627 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:38:27,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:38:27,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:38:27,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:38:27,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:38:27,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:38:27,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:38:27,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:38:27,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:38:27,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:38:27,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:38:27,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:38:27,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:38:27,849 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:38:27,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:38:27,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:38:27,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:38:27,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:38:27,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:38:27,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:38:27,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:38:27,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:38:28,002 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:38:28,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:38:28,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:38:28,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:38:28,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:38:28,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:38:28,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:38:28,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:38:28,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:38:28,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:38:28,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:38:28,219 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:38:28,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:38:28,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:38:28,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:38:28,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:38:28,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:38:28,341 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:38:28,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:38:28,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:38:28,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:38:28,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:38:28,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:38:28,446 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:38:28,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:38:28,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:38:28,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:38:28,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:38:28,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:38:28,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:38:28,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:38:28,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:38:28,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:38:28,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:38:28,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:38:28,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:38:28,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:38:28,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:38:28,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:38:28,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:38:28,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:38:28,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:38:28,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:38:28,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:38:28,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:38:28,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:38:28,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:38:28,849 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:38:28,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:38:28,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:38:28,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:38:28,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:38:28,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:38:28,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:38:28,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:38:29,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:38:29,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:38:29,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:38:29,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:38:29,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:38:29,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:38:29,102 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 13:38:29,104 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:38:29,108 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:38:29,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:38:30,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:38:30,142 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:38:30,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:38:30,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:38:30,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:38:30,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:38:30,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:38:30,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:38:30,255 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:38:30,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:38:30,859 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:38:30,859 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.75599980354. 2022-02-08 13:38:30,861 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:38:30,861 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:38:31,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:38:32,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:38:32,275 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:38:32,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:38:32,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:38:32,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:38:32,341 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:38:32,364 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:38:32,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:38:32,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:38:32,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:38:33,198 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:38:33,200 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.33899998665. 2022-02-08 13:38:33,200 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:38:33,201 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:38:34,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:38:34,678 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:38:34,865 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:38:34,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:38:34,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:38:34,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:38:34,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:38:34,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:38:34,993 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:38:35,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:38:35,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:38:35,799 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:38:35,799 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.59899997711. 2022-02-08 13:38:35,799 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:38:35,802 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:38:36,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:38:37,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:38:37,500 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:38:37,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:38:37,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:38:37,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:38:37,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:38:37,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:38:37,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:38:37,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:38:37,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:38:38,381 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:38:38,381 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.58200001717. 2022-02-08 13:38:38,381 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:38:38,384 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:38:39,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:38:39,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:38:40,038 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:38:40,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:38:40,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:38:40,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:38:40,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:38:40,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:38:40,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:38:40,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:38:40,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:38:41,351 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:38:41,351 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.96900010109. 2022-02-08 13:38:41,351 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:38:41,352 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:38:42,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:38:42,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:38:42,512 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:38:42,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:38:42,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:38:42,558 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:38:42,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:38:42,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:38:42,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:38:42,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:38:42,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:38:43,760 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:38:43,762 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.41100001335. 2022-02-08 13:38:43,762 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:38:43,763 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:38:44,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:38:45,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:38:45,872 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:38:45,874 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:38:45,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:38:45,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:38:45,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:38:45,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:38:46,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:38:46,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:38:46,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:38:49,917 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:38:49,917 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.15499997139. 2022-02-08 13:38:49,917 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:38:49,918 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 5 queries submitted, 0 failed, 17392 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:38:50,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:38:50,936 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:38:51,082 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:38:51,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:38:51,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:38:51,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:38:51,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:38:51,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:38:51,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:38:51,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:38:51,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:38:56,065 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:38:56,065 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.14899992943. 2022-02-08 13:38:56,065 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:38:56,068 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:38:56,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:38:57,448 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:38:57,691 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:38:57,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:38:57,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:38:57,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:38:57,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:38:57,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:38:57,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:38:57,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:38:57,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:39:03,382 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:39:03,382 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 7.31699991226. ############################################################ # # RAM FootPrint: 4695.7109375 # ############################################################ 2022-02-08 13:39:03,414 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 13:39:03,414 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 67 seconds. Reason: Iteration:19 waiting to kill memc on all nodes Loader_GleamBookUsers0__default_VolumeCollection11_5_1644308335.44 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection0_-9_1644308349.6 5 queries submitted, 0 failed, 17419 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:40:10,420 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 13:40:10,421 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 13:40:15,423 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:40:16,227 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:40:16,823 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:40:17,122 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:40:17,124 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:40:17,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:40:17,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:40:17,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:40:17,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:40:17,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:40:17,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:40:17,408 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:40:17,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:40:17,490 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['81388 memcached\n'], []) 2022-02-08 13:40:17,492 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 13:40:17,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:40:17,572 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['81388 memcached\n'], []) 2022-02-08 13:40:17,572 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:40:17,573 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:40:18,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:40:18,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:40:19,132 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:40:19,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:40:19,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:40:19,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:40:19,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:40:19,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:40:19,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:40:19,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:40:19,352 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:40:19,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:40:19,421 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['7567 memcached\n'], []) 2022-02-08 13:40:19,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 13:40:19,459 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:40:19,490 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['7567 memcached\n'], []) 2022-02-08 13:40:19,490 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:40:19,493 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:40:19,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:40:20,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:40:20,404 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:40:20,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:40:20,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:40:20,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:40:20,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:40:20,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:40:20,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:40:20,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:40:20,628 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:40:20,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:40:20,934 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['122954 memcached\n'], []) 2022-02-08 13:40:20,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 13:40:20,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:40:21,020 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['122954 memcached\n'], []) 2022-02-08 13:40:21,023 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:40:21,026 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:40:21,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:40:22,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:40:22,691 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:40:22,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:40:22,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:40:22,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:40:22,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:40:22,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:40:22,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:40:22,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:40:22,858 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:40:22,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:40:22,931 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['15755 memcached\n'], []) 2022-02-08 13:40:22,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 13:40:22,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:40:22,987 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['15755 memcached\n'], []) 2022-02-08 13:40:22,989 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:40:22,990 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:40:23,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:40:23,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:40:23,759 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:40:23,760 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:40:23,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:40:23,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:40:23,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:40:23,905 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:40:23,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:40:24,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:40:24,107 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:40:24,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:40:24,252 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['9642 memcached\n'], []) 2022-02-08 13:40:24,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 13:40:24,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:40:24,308 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['9642 memcached\n'], []) 2022-02-08 13:40:24,308 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:40:24,311 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:40:25,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:40:26,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:40:26,250 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:40:26,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:40:26,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:40:26,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:40:26,329 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:40:26,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:40:26,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:40:26,418 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:40:26,434 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:40:26,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:40:26,499 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['44823 memcached\n'], []) 2022-02-08 13:40:26,499 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 13:40:26,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:40:26,549 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['44823 memcached\n'], []) 2022-02-08 13:40:26,551 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:40:26,552 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:40:27,418 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:40:27,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:40:28,217 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:40:28,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:40:28,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:40:28,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:40:28,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:40:28,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:40:28,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:40:28,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:40:28,371 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:40:28,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:40:28,420 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['108127 memcached\n'], []) 2022-02-08 13:40:28,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 13:40:28,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:40:28,476 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['108127 memcached\n'], []) 2022-02-08 13:40:28,477 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:40:28,480 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:40:28,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:40:29,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:40:29,670 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:40:29,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:40:29,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:40:29,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:40:29,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:40:29,760 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:40:29,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:40:29,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:40:29,809 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:40:29,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:40:29,910 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['123341 memcached\n'], []) 2022-02-08 13:40:29,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 13:40:29,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:40:29,959 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['123341 memcached\n'], []) 2022-02-08 13:40:29,960 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:40:29,963 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:40:30,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:40:30,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:40:30,664 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:40:30,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:40:30,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:40:30,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:40:31,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:40:31,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:40:31,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:40:31,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:40:31,119 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:40:31,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:40:31,217 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['60772 memcached\n'], []) 2022-02-08 13:40:31,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 13:40:31,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:40:31,265 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['60772 memcached\n'], []) 2022-02-08 13:40:31,265 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:40:31,267 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:40:31,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:40:31,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:40:31,698 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:40:31,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:40:31,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:40:31,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:40:31,742 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:40:31,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:40:31,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:40:31,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:40:31,806 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:40:31,822 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 13:40:31,852 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:40:31,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:40:38,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:40:39,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:40:40,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:40:40,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:40:42,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:40:43,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:40:43,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:40:43,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:40:43,440 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 13:40:43,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:40:43,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:40:43,520 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:40:43,533 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 13:40:43,535 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:40:43,536 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:40:43,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:40:43,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:40:43,913 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:40:43,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:40:43,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:40:43,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:40:43,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:40:43,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:40:43,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:40:43,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:40:44,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:40:44,023 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 13:40:44,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:40:44,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 17443 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:40:51,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:40:52,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:40:53,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:40:53,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:40:55,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:40:55,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:40:55,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:40:55,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:40:55,881 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 13:40:55,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:40:55,895 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:40:55,974 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:40:55,989 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 13:40:55,990 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:40:55,992 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 13:40:56,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:40:56,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 13:40:56,368 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:40:56,368 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 13:40:56,382 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 13:40:56,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 13:40:56,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 13:40:56,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 13:40:56,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 13:40:56,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 13:40:56,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:40:56,482 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 13:40:56,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:40:56,520 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:40:56,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:40:56,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:40:56,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:40:56,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:40:56,591 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 13:40:56,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:40:56,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:40:56,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:40:56,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:40:56,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:40:56,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:40:56,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:40:56,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:40:56,793 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:40:56,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:40:56,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:40:56,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:40:56,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:40:56,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:40:56,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:40:56,997 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:40:57,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:40:57,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:40:57,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:40:57,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:40:57,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:40:57,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:40:57,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:40:57,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:40:57,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:40:57,177 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:40:57,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:40:57,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:40:57,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:40:57,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:40:57,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:40:57,280 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:40:57,298 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:40:57,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:40:57,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:40:57,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:40:57,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:40:57,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:40:57,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:40:57,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:40:57,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:40:57,448 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:40:57,464 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:40:57,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:40:57,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:40:57,515 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:40:57,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:40:57,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:40:57,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:40:57,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:40:57,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:40:57,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:40:57,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:40:57,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:40:57,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:40:57,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:40:57,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:40:57,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:40:57,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:40:57,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:40:57,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:40:57,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:40:57,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:40:57,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:40:57,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:40:57,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:40:57,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:40:57,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:40:57,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:40:57,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:40:57,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:40:57,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:40:58,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:40:58,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:40:58,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:40:58,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:40:58,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:40:58,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:40:58,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:40:58,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:40:58,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:40:58,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:40:58,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:40:58,203 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:40:58,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:40:58,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:40:58,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:40:58,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:40:58,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:40:58,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:40:58,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:40:58,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:40:58,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:40:58,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:40:58,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:40:58,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:40:58,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:40:58,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:40:58,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:40:58,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:40:58,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:40:58,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:40:58,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:40:58,565 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 13:40:58,566 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:40:58,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:40:59,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:40:59,858 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:40:59,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:40:59,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:40:59,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:40:59,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:40:59,959 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:40:59,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:40:59,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:41:00,002 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:41:00,017 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 13:41:00,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:41:00,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:41:06,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:41:07,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:41:08,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:41:08,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:41:08,608 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 13:41:08,609 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:41:08,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:41:08,709 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:41:08,723 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 13:41:08,723 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:41:08,726 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 13:41:08,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:41:09,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 13:41:09,092 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:41:09,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 13:41:09,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 13:41:09,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 13:41:09,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 13:41:09,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 13:41:09,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 13:41:09,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 13:41:09,239 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:41:09,256 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 13:41:09,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:41:09,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:41:09,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:41:09,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:41:09,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:41:09,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:41:09,388 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 13:41:09,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:41:09,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:41:09,423 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:41:09,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:09,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:41:09,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:09,538 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:09,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:09,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:09,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:09,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:09,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:09,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:09,760 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:09,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:41:09,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:09,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:09,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:09,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:09,858 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:09,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:09,891 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:09,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:09,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:09,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:41:09,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:09,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:09,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:10,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:10,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:10,051 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:10,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:10,086 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:10,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:10,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:41:10,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:10,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:10,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:10,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:10,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:10,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:10,234 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:10,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:10,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:10,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:41:10,303 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:10,321 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:10,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:10,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:10,369 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:10,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:10,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:10,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:10,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:10,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:41:10,477 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:10,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:10,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:10,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:10,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:10,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:10,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:10,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:10,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:10,630 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:41:10,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:10,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:10,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:10,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:10,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:10,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:10,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:10,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:10,785 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:10,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:41:10,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:10,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:10,854 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:10,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:10,890 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:10,905 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:10,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:10,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:10,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:10,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:41:11,010 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:11,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:11,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:11,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:11,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:11,086 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:11,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:11,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:11,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:11,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:41:11,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:11,183 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:11,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:11,214 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:11,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:11,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:11,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:11,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:11,296 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 13:41:11,299 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:41:11,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:41:11,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:41:11,710 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:41:11,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:41:11,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:41:11,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:41:11,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:41:11,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:41:11,789 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:41:11,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:41:11,823 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:41:11,842 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 13:41:11,869 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:41:11,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:41:18,244 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:41:19,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:41:19,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:41:20,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:41:22,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:41:22,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:41:22,887 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:41:23,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:41:23,052 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 13:41:23,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:41:23,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:41:23,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:41:23,151 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 13:41:23,151 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:41:23,154 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:41:23,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:41:23,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:41:23,642 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:41:23,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:41:23,660 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:41:23,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:41:23,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:41:23,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:41:23,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:41:23,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:41:23,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:41:23,812 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 13:41:23,836 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:41:23,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:41:31,665 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:41:32,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:41:33,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:41:33,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:41:35,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:41:35,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:41:36,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:41:36,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:41:36,299 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 13:41:36,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:41:36,342 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:41:36,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:41:36,467 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 13:41:36,467 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:41:36,470 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 13:41:36,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:41:36,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 13:41:36,881 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:41:36,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 13:41:36,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 13:41:36,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 13:41:36,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 13:41:36,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 13:41:36,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 13:41:36,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 13:41:36,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:41:37,013 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 13:41:37,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:41:37,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:41:37,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:41:37,094 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:41:37,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:41:37,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:41:37,145 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 13:41:37,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:41:37,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:41:37,183 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:41:37,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:37,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:41:37,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:37,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:37,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:37,377 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:37,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:37,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:37,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:37,507 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:37,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:37,573 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:41:37,594 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:37,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:37,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:37,645 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:37,664 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:37,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:37,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:37,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:37,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:37,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:41:37,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:37,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:37,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:37,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:37,854 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:37,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:37,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:37,911 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:37,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:37,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:41:37,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:37,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:38,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:38,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:38,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:38,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:38,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:38,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:38,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:38,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:41:38,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:38,186 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:38,204 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:38,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:38,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:38,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:38,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:38,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:38,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:38,346 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:41:38,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:38,392 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:38,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:38,433 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:38,450 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:38,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:38,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:38,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:38,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:38,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:41:38,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:38,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:38,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:38,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:38,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:38,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:38,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:38,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:38,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:38,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:41:38,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:38,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:38,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:38,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:38,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:38,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:38,880 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:38,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:38,918 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:38,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:41:38,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:38,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:38,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:39,017 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:39,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:39,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:39,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:39,094 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:39,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:39,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:41:39,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:39,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:39,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:39,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:39,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:39,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:39,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:39,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:39,293 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 13:41:39,296 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 13:41:39,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:41:39,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 13:41:39,665 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:41:39,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 13:41:39,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 13:41:39,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 13:41:39,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 13:41:39,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 13:41:39,749 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 13:41:39,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 13:41:39,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:41:39,792 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 13:41:39,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:41:39,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:41:39,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:41:39,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:41:39,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:41:39,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:41:39,907 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 13:41:39,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:41:39,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:41:39,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:41:39,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:39,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:41:40,010 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:40,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:40,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:40,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:40,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:40,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:40,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:40,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:41:40,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:40,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:41:40,315 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:40,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:40,349 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:40,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:40,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:40,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:40,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:40,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:41:40,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:40,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:41:40,500 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:40,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:40,535 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:40,553 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:40,571 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:40,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:40,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:40,641 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:41:40,658 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:40,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:41:40,691 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:40,710 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:40,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:40,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:40,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:40,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:40,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:40,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:41:40,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:40,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:41:40,867 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:40,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:40,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:40,917 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:40,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:40,950 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:40,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:40,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:41:41,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:41,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:41:41,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:41,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:41,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:41,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:41,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:41,122 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:41,140 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:41,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:41:41,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:41,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:41:41,214 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:41,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:41,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:41,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:41,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:41,306 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:41,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:41,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:41:41,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:41,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:41:41,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:41,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:41,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:41,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:41,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:41,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:41,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:41,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:41:41,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:41,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:41:41,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:41,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:41,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:41,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:41,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:41,655 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:41,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:41,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:41:41,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:41,719 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:41:41,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:41,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:41,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:41,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:41,808 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:41,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:41,839 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:41,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:41:41,871 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 13:41:41,872 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:41:42,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:41:43,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:41:43,312 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:41:43,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:41:43,334 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:41:43,357 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:41:43,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:41:43,392 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:41:43,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:41:43,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:41:43,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:41:43,490 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 13:41:43,522 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:41:43,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 17468 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:41:50,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:41:51,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:41:52,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:41:52,650 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:41:54,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:41:54,790 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:41:54,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:41:55,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:41:55,098 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 13:41:55,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:41:55,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:41:55,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:41:55,242 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 13:41:55,242 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:41:55,244 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:41:55,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:41:55,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:41:55,739 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:41:55,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:41:55,760 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:41:55,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:41:55,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:41:55,845 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:41:55,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:41:55,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:41:55,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:41:55,931 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 13:41:55,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:41:55,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:42:03,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:42:04,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:42:05,086 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:42:05,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:42:05,657 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 13:42:05,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:42:05,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:42:05,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:42:05,746 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 13:42:05,747 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:42:05,750 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:42:05,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:42:06,263 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:42:06,351 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:42:06,351 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:42:06,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:42:06,394 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:42:06,410 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:42:06,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:42:06,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:42:06,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:42:06,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:42:06,496 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 13:42:06,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:42:06,543 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:42:13,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:42:14,186 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:42:14,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:42:15,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:42:15,153 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 13:42:15,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:42:15,170 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:42:15,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:42:15,246 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 13:42:15,246 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:42:15,250 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:42:16,154 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:42:16,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:42:16,880 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:42:16,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:42:16,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:42:16,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:42:16,948 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:42:16,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:42:17,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:42:17,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:42:17,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:42:17,091 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 13:42:17,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:42:17,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:42:26,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:42:27,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:42:28,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:42:28,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:42:32,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:42:32,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:42:32,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:42:33,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:42:33,210 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 13:42:33,210 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:42:33,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:42:33,313 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:42:33,335 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 13:42:33,335 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:42:33,348 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 13:42:33,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:42:33,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 13:42:33,749 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:42:33,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 13:42:33,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 13:42:33,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 13:42:33,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 13:42:33,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 13:42:33,831 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 13:42:33,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 13:42:33,862 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:42:33,880 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 13:42:33,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:42:33,930 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:42:33,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:42:33,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:42:33,976 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:42:33,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:42:34,005 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 13:42:34,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:42:34,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:42:34,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:42:34,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:42:34,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:42:34,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:42:34,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:42:34,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:42:34,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:42:34,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:42:34,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:42:34,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:42:34,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:42:34,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:42:34,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:42:34,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:42:34,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:42:34,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:42:34,476 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:42:34,494 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:42:34,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:42:34,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:42:34,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:42:34,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:42:34,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:42:34,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:42:34,619 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:42:34,640 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:42:34,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:42:34,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:42:34,693 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:42:34,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:42:34,729 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:42:34,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:42:34,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:42:34,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:42:34,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:42:34,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:42:34,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:42:34,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:42:34,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:42:34,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:42:34,905 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:42:34,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:42:34,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:42:34,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:42:34,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:42:35,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:42:35,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:42:35,036 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:42:35,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:42:35,071 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:42:35,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:42:35,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:42:35,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:42:35,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:42:35,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:42:35,190 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:42:35,209 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:42:35,227 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:42:35,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:42:35,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:42:35,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:42:35,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:42:35,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:42:35,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:42:35,354 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:42:35,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:42:35,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:42:35,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:42:35,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:42:35,443 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:42:35,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:42:35,477 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:42:35,494 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:42:35,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:42:35,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:42:35,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:42:35,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:42:35,584 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:42:35,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:42:35,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:42:35,637 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:42:35,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:42:35,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:42:35,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:42:35,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:42:35,720 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:42:35,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:42:35,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:42:35,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:42:35,786 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:42:35,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:42:35,822 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:42:35,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:42:35,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:42:35,872 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:42:35,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:42:35,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:42:35,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:42:35,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:42:35,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:42:35,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:42:36,006 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 13:42:36,006 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:42:36,009 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:42:36,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:42:36,941 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:42:37,049 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:42:37,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:42:37,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:42:37,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:42:37,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:42:37,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:42:37,132 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:42:37,160 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:42:37,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:42:37,760 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:42:37,760 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.7539999485. 2022-02-08 13:42:37,760 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:42:37,763 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:42:38,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:42:38,569 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:42:38,684 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:42:38,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:42:38,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:42:38,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:42:38,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:42:38,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:42:38,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:42:38,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:42:38,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:42:39,601 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:42:39,601 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.84100008011. 2022-02-08 13:42:39,602 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:42:39,605 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:42:40,492 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:42:41,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:42:41,459 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:42:41,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:42:41,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:42:41,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:42:41,555 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:42:41,589 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:42:41,614 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:42:41,648 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:42:41,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:42:42,362 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:42:42,362 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.75999999046. 2022-02-08 13:42:42,362 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:42:42,365 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:42:43,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:42:43,857 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:42:44,051 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:42:44,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:42:44,073 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:42:44,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:42:44,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:42:44,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:42:44,161 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:42:44,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:42:44,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:42:44,960 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:42:44,960 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.59800004959. 2022-02-08 13:42:44,961 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:42:44,963 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:42:45,964 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:42:46,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:42:46,661 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:42:46,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:42:46,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:42:46,704 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:42:46,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:42:46,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:42:46,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:42:46,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:42:46,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:42:47,503 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:42:47,512 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.54999995232. 2022-02-08 13:42:47,512 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:42:47,515 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:42:48,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:42:48,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:42:48,677 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:42:48,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:42:48,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:42:48,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:42:48,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:42:48,756 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:42:48,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:42:48,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:42:48,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:42:49,408 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:42:49,408 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.89699983597. 2022-02-08 13:42:49,408 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:42:49,411 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:42:50,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 5 queries submitted, 0 failed, 17492 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:42:50,647 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:42:50,823 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:42:50,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:42:50,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:42:50,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:42:50,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:42:50,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:42:50,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:42:50,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:42:50,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:42:51,496 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:42:51,496 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.08700013161. 2022-02-08 13:42:51,496 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:42:51,497 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:42:52,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:42:52,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:42:53,105 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:42:53,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:42:53,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:42:53,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:42:53,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:42:53,200 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:42:53,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:42:53,265 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:42:53,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:42:53,911 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:42:53,913 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.41700005531. 2022-02-08 13:42:53,913 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:42:53,914 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:42:54,099 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:42:54,526 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:42:55,358 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:42:55,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:42:55,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:42:55,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:42:55,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:42:55,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:42:55,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:42:55,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:42:55,964 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:42:57,298 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 13:42:57,298 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 13:42:59,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:42:59,694 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 13:42:59,694 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 13:43:01,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:43:02,322 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 13:43:02,322 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 13:43:04,323 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:43:04,763 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 13:43:04,763 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 13:43:06,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:43:07,105 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 13:43:07,107 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 13:43:09,107 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:43:09,319 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4206 2022-02-08 13:43:09,319 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.120.170 2022-02-08 13:43:11,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:43:12,915 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:43:12,917 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 19.0039999485. ############################################################ # # RAM FootPrint: 4151.375 # ############################################################ 2022-02-08 13:43:12,951 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L828 2022-02-08 13:43:12,951 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 111 seconds. Reason: Iteration:20 waiting to kill memc on all nodes 5 queries submitted, 0 failed, 17515 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 17539 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:45:03,960 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L844 2022-02-08 13:45:03,960 | test | INFO | Thread-19496 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2022-02-08 13:45:08,964 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:45:09,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:45:10,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:45:10,631 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:45:10,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:45:10,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:45:10,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:45:10,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:45:10,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:45:10,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:45:10,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:45:10,905 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.116 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:45:10,905 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:45:10,986 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.116 - Process info before sending signal: (['82125 memcached\n'], []) 2022-02-08 13:45:10,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: kill -9 $(pgrep memcached) 2022-02-08 13:45:11,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: pgrep -l memcached 2022-02-08 13:45:11,121 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.116 - Process info after sending signal: (['82125 memcached\n'], []) 2022-02-08 13:45:11,122 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:45:11,125 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:45:12,211 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:45:12,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:45:12,947 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:45:12,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:45:12,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:45:12,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:45:13,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:45:13,033 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:45:13,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:45:13,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:45:13,157 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.123 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:45:13,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:45:13,224 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.123 - Process info before sending signal: (['8333 memcached\n'], []) 2022-02-08 13:45:13,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: kill -9 $(pgrep memcached) 2022-02-08 13:45:13,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: pgrep -l memcached 2022-02-08 13:45:13,301 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.123 - Process info after sending signal: (['8333 memcached\n'], []) 2022-02-08 13:45:13,302 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:45:13,303 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:45:14,316 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:45:15,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:45:15,484 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:45:15,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:45:15,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:45:15,536 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:45:15,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:45:15,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:45:15,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:45:15,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:45:15,713 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.124 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:45:15,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:45:15,782 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.124 - Process info before sending signal: (['123826 memcached\n'], []) 2022-02-08 13:45:15,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: kill -9 $(pgrep memcached) 2022-02-08 13:45:15,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: pgrep -l memcached 2022-02-08 13:45:15,861 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.124 - Process info after sending signal: (['123826 memcached\n'], []) 2022-02-08 13:45:15,862 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:45:15,865 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:45:16,760 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:45:17,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:45:18,046 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:45:18,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:45:18,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:45:18,091 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:45:18,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:45:18,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:45:18,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:45:18,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:45:18,217 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.126 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:45:18,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:45:18,301 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.126 - Process info before sending signal: (['16568 memcached\n'], []) 2022-02-08 13:45:18,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: kill -9 $(pgrep memcached) 2022-02-08 13:45:18,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: pgrep -l memcached 2022-02-08 13:45:18,369 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.126 - Process info after sending signal: (['16568 memcached\n'], []) 2022-02-08 13:45:18,369 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:45:18,372 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:45:19,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:45:20,793 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:45:21,013 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:45:21,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:45:21,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:45:21,068 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:45:21,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:45:21,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:45:21,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:45:21,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:45:21,174 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.127 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:45:21,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:45:21,240 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.127 - Process info before sending signal: (['10442 memcached\n'], []) 2022-02-08 13:45:21,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: kill -9 $(pgrep memcached) 2022-02-08 13:45:21,272 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: pgrep -l memcached 2022-02-08 13:45:21,296 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.127 - Process info after sending signal: (['10442 memcached\n'], []) 2022-02-08 13:45:21,296 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:45:21,301 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:45:22,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:45:23,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:45:23,447 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:45:23,447 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:45:23,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:45:23,515 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:45:23,529 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:45:23,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:45:23,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:45:23,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:45:23,625 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.129 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:45:23,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:45:23,687 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.129 - Process info before sending signal: (['45588 memcached\n'], []) 2022-02-08 13:45:23,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: kill -9 $(pgrep memcached) 2022-02-08 13:45:23,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: pgrep -l memcached 2022-02-08 13:45:23,740 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.129 - Process info after sending signal: (['45588 memcached\n'], []) 2022-02-08 13:45:23,740 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:45:23,743 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:45:24,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:45:25,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:45:25,683 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:45:25,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:45:25,698 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:45:25,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:45:25,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:45:25,766 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:45:25,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:45:25,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:45:25,834 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.135 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:45:25,835 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:45:25,903 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.135 - Process info before sending signal: (['108869 memcached\n'], []) 2022-02-08 13:45:25,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: kill -9 $(pgrep memcached) 2022-02-08 13:45:25,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: pgrep -l memcached 2022-02-08 13:45:25,964 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.135 - Process info after sending signal: (['108869 memcached\n'], []) 2022-02-08 13:45:25,964 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:45:25,967 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:45:26,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:45:27,311 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:45:27,509 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:45:27,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:45:27,528 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:45:27,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:45:27,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:45:27,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:45:27,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:45:27,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:45:27,653 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.121.115 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:45:27,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:45:27,717 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.121.115 - Process info before sending signal: (['124080 memcached\n'], []) 2022-02-08 13:45:27,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: kill -9 $(pgrep memcached) 2022-02-08 13:45:27,744 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: pgrep -l memcached 2022-02-08 13:45:27,769 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.121.115 - Process info after sending signal: (['124080 memcached\n'], []) 2022-02-08 13:45:27,770 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:45:27,773 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:45:28,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:45:29,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:45:29,581 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:45:29,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:45:29,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:45:29,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:45:29,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:45:29,661 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:45:29,677 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:45:29,701 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:45:29,716 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:534] 172.23.120.170 - Sending 9 signal to process: memcached, service: memcached 2022-02-08 13:45:29,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:45:29,780 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:541] 172.23.120.170 - Process info before sending signal: (['61542 memcached\n'], []) 2022-02-08 13:45:29,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: kill -9 $(pgrep memcached) 2022-02-08 13:45:29,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: pgrep -l memcached 2022-02-08 13:45:29,832 | infra | DEBUG | Thread-19496 | [remote_util:kill_process:546] 172.23.120.170 - Process info after sending signal: (['61542 memcached\n'], []) 2022-02-08 13:45:29,832 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:45:29,835 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:45:29,921 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:45:30,141 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:45:30,204 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:45:30,204 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:45:30,220 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:45:30,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:45:30,257 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:45:30,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:45:30,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:45:30,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:45:30,325 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:45:30,341 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.120.170: No crash files found 2022-02-08 13:45:30,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:45:30,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:45:37,165 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:45:38,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:45:38,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:45:39,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:45:40,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:45:41,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:45:41,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:45:41,549 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:45:41,563 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 13:45:41,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:45:41,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:45:41,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:45:41,670 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 13:45:41,670 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:45:41,671 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:45:41,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:45:41,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:45:42,043 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:45:42,045 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:45:42,058 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:45:42,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:45:42,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:45:42,098 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:45:42,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:45:42,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:45:42,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:45:42,151 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 13:45:42,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:45:42,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:45:49,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 5 queries submitted, 0 failed, 17561 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:45:50,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:45:51,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:45:51,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:45:53,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:45:53,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:45:53,739 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:45:53,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:45:53,877 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 13:45:53,877 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:45:53,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:45:53,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:45:54,009 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 13:45:54,009 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:45:54,012 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 13:45:54,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:45:54,332 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 13:45:54,397 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:45:54,397 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 13:45:54,413 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 13:45:54,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 13:45:54,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 13:45:54,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 13:45:54,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 13:45:54,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 13:45:54,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:45:54,513 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 13:45:54,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:45:54,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:45:54,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:45:54,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:45:54,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:45:54,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:45:54,638 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 13:45:54,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:45:54,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:45:54,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:45:54,683 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:45:54,713 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:45:54,752 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:45:54,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:45:54,823 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:45:54,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:45:54,892 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:45:54,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:45:54,960 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:45:54,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:45:55,026 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:45:55,043 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:45:55,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:45:55,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:45:55,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:45:55,114 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:45:55,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:45:55,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:45:55,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:45:55,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:45:55,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:45:55,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:45:55,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:45:55,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:45:55,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:45:55,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:45:55,301 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:45:55,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:45:55,335 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:45:55,352 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:45:55,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:45:55,387 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:45:55,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:45:55,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:45:55,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:45:55,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:45:55,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:45:55,489 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:45:55,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:45:55,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:45:55,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:45:55,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:45:55,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:45:55,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:45:55,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:45:55,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:45:55,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:45:55,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:45:55,688 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:45:55,706 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:45:55,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:45:55,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:45:55,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:45:55,779 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:45:55,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:45:55,821 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:45:55,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:45:55,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:45:55,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:45:55,897 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:45:55,915 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:45:55,934 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:45:55,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:45:55,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:45:55,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:45:56,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:45:56,025 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:45:56,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:45:56,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:45:56,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:45:56,094 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:45:56,111 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:45:56,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:45:56,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:45:56,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:45:56,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:45:56,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:45:56,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:45:56,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:45:56,253 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:45:56,269 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:45:56,286 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:45:56,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:45:56,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:45:56,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:45:56,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:45:56,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:45:56,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:45:56,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:45:56,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:45:56,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:45:56,480 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:45:56,497 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:45:56,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:45:56,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:45:56,548 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:45:56,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:45:56,581 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:45:56,596 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:45:56,612 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:45:56,628 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 13:45:56,630 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:45:56,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:45:56,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:45:57,033 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:45:57,035 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:45:57,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:45:57,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:45:57,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:45:57,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:45:57,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:45:57,119 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:45:57,134 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:45:57,151 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 13:45:57,178 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:45:57,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:46:04,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:46:05,608 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:46:06,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:46:06,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:46:06,969 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 13:46:06,970 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:46:07,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:46:07,278 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:46:07,309 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 13:46:07,309 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:46:07,312 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 13:46:07,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:46:07,671 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 13:46:07,736 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:46:07,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 13:46:07,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 13:46:07,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 13:46:07,778 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 13:46:07,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 13:46:07,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 13:46:07,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 13:46:07,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:46:07,858 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 13:46:07,890 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:46:07,905 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:46:07,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:46:07,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:46:07,957 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:46:07,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:46:07,989 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 13:46:07,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:46:08,013 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:46:08,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:46:08,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:08,082 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:46:08,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:08,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:08,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:08,229 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:08,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:08,299 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:08,336 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:08,371 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:08,403 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:08,418 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:46:08,436 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:08,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:08,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:08,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:08,502 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:08,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:08,533 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:08,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:08,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:08,586 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:46:08,607 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:08,627 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:08,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:08,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:08,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:08,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:08,714 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:08,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:08,747 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:08,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:46:08,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:08,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:08,813 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:08,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:08,844 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:08,859 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:08,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:08,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:08,910 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:08,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:46:08,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:08,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:08,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:09,009 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:09,030 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:09,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:09,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:09,088 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:09,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:09,124 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:46:09,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:09,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:09,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:09,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:09,226 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:09,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:09,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:09,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:09,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:09,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:46:09,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:09,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:09,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:09,391 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:09,407 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:09,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:09,443 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:09,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:09,477 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:09,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:46:09,515 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:09,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:09,548 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:09,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:09,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:09,599 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:09,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:09,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:09,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:09,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:46:09,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:09,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:09,717 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:09,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:09,753 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:09,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:09,792 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:09,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:09,826 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:09,846 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:46:09,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:09,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:09,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:09,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:09,937 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:09,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:09,969 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:09,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:10,002 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 13:46:10,005 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:46:10,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:46:10,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:46:10,378 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:46:10,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:46:10,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:46:10,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:46:10,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:46:10,441 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:46:10,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:46:10,469 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:46:10,483 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:46:10,497 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 13:46:10,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:46:10,539 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:46:17,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:46:18,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:46:18,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:46:19,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:46:21,510 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:46:21,811 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:46:21,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:46:22,142 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:46:22,158 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 13:46:22,158 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:46:22,173 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:46:22,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:46:22,250 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 13:46:22,250 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:46:22,253 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:46:22,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:46:22,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:46:22,904 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:46:22,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:46:22,923 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:46:22,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:46:23,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:46:23,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:46:23,072 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:46:23,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:46:23,121 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:46:23,141 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 13:46:23,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:46:23,196 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:46:31,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:46:32,426 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:46:32,943 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:46:33,414 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:46:35,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:46:35,828 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:46:35,992 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:46:36,181 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:46:36,246 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 13:46:36,247 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:46:36,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:46:36,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:46:36,377 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 13:46:36,377 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:46:36,378 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 13:46:36,460 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:46:36,701 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 13:46:36,776 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:46:36,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 13:46:36,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 13:46:36,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 13:46:36,825 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 13:46:36,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 13:46:36,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 13:46:36,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 13:46:36,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:46:36,918 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 13:46:36,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:46:36,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:46:36,983 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:46:37,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:46:37,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:46:37,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:46:37,052 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 13:46:37,052 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:46:37,069 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:46:37,089 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:46:37,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:37,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:46:37,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:37,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:37,256 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:37,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:37,331 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:37,365 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:37,401 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:37,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:37,471 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:37,493 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:46:37,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:37,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:37,551 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:37,568 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:37,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:37,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:37,624 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:37,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:37,663 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:37,681 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:46:37,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:37,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:37,743 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:37,763 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:37,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:37,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:37,823 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:37,842 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:37,861 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:37,884 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:46:37,904 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:37,924 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:37,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:37,967 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:37,984 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:38,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:38,029 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:38,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:38,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:38,108 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:46:38,137 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:38,157 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:38,176 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:38,193 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:38,213 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:38,233 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:38,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:38,283 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:38,305 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:38,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:46:38,348 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:38,367 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:38,384 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:38,404 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:38,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:38,448 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:38,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:38,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:38,512 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:38,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:46:38,552 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:38,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:38,592 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:38,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:38,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:38,654 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:38,674 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:38,696 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:38,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:38,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:46:38,759 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:38,776 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:38,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:38,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:38,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:38,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:38,868 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:38,885 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:38,903 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:38,920 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:46:38,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:38,954 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:38,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:38,989 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:39,006 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:39,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:39,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:39,061 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:39,078 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:39,094 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:46:39,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:39,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:39,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:39,168 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:39,184 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:39,201 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:39,221 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:39,240 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:39,256 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 13:46:39,259 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 13:46:39,361 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:46:39,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 13:46:39,653 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:46:39,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 13:46:39,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 13:46:39,680 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 13:46:39,694 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 13:46:39,711 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 13:46:39,724 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 13:46:39,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 13:46:39,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:46:39,769 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 13:46:39,802 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:46:39,818 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:46:39,834 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:46:39,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:46:39,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:46:39,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:46:39,901 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 13:46:39,901 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:46:39,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:46:39,931 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:46:39,947 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:39,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:46:40,020 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:40,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:40,105 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:40,138 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:40,174 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:40,207 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:40,243 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:40,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:46:40,309 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:40,326 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:46:40,344 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:40,359 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:40,375 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:40,390 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:40,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:40,421 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:40,437 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:40,453 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:46:40,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:40,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:46:40,506 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:40,523 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:40,540 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:40,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:40,578 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:40,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:40,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:40,632 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:46:40,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:40,667 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:46:40,686 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:40,703 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:40,721 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:40,737 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:40,755 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:40,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:40,795 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:40,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:46:40,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:40,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:46:40,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:40,888 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:40,907 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:40,926 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:40,944 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:40,961 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:40,980 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:41,000 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:46:41,019 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:41,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:46:41,056 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:41,075 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:41,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:41,109 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:41,127 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:41,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:41,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:41,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:46:41,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:41,214 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:46:41,232 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:41,250 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:41,266 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:41,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:41,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:41,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:41,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:41,355 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:46:41,372 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:41,388 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:46:41,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:41,423 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:41,440 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:41,457 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:41,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:41,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:41,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:41,545 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:46:41,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:41,576 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:46:41,595 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:41,611 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:41,625 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:41,642 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:41,657 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:41,676 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:41,693 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:41,709 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:46:41,723 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:41,740 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:46:41,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:41,780 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:41,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:41,816 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:41,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:41,848 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:41,864 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:41,881 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:46:41,897 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 13:46:41,900 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:46:42,769 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:46:43,237 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:46:43,378 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:46:43,380 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:46:43,398 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:46:43,424 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:46:43,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:46:43,467 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:46:43,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:46:43,519 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:46:43,538 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:46:43,558 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.123: No crash files found 2022-02-08 13:46:43,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:46:43,602 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 17583 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:46:51,308 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:46:52,282 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:46:52,871 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:46:53,408 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:46:55,815 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:46:56,135 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:46:56,289 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:46:56,430 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:46:56,470 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 13:46:56,470 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:46:56,486 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:46:56,585 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:46:56,602 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 13:46:56,604 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:46:56,605 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:46:56,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:46:56,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:46:57,101 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:46:57,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:46:57,125 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:46:57,144 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:46:57,163 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:46:57,181 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:46:57,198 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:46:57,216 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:46:57,234 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:46:57,256 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 13:46:57,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:46:57,318 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:47:04,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:47:06,003 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:47:06,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:47:07,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:47:07,644 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 13:47:07,644 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:47:07,673 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:47:07,733 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:47:07,759 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 13:47:07,760 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:47:07,765 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:47:08,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:47:08,908 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:47:09,038 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:47:09,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:47:09,055 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:47:09,092 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:47:09,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:47:09,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:47:09,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:47:09,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:47:09,217 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:47:09,233 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 13:47:09,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:47:09,276 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:47:16,063 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:47:17,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:47:17,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:47:17,994 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:47:18,013 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 13:47:18,015 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:47:18,032 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:47:18,081 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:47:18,098 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 13:47:18,099 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:47:18,101 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 13:47:18,273 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:47:18,548 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:47:18,736 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:47:18,736 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:47:18,760 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:47:18,773 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:47:18,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:47:18,865 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:47:18,913 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:47:18,928 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:47:18,946 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:47:18,983 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 13:47:19,016 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:47:19,053 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:47:28,038 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:47:29,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:47:30,562 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:47:31,338 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:47:34,798 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:47:35,292 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:47:35,591 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2022-02-08 13:47:35,841 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:47:35,917 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 13:47:35,918 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:47:35,940 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:47:36,046 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:47:36,085 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 13:47:36,085 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:47:36,088 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 13:47:36,194 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:47:36,420 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 13:47:36,480 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:47:36,482 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 13:47:36,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 13:47:36,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 13:47:36,525 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 13:47:36,542 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 13:47:36,559 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 13:47:36,572 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 13:47:36,588 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 13:47:36,605 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 13:47:36,635 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 13:47:36,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:47:36,670 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 13:47:36,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:47:36,700 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 13:47:36,716 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 13:47:36,730 | test | DEBUG | Thread-19496 | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 13:47:36,732 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 13:47:36,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 13:47:36,767 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 13:47:36,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:47:36,812 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 13:47:36,851 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:47:36,882 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:47:36,927 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:47:36,966 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:47:37,005 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:47:37,042 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:47:37,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:47:37,115 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 13:47:37,147 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:47:37,164 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz | grep -v 'fatal remote' 2022-02-08 13:47:37,180 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:47:37,197 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:47:37,214 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:47:37,230 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:47:37,249 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:47:37,267 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:47:37,285 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:47:37,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz 2022-02-08 13:47:37,319 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:47:37,339 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz | grep -v 'fatal remote' 2022-02-08 13:47:37,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:47:37,378 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:47:37,395 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:47:37,411 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:47:37,427 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:47:37,444 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:47:37,463 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:47:37,479 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz 2022-02-08 13:47:37,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:47:37,513 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz | grep -v 'fatal remote' 2022-02-08 13:47:37,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:47:37,548 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:47:37,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:47:37,584 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:47:37,601 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:47:37,618 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:47:37,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:47:37,651 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.3.gz 2022-02-08 13:47:37,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:47:37,687 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz | grep -v 'fatal remote' 2022-02-08 13:47:37,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:47:37,726 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:47:37,746 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:47:37,765 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:47:37,782 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:47:37,801 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:47:37,819 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:47:37,838 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.4.gz 2022-02-08 13:47:37,855 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:47:37,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz | grep -v 'fatal remote' 2022-02-08 13:47:37,894 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:47:37,914 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:47:37,933 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:47:37,963 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:47:37,982 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:47:38,002 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:47:38,022 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:47:38,039 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.5.gz 2022-02-08 13:47:38,059 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:47:38,076 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz | grep -v 'fatal remote' 2022-02-08 13:47:38,095 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:47:38,112 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:47:38,130 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:47:38,153 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:47:38,171 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:47:38,188 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:47:38,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:47:38,223 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.6.gz 2022-02-08 13:47:38,242 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:47:38,259 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz | grep -v 'fatal remote' 2022-02-08 13:47:38,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:47:38,296 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:47:38,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:47:38,328 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:47:38,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:47:38,362 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:47:38,381 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:47:38,400 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.7.gz 2022-02-08 13:47:38,415 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:47:38,434 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz | grep -v 'fatal remote' 2022-02-08 13:47:38,454 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:47:38,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:47:38,490 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:47:38,509 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:47:38,528 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:47:38,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:47:38,563 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:47:38,582 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.8.gz 2022-02-08 13:47:38,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:47:38,615 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz | grep -v 'fatal remote' 2022-02-08 13:47:38,634 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:47:38,653 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:47:38,684 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:47:38,707 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:47:38,727 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:47:38,750 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:47:38,775 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:47:38,799 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log.9.gz 2022-02-08 13:47:38,815 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 13:47:38,815 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:47:38,816 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 13:47:39,494 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:47:39,805 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 13:47:39,937 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:47:39,938 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 13:47:39,953 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 13:47:39,973 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 13:47:39,990 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 13:47:40,012 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 13:47:40,023 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 13:47:40,049 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 13:47:40,065 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:47:40,676 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 13:47:40,677 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.86000013351. 2022-02-08 13:47:40,677 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:47:40,678 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 13:47:41,505 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:47:41,878 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 13:47:42,046 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:47:42,048 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 13:47:42,066 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 13:47:42,096 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 13:47:42,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 13:47:42,148 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 13:47:42,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 13:47:42,206 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 13:47:42,252 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:47:42,926 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 13:47:42,927 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.25. 2022-02-08 13:47:42,927 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:47:42,930 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 13:47:43,796 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:47:44,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 13:47:44,417 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:47:44,417 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 13:47:44,438 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 13:47:44,484 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 13:47:44,517 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 13:47:44,545 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 13:47:44,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 13:47:44,605 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 13:47:44,628 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:47:45,260 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 13:47:45,260 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.33399987221. 2022-02-08 13:47:45,260 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:47:45,263 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 13:47:46,316 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:47:46,832 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 13:47:46,986 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:47:46,986 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 13:47:47,007 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 13:47:47,040 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 13:47:47,062 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 13:47:47,085 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 13:47:47,104 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 13:47:47,131 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 13:47:47,151 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:47:47,801 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 13:47:47,802 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.54100012779. 2022-02-08 13:47:47,802 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:47:47,803 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 13:47:48,631 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:47:49,118 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 13:47:49,259 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:47:49,260 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 13:47:49,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 13:47:49,302 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 13:47:49,322 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 13:47:49,345 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 13:47:49,358 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 13:47:49,405 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 13:47:49,428 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:47:50,036 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 13:47:50,036 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.2349998951. 2022-02-08 13:47:50,036 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:47:50,039 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 5 queries submitted, 0 failed, 17606 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:47:50,638 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:47:50,979 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 13:47:51,082 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:47:51,084 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 13:47:51,101 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 13:47:51,128 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 13:47:51,145 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 13:47:51,167 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 13:47:51,187 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 13:47:51,224 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 13:47:51,246 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:47:51,890 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 13:47:51,890 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 1.85300016403. 2022-02-08 13:47:51,891 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:47:51,891 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 13:47:52,762 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:47:53,279 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 13:47:53,473 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:47:53,474 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 13:47:53,494 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 13:47:53,516 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 13:47:53,530 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 13:47:53,561 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 13:47:53,575 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 13:47:53,604 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 13:47:53,621 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:47:54,164 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 13:47:54,164 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.27399992943. 2022-02-08 13:47:54,164 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:47:54,167 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 13:47:54,996 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:47:55,532 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 13:47:55,697 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:47:55,697 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 13:47:55,730 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 13:47:55,757 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 13:47:55,783 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 13:47:55,809 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 13:47:55,829 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 13:47:55,875 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 13:47:55,898 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:47:56,740 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 13:47:56,740 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 2.5759999752. 2022-02-08 13:47:56,740 | test | DEBUG | Thread-19496 | [bucket_ready_functions:_wait_warmup_completed:4189] Waiting for bucket GleamBookUsers0 to complete warm-up 2022-02-08 13:47:56,743 | infra | DEBUG | Thread-19496 | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 13:47:57,668 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 13:47:58,312 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 13:47:58,450 | infra | DEBUG | Thread-19496 | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 13:47:58,451 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 13:47:58,473 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 13:47:58,496 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 13:47:58,515 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 13:47:58,546 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 13:47:58,565 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 13:47:58,598 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 13:47:58,622 | infra | DEBUG | Thread-19496 | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2022-02-08 13:48:03,509 | infra | DEBUG | Thread-19496 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 13:48:03,509 | test | INFO | Thread-19496 | [opd:check_warmup_complete:882] Bucket:GleamBookUsers0 warm-up completed in 6.76800012589. 2022-02-08 13:48:03,512 | test | DEBUG | Thread-19496 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L836 2022-02-08 13:48:03,513 | test | DEBUG | Thread-19496 | [common_lib:sleep:24] Sleep 300 seconds. Reason: None 5 queries submitted, 0 failed, 17627 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 17649 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 17672 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 17696 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 17716 passed, 0 rejected, 0 cancelled, 0 timeout ############################################################ # # Step 14: Updating the bucket replica to 2 # ############################################################ 2022-02-08 13:53:03,538 | infra | INFO | MainThread | [BucketOperations_Rest:change_bucket_props:412] Updating bucket properties for GleamBookUsers0 2022-02-08 13:53:03,543 | infra | DEBUG | MainThread | [BucketOperations_Rest:change_bucket_props:413] http://172.23.120.170:8091/pools/default/buckets/GleamBookUsers0 with param: replicaNumber=2 2022-02-08 13:53:03,966 | infra | DEBUG | MainThread | [BucketOperations_Rest:change_bucket_props:422] Bucket GleamBookUsers0 updated KV nodes in cluster: ['172.23.121.116', '172.23.121.123', '172.23.121.124', '172.23.121.126', '172.23.121.127', '172.23.121.129', '172.23.121.135', '172.23.121.115'] CBAS nodes in cluster: [] INDEX nodes in cluster: ['172.23.121.130', '172.23.121.131', '172.23.121.132', '172.23.121.133', '172.23.121.134'] FTS nodes in cluster: [] QUERY nodes in cluster: ['172.23.121.130', '172.23.121.131', '172.23.121.132', '172.23.121.133', '172.23.121.134'] EVENTING nodes in cluster: [] AVAILABLE nodes for cluster: ['172.23.121.136', '172.23.121.128'] Servers coming in : ['172.23.121.128'] with services: ['kv'] Servers going out : [] NEW AVAILABLE nodes for cluster: ['172.23.121.136'] 2022-02-08 13:53:04,033 | infra | DEBUG | MainThread | [task:__init__:249] Last known rebalance status_id: a8a4a83090d7f7584b2b89af630d0b27 2022-02-08 13:53:04,033 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[172.23.121.128]_OUT=[]_1644357183.97 2022-02-08 13:53:04,035 | infra | INFO | pool-3-thread-17 | [task:start_task:91] Thread 'Rebalance_task_IN=[172.23.121.128]_OUT=[]_1644357183.97' started 2022-02-08 13:53:04,039 | test | DEBUG | pool-3-thread-17 | [rest_client:add_node:1199] Adding remote node 172.23.121.128:8091 to cluster 172.23.120.170:8091 2022-02-08 13:53:04,043 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/Hospital.py -> SystemTestMagma():L705 2022-02-08 13:53:04,043 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep for 60s for rebalance to start 2022-02-08 13:53:10,141 | test | DEBUG | pool-3-thread-17 | [rest_client:rebalance:1454] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.121.128,ns_1@172.23.121.115,ns_1@172.23.121.132,ns_1@172.23.121.135,ns_1@172.23.120.170,ns_1@172.23.121.129,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.134,ns_1@172.23.121.116,ns_1@172.23.121.127,ns_1@172.23.121.123,ns_1@172.23.121.131,ns_1@172.23.121.130,ns_1@172.23.121.133', 'ejectedNodes': '', 'user': 'Administrator'} 2022-02-08 13:53:11,164 | test | DEBUG | pool-3-thread-17 | [rest_client:rebalance:1459] Rebalance operation started 2022-02-08 13:53:11,171 | test | INFO | pool-3-thread-17 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.121.115 | kv | 7.1.0-2255-enterprise | 87.8537885518 | Cluster node | | 172.23.121.132 | index, n1ql | 7.1.0-2255-enterprise | 12.566671003 | Cluster node | | 172.23.121.135 | kv | 7.1.0-2255-enterprise | 85.7160995429 | Cluster node | | 172.23.120.170 | kv | 7.1.0-2255-enterprise | 86.9256156385 | Cluster node | | 172.23.121.129 | kv | 7.1.0-2255-enterprise | 86.3561365656 | Cluster node | | 172.23.121.124 | kv | 7.1.0-2255-enterprise | 86.4782276547 | Cluster node | | 172.23.121.126 | kv | 7.1.0-2255-enterprise | 85.4415881904 | Cluster node | | 172.23.121.134 | index, n1ql | 7.1.0-2255-enterprise | 23.205772452 | Cluster node | | 172.23.121.116 | kv | 7.1.0-2255-enterprise | 86.1591255719 | Cluster node | | 172.23.121.127 | kv | 7.1.0-2255-enterprise | 87.1986805379 | Cluster node | | 172.23.121.123 | kv | 7.1.0-2255-enterprise | 87.1020925808 | Cluster node | | 172.23.121.131 | index, n1ql | 7.1.0-2255-enterprise | 12.8511966701 | Cluster node | | 172.23.121.130 | index, n1ql | 7.1.0-2255-enterprise | 18.2535575679 | Cluster node | | 172.23.121.133 | index, n1ql | 7.1.0-2255-enterprise | 11.2688782755 | Cluster node | | 172.23.121.128 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2022-02-08 13:53:11,184 | infra | DEBUG | pool-3-thread-17 | [task:call:316] New rebalance status_id: cce34137c17507da2aedba113637d202 2022-02-08 13:53:11,213 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 13:53:11,213 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 13:53:11,242 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:53:11,243 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:53:11,244 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:53:21,286 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 13:53:21,286 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 13:53:21,338 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:53:21,341 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:53:21,342 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:53:31,361 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 13:53:31,361 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 13:53:31,397 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:53:31,400 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:53:31,401 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:53:41,414 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 13:53:41,415 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 13:53:41,464 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:53:41,467 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:53:41,467 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 17740 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:53:51,486 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 13:53:51,486 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 13:53:51,526 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:53:51,529 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:53:51,530 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:54:01,551 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 13:54:01,551 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 13:54:01,585 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:54:01,586 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:54:01,588 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:54:04,045 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[172.23.121.128]_OUT=[]_1644357183.97 2022-02-08 13:54:11,631 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.00 % 2022-02-08 13:54:11,631 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.0 2022-02-08 13:54:11,671 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:54:11,674 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:54:11,674 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:54:21,693 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.12 % 2022-02-08 13:54:21,694 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.12 2022-02-08 13:54:21,723 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:54:21,724 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:54:21,726 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:54:31,744 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.12 % 2022-02-08 13:54:31,746 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.12 2022-02-08 13:54:31,773 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:54:31,776 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:54:31,776 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:54:41,798 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.18 % 2022-02-08 13:54:41,798 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.18 2022-02-08 13:54:41,831 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:54:41,834 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:54:41,834 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 17767 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:54:51,852 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.28 % 2022-02-08 13:54:51,852 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.28 2022-02-08 13:54:51,887 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:54:51,888 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:54:51,890 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:55:01,908 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.28 % 2022-02-08 13:55:01,910 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.28 2022-02-08 13:55:01,950 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:55:01,953 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:55:01,953 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:55:11,969 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.28 % 2022-02-08 13:55:11,969 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.28 2022-02-08 13:55:12,003 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:55:12,005 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:55:12,005 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:55:22,022 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.40 % 2022-02-08 13:55:22,022 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.4 2022-02-08 13:55:22,052 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:55:22,053 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:55:22,055 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:55:32,072 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.47 % 2022-02-08 13:55:32,072 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.47 2022-02-08 13:55:32,105 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:55:32,108 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:55:32,108 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:55:42,125 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.53 % 2022-02-08 13:55:42,125 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.53 2022-02-08 13:55:42,153 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:55:42,154 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:55:42,154 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 17791 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:55:52,168 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.53 % 2022-02-08 13:55:52,170 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.53 2022-02-08 13:55:52,217 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:55:52,220 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:55:52,220 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:56:02,236 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.59 % 2022-02-08 13:56:02,236 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.59 2022-02-08 13:56:02,262 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:56:02,263 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:56:02,265 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:56:12,280 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.65 % 2022-02-08 13:56:12,280 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.65 2022-02-08 13:56:12,311 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:56:12,313 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:56:12,313 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:56:22,342 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.71 % 2022-02-08 13:56:22,342 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.71 2022-02-08 13:56:22,382 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:56:22,385 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:56:22,385 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:56:32,404 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.77 % 2022-02-08 13:56:32,404 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.77 2022-02-08 13:56:32,428 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:56:32,430 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:56:32,430 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:56:42,446 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 0.90 % 2022-02-08 13:56:42,446 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 0.9 2022-02-08 13:56:42,477 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:56:42,480 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:56:42,480 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 17812 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:56:52,496 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.02 % 2022-02-08 13:56:52,496 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.02 2022-02-08 13:56:52,525 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:56:52,526 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:56:52,526 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:57:02,542 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.02 % 2022-02-08 13:57:02,542 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.02 2022-02-08 13:57:02,572 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:57:02,575 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:57:02,575 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:57:12,592 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.02 % 2022-02-08 13:57:12,592 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.02 2022-02-08 13:57:12,621 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:57:12,622 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:57:12,622 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:57:22,635 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.02 % 2022-02-08 13:57:22,635 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.02 2022-02-08 13:57:22,667 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:57:22,670 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:57:22,671 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:57:32,684 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.02 % 2022-02-08 13:57:32,684 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.02 2022-02-08 13:57:32,713 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:57:32,716 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:57:32,716 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:57:42,730 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.08 % 2022-02-08 13:57:42,732 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.08 2022-02-08 13:57:42,766 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:57:42,769 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:57:42,769 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 17833 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:57:52,782 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.08 % 2022-02-08 13:57:52,782 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.08 2022-02-08 13:57:52,812 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:57:52,813 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:57:52,813 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:58:02,834 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.18 % 2022-02-08 13:58:02,834 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.18 2022-02-08 13:58:02,871 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:58:02,875 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:58:02,875 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:58:12,894 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.18 % 2022-02-08 13:58:12,894 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.18 2022-02-08 13:58:12,930 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:58:12,931 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:58:12,933 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:58:22,950 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.18 % 2022-02-08 13:58:22,950 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.18 2022-02-08 13:58:22,977 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:58:22,980 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:58:22,980 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:58:32,996 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.18 % 2022-02-08 13:58:32,996 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.18 2022-02-08 13:58:33,025 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:58:33,028 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:58:33,028 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:58:43,040 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.18 % 2022-02-08 13:58:43,040 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.18 2022-02-08 13:58:43,069 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:58:43,071 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:58:43,072 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 17859 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:58:53,085 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.18 % 2022-02-08 13:58:53,085 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.18 2022-02-08 13:58:53,115 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:58:53,117 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:58:53,117 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:59:03,135 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.24 % 2022-02-08 13:59:03,135 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.24 2022-02-08 13:59:03,164 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:59:03,167 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:59:03,167 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:59:13,181 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.43 % 2022-02-08 13:59:13,183 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.43 2022-02-08 13:59:13,217 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:59:13,219 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:59:13,220 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:59:23,239 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.49 % 2022-02-08 13:59:23,239 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.49 2022-02-08 13:59:23,270 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:59:23,273 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:59:23,273 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:59:33,292 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.61 % 2022-02-08 13:59:33,292 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.61 2022-02-08 13:59:33,331 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:59:33,334 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:59:33,334 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 13:59:43,349 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.61 % 2022-02-08 13:59:43,349 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.61 2022-02-08 13:59:43,381 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:59:43,384 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:59:43,385 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 17881 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 13:59:53,400 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.61 % 2022-02-08 13:59:53,401 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.61 2022-02-08 13:59:53,430 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 13:59:53,433 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 13:59:53,433 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:00:03,463 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.67 % 2022-02-08 14:00:03,463 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.67 2022-02-08 14:00:03,523 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:00:03,526 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:00:03,526 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:00:13,543 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.67 % 2022-02-08 14:00:13,545 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.67 2022-02-08 14:00:13,573 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:00:13,576 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:00:13,576 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:00:23,594 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.73 % 2022-02-08 14:00:23,594 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.73 2022-02-08 14:00:23,634 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:00:23,637 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:00:23,637 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:00:33,655 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.73 % 2022-02-08 14:00:33,655 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.73 2022-02-08 14:00:33,700 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:00:33,703 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:00:33,704 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:00:43,723 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.80 % 2022-02-08 14:00:43,723 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.8 2022-02-08 14:00:43,755 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:00:43,756 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:00:43,757 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 17904 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:00:53,773 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.80 % 2022-02-08 14:00:53,773 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.8 2022-02-08 14:00:53,805 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:00:53,809 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:00:53,809 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:01:03,823 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.89 % 2022-02-08 14:01:03,823 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.89 2022-02-08 14:01:03,857 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:01:03,861 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:01:03,861 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:01:13,882 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 1.99 % 2022-02-08 14:01:13,884 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 1.99 2022-02-08 14:01:13,913 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:01:13,915 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:01:13,915 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:01:23,930 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.05 % 2022-02-08 14:01:23,930 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 2.05 2022-02-08 14:01:23,969 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:01:23,971 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:01:23,971 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:01:33,986 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.05 % 2022-02-08 14:01:33,987 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 2.05 2022-02-08 14:01:34,032 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:01:34,036 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:01:34,036 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:01:44,052 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.05 % 2022-02-08 14:01:44,052 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 2.05 2022-02-08 14:01:44,091 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:01:44,092 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:01:44,094 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 17931 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:01:54,115 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.11 % 2022-02-08 14:01:54,115 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 2.11 2022-02-08 14:01:54,151 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:01:54,154 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:01:54,154 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:02:04,170 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.33 % 2022-02-08 14:02:04,171 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 2.33 2022-02-08 14:02:04,203 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:02:04,204 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:02:04,206 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:02:14,226 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.33 % 2022-02-08 14:02:14,226 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 2.33 2022-02-08 14:02:14,272 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:02:14,275 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:02:14,275 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:02:24,289 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.49 % 2022-02-08 14:02:24,289 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 2.49 2022-02-08 14:02:24,322 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:02:24,325 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:02:24,325 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:02:34,338 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.55 % 2022-02-08 14:02:34,338 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 2.55 2022-02-08 14:02:34,365 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:02:34,368 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:02:34,368 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:02:44,382 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.55 % 2022-02-08 14:02:44,384 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 2.55 2022-02-08 14:02:44,417 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:02:44,420 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:02:44,420 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 17958 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:02:54,436 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.61 % 2022-02-08 14:02:54,436 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 2.61 2022-02-08 14:02:54,467 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:02:54,470 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:02:54,470 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:03:04,486 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.67 % 2022-02-08 14:03:04,486 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 2.67 2022-02-08 14:03:04,516 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:03:04,519 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:03:04,519 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:03:14,535 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.80 % 2022-02-08 14:03:14,535 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 2.8 2022-02-08 14:03:14,569 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:03:14,575 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:03:14,575 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:03:24,595 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.80 % 2022-02-08 14:03:24,595 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 2.8 2022-02-08 14:03:24,624 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:03:24,627 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:03:24,627 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:03:34,641 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.86 % 2022-02-08 14:03:34,641 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 2.86 2022-02-08 14:03:34,673 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:03:34,677 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:03:34,677 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:03:44,697 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.86 % 2022-02-08 14:03:44,697 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 2.86 2022-02-08 14:03:44,733 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:03:44,734 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:03:44,734 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 17984 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:03:54,750 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.86 % 2022-02-08 14:03:54,750 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 2.86 2022-02-08 14:03:54,783 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:03:54,786 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:03:54,786 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:04:04,806 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 2.86 % 2022-02-08 14:04:04,808 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 2.86 2022-02-08 14:04:04,839 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:04:04,842 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:04:04,842 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:04:14,862 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.02 % 2022-02-08 14:04:14,862 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.02 2022-02-08 14:04:14,905 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:04:14,908 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:04:14,908 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:04:24,924 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.08 % 2022-02-08 14:04:24,924 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.08 2022-02-08 14:04:24,954 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:04:24,957 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:04:24,957 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:04:34,974 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.14 % 2022-02-08 14:04:34,976 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.14 2022-02-08 14:04:35,010 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:04:35,013 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:04:35,013 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:04:45,029 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.26 % 2022-02-08 14:04:45,030 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.26 2022-02-08 14:04:45,063 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:04:45,068 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:04:45,068 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18009 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:04:55,088 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.32 % 2022-02-08 14:04:55,088 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.32 2022-02-08 14:04:55,122 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:04:55,127 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:04:55,127 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:05:05,144 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.42 % 2022-02-08 14:05:05,144 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.42 2022-02-08 14:05:05,194 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:05:05,198 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:05:05,200 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:05:15,220 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.42 % 2022-02-08 14:05:15,220 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.42 2022-02-08 14:05:15,253 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:05:15,256 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:05:15,257 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:05:25,276 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.52 % 2022-02-08 14:05:25,276 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.52 2022-02-08 14:05:25,316 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:05:25,319 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:05:25,319 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:05:35,338 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.61 % 2022-02-08 14:05:35,338 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.61 2022-02-08 14:05:35,375 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:05:35,378 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:05:35,380 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:05:45,401 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.67 % 2022-02-08 14:05:45,401 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.67 2022-02-08 14:05:45,440 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:05:45,444 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:05:45,444 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18031 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:05:55,463 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.67 % 2022-02-08 14:05:55,463 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.67 2022-02-08 14:05:55,503 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:05:55,506 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:05:55,506 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:06:05,523 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.67 % 2022-02-08 14:06:05,523 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.67 2022-02-08 14:06:05,556 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:06:05,561 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:06:05,561 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:06:15,575 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.67 % 2022-02-08 14:06:15,576 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.67 2022-02-08 14:06:15,615 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:06:15,618 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:06:15,619 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:06:25,638 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.77 % 2022-02-08 14:06:25,638 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.77 2022-02-08 14:06:25,674 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:06:25,677 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:06:25,677 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:06:35,697 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.77 % 2022-02-08 14:06:35,697 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.77 2022-02-08 14:06:35,730 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:06:35,733 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:06:35,733 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:06:45,749 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.77 % 2022-02-08 14:06:45,750 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.77 2022-02-08 14:06:45,780 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:06:45,785 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:06:45,785 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18054 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:06:55,806 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.83 % 2022-02-08 14:06:55,806 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.83 2022-02-08 14:06:55,845 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:06:55,848 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:06:55,848 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:07:05,891 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.89 % 2022-02-08 14:07:05,892 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.89 2022-02-08 14:07:05,921 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:07:05,926 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:07:05,926 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:07:15,944 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 3.95 % 2022-02-08 14:07:15,946 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 3.95 2022-02-08 14:07:15,980 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:07:15,983 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:07:15,983 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:07:25,999 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.02 % 2022-02-08 14:07:25,999 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.02 2022-02-08 14:07:26,029 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:07:26,032 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:07:26,033 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:07:36,052 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.02 % 2022-02-08 14:07:36,055 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.02 2022-02-08 14:07:36,085 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:07:36,088 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:07:36,088 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:07:46,104 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.02 % 2022-02-08 14:07:46,105 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.02 2022-02-08 14:07:46,138 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:07:46,141 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:07:46,141 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18079 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:07:56,154 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.24 % 2022-02-08 14:07:56,155 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.24 2022-02-08 14:07:56,187 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:07:56,190 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:07:56,190 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:08:06,206 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.24 % 2022-02-08 14:08:06,206 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.24 2022-02-08 14:08:06,243 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:08:06,246 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:08:06,247 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:08:16,265 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.30 % 2022-02-08 14:08:16,265 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.3 2022-02-08 14:08:16,299 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:08:16,302 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:08:16,302 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:08:26,318 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.30 % 2022-02-08 14:08:26,319 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.3 2022-02-08 14:08:26,348 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:08:26,351 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:08:26,351 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:08:36,365 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.30 % 2022-02-08 14:08:36,365 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.3 2022-02-08 14:08:36,394 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:08:36,398 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:08:36,398 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:08:46,415 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.42 % 2022-02-08 14:08:46,415 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.42 2022-02-08 14:08:46,441 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:08:46,444 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:08:46,444 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18105 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:08:56,459 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.54 % 2022-02-08 14:08:56,459 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.54 2022-02-08 14:08:56,503 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:08:56,507 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:08:56,507 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:09:06,523 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.61 % 2022-02-08 14:09:06,523 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.61 2022-02-08 14:09:06,552 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:09:06,555 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:09:06,556 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:09:16,575 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.67 % 2022-02-08 14:09:16,575 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.67 2022-02-08 14:09:16,615 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:09:16,618 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:09:16,618 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:09:26,637 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.67 % 2022-02-08 14:09:26,638 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.67 2022-02-08 14:09:26,670 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:09:26,673 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:09:26,673 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:09:36,690 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.73 % 2022-02-08 14:09:36,690 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.73 2022-02-08 14:09:36,730 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:09:36,734 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:09:36,734 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:09:46,750 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.73 % 2022-02-08 14:09:46,750 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.73 2022-02-08 14:09:46,782 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:09:46,783 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:09:46,783 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18127 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:09:56,799 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.73 % 2022-02-08 14:09:56,799 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.73 2022-02-08 14:09:56,834 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:09:56,835 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:09:56,835 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:10:06,861 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.89 % 2022-02-08 14:10:06,861 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.89 2022-02-08 14:10:06,891 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:10:06,894 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:10:06,894 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:10:16,911 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.89 % 2022-02-08 14:10:16,911 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.89 2022-02-08 14:10:16,943 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:10:16,946 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:10:16,946 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:10:26,969 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.89 % 2022-02-08 14:10:26,969 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.89 2022-02-08 14:10:27,022 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:10:27,025 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:10:27,026 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:10:37,042 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 4.95 % 2022-02-08 14:10:37,042 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 4.95 2022-02-08 14:10:37,078 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:10:37,081 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:10:37,081 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:10:47,128 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.01 % 2022-02-08 14:10:47,128 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.01 2022-02-08 14:10:47,173 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:10:47,177 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:10:47,177 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18147 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:10:57,193 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.01 % 2022-02-08 14:10:57,193 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.01 2022-02-08 14:10:57,226 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:10:57,227 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:10:57,227 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:11:07,243 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.01 % 2022-02-08 14:11:07,243 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.01 2022-02-08 14:11:07,278 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:11:07,280 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:11:07,280 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:11:17,298 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.07 % 2022-02-08 14:11:17,298 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.07 2022-02-08 14:11:17,361 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:11:17,364 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:11:17,364 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:11:27,378 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.07 % 2022-02-08 14:11:27,378 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.07 2022-02-08 14:11:27,407 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:11:27,410 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:11:27,410 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:11:37,423 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.07 % 2022-02-08 14:11:37,423 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.07 2022-02-08 14:11:37,454 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:11:37,459 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:11:37,459 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:11:47,473 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.07 % 2022-02-08 14:11:47,473 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.07 2022-02-08 14:11:47,510 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:11:47,513 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:11:47,513 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18172 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:11:57,526 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.20 % 2022-02-08 14:11:57,528 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.2 2022-02-08 14:11:57,569 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:11:57,572 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:11:57,572 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:12:07,585 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.26 % 2022-02-08 14:12:07,585 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.26 2022-02-08 14:12:07,621 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:12:07,624 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:12:07,624 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:12:17,638 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.35 % 2022-02-08 14:12:17,638 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.35 2022-02-08 14:12:17,670 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:12:17,673 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:12:17,673 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:12:27,690 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.35 % 2022-02-08 14:12:27,690 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.35 2022-02-08 14:12:27,724 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:12:27,727 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:12:27,727 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:12:37,752 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.35 % 2022-02-08 14:12:37,753 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.35 2022-02-08 14:12:37,785 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:12:37,788 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:12:37,788 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:12:47,809 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.48 % 2022-02-08 14:12:47,809 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.48 2022-02-08 14:12:47,846 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:12:47,848 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:12:47,849 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18196 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:12:57,888 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.60 % 2022-02-08 14:12:57,888 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.6 2022-02-08 14:12:57,924 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:12:57,927 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:12:57,927 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:13:07,941 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.66 % 2022-02-08 14:13:07,943 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.66 2022-02-08 14:13:07,970 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:13:07,973 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:13:07,973 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:13:17,986 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.66 % 2022-02-08 14:13:17,986 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.66 2022-02-08 14:13:18,016 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:13:18,017 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:13:18,019 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:13:28,035 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.66 % 2022-02-08 14:13:28,035 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.66 2022-02-08 14:13:28,066 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:13:28,069 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:13:28,069 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:13:38,086 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.66 % 2022-02-08 14:13:38,086 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.66 2022-02-08 14:13:38,117 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:13:38,118 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:13:38,118 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:13:48,138 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.66 % 2022-02-08 14:13:48,138 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.66 2022-02-08 14:13:48,168 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:13:48,171 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:13:48,171 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18218 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:13:58,184 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.76 % 2022-02-08 14:13:58,184 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.76 2022-02-08 14:13:58,216 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:13:58,219 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:13:58,219 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:14:08,233 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.76 % 2022-02-08 14:14:08,233 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.76 2022-02-08 14:14:08,269 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:14:08,273 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:14:08,273 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:14:18,289 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.82 % 2022-02-08 14:14:18,289 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.82 2022-02-08 14:14:18,319 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:14:18,322 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:14:18,322 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:14:28,341 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.88 % 2022-02-08 14:14:28,342 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.88 2022-02-08 14:14:28,372 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:14:28,375 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:14:28,375 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:14:38,391 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.88 % 2022-02-08 14:14:38,391 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.88 2022-02-08 14:14:38,424 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:14:38,427 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:14:38,427 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:14:48,444 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.88 % 2022-02-08 14:14:48,446 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.88 2022-02-08 14:14:48,476 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:14:48,479 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:14:48,479 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18245 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:14:58,496 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 5.94 % 2022-02-08 14:14:58,496 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 5.94 2022-02-08 14:14:58,529 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:14:58,532 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:14:58,532 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:15:08,549 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.00 % 2022-02-08 14:15:08,549 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.0 2022-02-08 14:15:08,578 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:15:08,581 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:15:08,582 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:15:18,598 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.00 % 2022-02-08 14:15:18,598 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.0 2022-02-08 14:15:18,628 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:15:18,631 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:15:18,631 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:15:28,654 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.00 % 2022-02-08 14:15:28,654 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.0 2022-02-08 14:15:28,684 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:15:28,687 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:15:28,687 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:15:38,703 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.06 % 2022-02-08 14:15:38,703 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.06 2022-02-08 14:15:38,734 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:15:38,736 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:15:38,736 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:15:48,753 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.16 % 2022-02-08 14:15:48,753 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.16 2022-02-08 14:15:48,785 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:15:48,788 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:15:48,788 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18268 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:15:58,803 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.16 % 2022-02-08 14:15:58,803 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.16 2022-02-08 14:15:58,838 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:15:58,841 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:15:58,842 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:16:08,857 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.16 % 2022-02-08 14:16:08,858 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.16 2022-02-08 14:16:08,900 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:16:08,904 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:16:08,904 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:16:18,920 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.16 % 2022-02-08 14:16:18,920 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.16 2022-02-08 14:16:18,956 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:16:18,957 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:16:18,957 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:16:28,976 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.28 % 2022-02-08 14:16:28,976 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.28 2022-02-08 14:16:29,016 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:16:29,019 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:16:29,019 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:16:39,033 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.28 % 2022-02-08 14:16:39,033 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.28 2022-02-08 14:16:39,069 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:16:39,072 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:16:39,072 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:16:49,088 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.41 % 2022-02-08 14:16:49,088 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.41 2022-02-08 14:16:49,122 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:16:49,127 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:16:49,127 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18290 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:16:59,151 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.41 % 2022-02-08 14:16:59,151 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.41 2022-02-08 14:16:59,183 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:16:59,184 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:16:59,184 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:17:09,201 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.53 % 2022-02-08 14:17:09,201 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.53 2022-02-08 14:17:09,232 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:17:09,233 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:17:09,233 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:17:19,250 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.59 % 2022-02-08 14:17:19,252 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.59 2022-02-08 14:17:19,302 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:17:19,305 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:17:19,305 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:17:29,322 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.59 % 2022-02-08 14:17:29,323 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.59 2022-02-08 14:17:29,349 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:17:29,351 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:17:29,351 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:17:39,368 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.65 % 2022-02-08 14:17:39,368 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.65 2022-02-08 14:17:39,400 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:17:39,403 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:17:39,403 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:17:49,417 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.65 % 2022-02-08 14:17:49,417 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.65 2022-02-08 14:17:49,444 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:17:49,447 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:17:49,447 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18315 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:17:59,460 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.65 % 2022-02-08 14:17:59,460 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.65 2022-02-08 14:17:59,490 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:17:59,492 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:17:59,493 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:18:09,506 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.81 % 2022-02-08 14:18:09,506 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.81 2022-02-08 14:18:09,539 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:18:09,542 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:18:09,543 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:18:19,559 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.81 % 2022-02-08 14:18:19,559 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.81 2022-02-08 14:18:19,589 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:18:19,594 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:18:19,594 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:18:29,608 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.81 % 2022-02-08 14:18:29,609 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.81 2022-02-08 14:18:29,638 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:18:29,641 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:18:29,641 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:18:39,657 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 6.94 % 2022-02-08 14:18:39,657 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 6.94 2022-02-08 14:18:39,694 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:18:39,697 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:18:39,697 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:18:49,716 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.09 % 2022-02-08 14:18:49,716 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.09 2022-02-08 14:18:49,749 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:18:49,752 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:18:49,752 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18339 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:18:59,795 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.15 % 2022-02-08 14:18:59,796 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.15 2022-02-08 14:18:59,855 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:18:59,859 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:18:59,859 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:19:09,891 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.28 % 2022-02-08 14:19:09,892 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.28 2022-02-08 14:19:09,927 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:19:09,931 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:19:09,933 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:19:19,987 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.28 % 2022-02-08 14:19:19,989 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.28 2022-02-08 14:19:20,059 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:19:20,068 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:19:20,069 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:19:30,099 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.34 % 2022-02-08 14:19:30,101 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.34 2022-02-08 14:19:30,151 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:19:30,157 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:19:30,158 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:19:40,206 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.34 % 2022-02-08 14:19:40,207 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.34 2022-02-08 14:19:40,279 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:19:40,286 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:19:40,286 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18364 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:19:50,305 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.34 % 2022-02-08 14:19:50,306 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.34 2022-02-08 14:19:50,358 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:19:50,365 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:19:50,365 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:20:00,404 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.34 % 2022-02-08 14:20:00,405 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.34 2022-02-08 14:20:00,512 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:20:00,525 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:20:00,526 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:20:10,552 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.34 % 2022-02-08 14:20:10,555 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.34 2022-02-08 14:20:10,607 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:20:10,612 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:20:10,614 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:20:20,664 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.40 % 2022-02-08 14:20:20,671 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.4 2022-02-08 14:20:20,730 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:20:20,736 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:20:20,736 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:20:30,779 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.40 % 2022-02-08 14:20:30,780 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.4 2022-02-08 14:20:30,836 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:20:30,846 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:20:30,848 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:20:40,875 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.40 % 2022-02-08 14:20:40,875 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.4 2022-02-08 14:20:40,923 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:20:40,930 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:20:40,930 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18389 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:20:50,966 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.52 % 2022-02-08 14:20:50,967 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.52 2022-02-08 14:20:51,039 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:20:51,046 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:20:51,046 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:21:01,094 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.59 % 2022-02-08 14:21:01,095 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.59 2022-02-08 14:21:01,161 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:21:01,170 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:21:01,171 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:21:11,217 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.59 % 2022-02-08 14:21:11,219 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.59 2022-02-08 14:21:11,285 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:21:11,289 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:21:11,289 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:21:21,325 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.59 % 2022-02-08 14:21:21,326 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.59 2022-02-08 14:21:21,384 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:21:21,397 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:21:21,398 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:21:31,431 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.71 % 2022-02-08 14:21:31,433 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.71 2022-02-08 14:21:31,480 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:21:31,487 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:21:31,489 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:21:41,529 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.83 % 2022-02-08 14:21:41,532 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.83 2022-02-08 14:21:41,628 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:21:41,654 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:21:41,657 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18412 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:21:51,711 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.83 % 2022-02-08 14:21:51,713 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.83 2022-02-08 14:21:51,776 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:21:51,780 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:21:51,782 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:22:01,825 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.83 % 2022-02-08 14:22:01,828 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.83 2022-02-08 14:22:01,903 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:22:01,923 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:22:01,924 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:22:11,954 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.93 % 2022-02-08 14:22:11,957 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.93 2022-02-08 14:22:12,006 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:22:12,013 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:22:12,015 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:22:22,062 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 7.93 % 2022-02-08 14:22:22,063 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 7.93 2022-02-08 14:22:22,137 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:22:22,147 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:22:22,148 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:22:32,187 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.05 % 2022-02-08 14:22:32,190 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.05 2022-02-08 14:22:32,252 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:22:32,260 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:22:32,262 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:22:42,309 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.05 % 2022-02-08 14:22:42,311 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.05 2022-02-08 14:22:42,461 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:22:42,469 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:22:42,470 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18433 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:22:52,506 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.05 % 2022-02-08 14:22:52,509 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.05 2022-02-08 14:22:52,572 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:22:52,575 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:22:52,576 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:23:02,604 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.11 % 2022-02-08 14:23:02,605 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.11 2022-02-08 14:23:02,661 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:23:02,674 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:23:02,676 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:23:12,696 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.18 % 2022-02-08 14:23:12,697 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.18 2022-02-08 14:23:12,727 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:23:12,730 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:23:12,730 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:23:22,760 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.24 % 2022-02-08 14:23:22,762 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.24 2022-02-08 14:23:22,822 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:23:22,836 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:23:22,838 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:23:32,892 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.24 % 2022-02-08 14:23:32,904 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.24 2022-02-08 14:23:32,996 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:23:32,999 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:23:33,000 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:23:43,076 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.24 % 2022-02-08 14:23:43,078 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.24 2022-02-08 14:23:43,158 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:23:43,161 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:23:43,163 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18453 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:23:53,213 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.24 % 2022-02-08 14:23:53,216 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.24 2022-02-08 14:23:53,282 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:23:53,289 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:23:53,290 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:24:03,357 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.24 % 2022-02-08 14:24:03,368 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.24 2022-02-08 14:24:03,460 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:24:03,471 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:24:03,473 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:24:13,509 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.30 % 2022-02-08 14:24:13,510 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.3 2022-02-08 14:24:13,578 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:24:13,585 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:24:13,585 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:24:23,611 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.30 % 2022-02-08 14:24:23,612 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.3 2022-02-08 14:24:23,683 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:24:23,686 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:24:23,686 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:24:33,742 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.30 % 2022-02-08 14:24:33,743 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.3 2022-02-08 14:24:33,845 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:24:33,861 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:24:33,862 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:24:43,897 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.39 % 2022-02-08 14:24:43,898 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.39 2022-02-08 14:24:43,943 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:24:43,953 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:24:43,954 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18480 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:24:53,993 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.64 % 2022-02-08 14:24:53,994 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.64 2022-02-08 14:24:54,061 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:24:54,068 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:24:54,069 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:25:04,117 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.64 % 2022-02-08 14:25:04,118 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.64 2022-02-08 14:25:04,233 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:25:04,239 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:25:04,239 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:25:14,262 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.64 % 2022-02-08 14:25:14,263 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.64 2022-02-08 14:25:14,315 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:25:14,325 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:25:14,325 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:25:24,355 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.64 % 2022-02-08 14:25:24,357 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.64 2022-02-08 14:25:24,403 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:25:24,413 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:25:24,414 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:25:34,447 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.64 % 2022-02-08 14:25:34,450 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.64 2022-02-08 14:25:34,503 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:25:34,510 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:25:34,512 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:25:44,553 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.89 % 2022-02-08 14:25:44,555 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.89 2022-02-08 14:25:44,611 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:25:44,619 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:25:44,619 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18503 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:25:54,641 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 8.89 % 2022-02-08 14:25:54,641 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 8.89 2022-02-08 14:25:54,687 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:25:54,697 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:25:54,700 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:26:04,740 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.01 % 2022-02-08 14:26:04,740 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.01 2022-02-08 14:26:04,793 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:26:04,802 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:26:04,805 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:26:14,864 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.14 % 2022-02-08 14:26:14,864 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.14 2022-02-08 14:26:14,920 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:26:14,938 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:26:14,940 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:26:24,974 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.14 % 2022-02-08 14:26:24,976 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.14 2022-02-08 14:26:25,039 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:26:25,055 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:26:25,059 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:26:35,104 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.20 % 2022-02-08 14:26:35,105 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.2 2022-02-08 14:26:35,157 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:26:35,164 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:26:35,164 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:26:45,194 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.20 % 2022-02-08 14:26:45,196 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.2 2022-02-08 14:26:45,256 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:26:45,262 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:26:45,263 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18526 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:26:55,299 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.20 % 2022-02-08 14:26:55,301 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.2 2022-02-08 14:26:55,371 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:26:55,384 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:26:55,387 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:27:05,411 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.20 % 2022-02-08 14:27:05,411 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.2 2022-02-08 14:27:05,470 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:27:05,476 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:27:05,477 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:27:15,499 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.20 % 2022-02-08 14:27:15,500 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.2 2022-02-08 14:27:15,542 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:27:15,549 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:27:15,552 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:27:25,575 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.26 % 2022-02-08 14:27:25,575 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.26 2022-02-08 14:27:25,621 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:27:25,628 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:27:25,628 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:27:35,667 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.26 % 2022-02-08 14:27:35,668 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.26 2022-02-08 14:27:35,730 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:27:35,734 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:27:35,736 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:27:45,802 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.32 % 2022-02-08 14:27:45,803 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.32 2022-02-08 14:27:45,857 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:27:45,864 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:27:45,865 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18552 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:27:55,897 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.32 % 2022-02-08 14:27:55,898 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.32 2022-02-08 14:27:55,971 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:27:55,996 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:27:55,996 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:28:06,025 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.38 % 2022-02-08 14:28:06,026 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.38 2022-02-08 14:28:06,085 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:28:06,101 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:28:06,105 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:28:16,144 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.38 % 2022-02-08 14:28:16,147 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.38 2022-02-08 14:28:16,197 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:28:16,203 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:28:16,204 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:28:26,226 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.38 % 2022-02-08 14:28:26,226 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.38 2022-02-08 14:28:26,272 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:28:26,278 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:28:26,279 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:28:36,315 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.38 % 2022-02-08 14:28:36,318 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.38 2022-02-08 14:28:36,364 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:28:36,371 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:28:36,371 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:28:46,447 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.51 % 2022-02-08 14:28:46,448 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.51 2022-02-08 14:28:46,539 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:28:46,546 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:28:46,548 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18573 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:28:56,588 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.51 % 2022-02-08 14:28:56,589 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.51 2022-02-08 14:28:56,634 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:28:56,641 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:28:56,644 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:29:06,683 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.57 % 2022-02-08 14:29:06,684 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.57 2022-02-08 14:29:06,742 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:29:06,744 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:29:06,746 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:29:16,776 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.66 % 2022-02-08 14:29:16,776 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.66 2022-02-08 14:29:16,826 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:29:16,834 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:29:16,835 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:29:26,861 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.72 % 2022-02-08 14:29:26,861 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.72 2022-02-08 14:29:26,924 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:29:26,930 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:29:26,933 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:29:36,960 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.85 % 2022-02-08 14:29:36,960 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.85 2022-02-08 14:29:37,009 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:29:37,015 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:29:37,016 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:29:47,055 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.85 % 2022-02-08 14:29:47,055 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.85 2022-02-08 14:29:47,131 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:29:47,134 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:29:47,135 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18596 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:29:57,193 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.85 % 2022-02-08 14:29:57,194 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.85 2022-02-08 14:29:57,249 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:29:57,255 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:29:57,256 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:30:07,286 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.85 % 2022-02-08 14:30:07,286 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.85 2022-02-08 14:30:07,325 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:30:07,334 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:30:07,335 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:30:17,367 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.85 % 2022-02-08 14:30:17,368 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.85 2022-02-08 14:30:17,401 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:30:17,407 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:30:17,408 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:30:27,450 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 9.97 % 2022-02-08 14:30:27,450 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 9.97 2022-02-08 14:30:27,520 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:30:27,529 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:30:27,530 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:30:37,556 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.03 % 2022-02-08 14:30:37,559 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 10.03 2022-02-08 14:30:37,608 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:30:37,615 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:30:37,615 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:30:47,647 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.10 % 2022-02-08 14:30:47,648 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 10.1 2022-02-08 14:30:47,700 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:30:47,707 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:30:47,707 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18620 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:30:57,759 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.10 % 2022-02-08 14:30:57,762 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 10.1 2022-02-08 14:30:57,914 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:30:57,934 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:30:57,937 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:31:08,023 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.10 % 2022-02-08 14:31:08,025 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 10.1 2022-02-08 14:31:08,095 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:31:08,101 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:31:08,101 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:31:18,164 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.10 % 2022-02-08 14:31:18,165 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 10.1 2022-02-08 14:31:18,273 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:31:18,290 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:31:18,292 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:31:28,369 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.19 % 2022-02-08 14:31:28,371 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 10.19 2022-02-08 14:31:28,434 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:31:28,444 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:31:28,444 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:31:38,552 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.19 % 2022-02-08 14:31:38,553 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 10.19 2022-02-08 14:31:38,736 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:31:38,759 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:31:38,760 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:31:48,842 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.25 % 2022-02-08 14:31:48,848 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 10.25 2022-02-08 14:31:48,973 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:31:48,986 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:31:48,986 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18642 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:31:59,141 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.31 % 2022-02-08 14:31:59,148 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 10.31 2022-02-08 14:31:59,237 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:31:59,250 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:31:59,253 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:32:09,328 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.31 % 2022-02-08 14:32:09,328 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 10.31 2022-02-08 14:32:09,372 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:32:09,392 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:32:09,394 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:32:19,525 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.31 % 2022-02-08 14:32:19,529 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 10.31 2022-02-08 14:32:19,676 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:32:19,769 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:32:19,789 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:32:29,880 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.38 % 2022-02-08 14:32:29,901 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 10.38 2022-02-08 14:32:30,069 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:32:30,992 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:32:30,994 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:32:41,206 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.47 % 2022-02-08 14:32:41,209 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 10.47 2022-02-08 14:32:41,326 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:32:41,457 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:32:41,457 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 5 queries submitted, 0 failed, 18663 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:32:52,844 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.47 % 2022-02-08 14:32:52,867 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 10.47 2022-02-08 14:32:53,020 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:32:53,440 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:32:53,441 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:33:03,500 | test | DEBUG | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1682] Rebalance percentage: 10.47 % 2022-02-08 14:33:03,503 | test | INFO | pool-3-thread-17 | [task:check:414] Rebalance - status: running, progress: 10.47 2022-02-08 14:33:03,638 | infra | DEBUG | pool-3-thread-17 | [task:check:441] Wait before next rebalance progress check 2022-02-08 14:33:03,650 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L442 2022-02-08 14:33:03,651 | infra | DEBUG | pool-3-thread-17 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2022-02-08 14:33:13,871 | test | ERROR | pool-3-thread-17 | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'0a77f2dd3724bebc7a78e3292f578067', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=83f0a7cc64006783d9bbc3b0f2d55d4b', u'status': u'notRunning'} - rebalance failed 2022-02-08 14:33:13,993 | test | INFO | pool-3-thread-17 | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.120.170: 2022-02-08 14:33:13,999 | test | ERROR | pool-3-thread-17 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644359591097L, u'shortText': u'message', u'serverTime': u'2022-02-08T14:33:11.097Z', u'text': u"Service 'memcached' exited with status 134. Restarting. Messages:\n #7 /opt/couchbase/bin/memcached() [0x400000+0x74184b]\n #8 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8140001000+0xa48f6]\n #9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8140001000+0xa4961]\n #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8140001000+0xa4bf4]\n #11 /opt/couchbase/bin/memcached() [0x400000+0xcd933]\n #12 /opt/couchbase/bin/memcached() [0x400000+0x42b48f]\n #13 /opt/couchbase/bin/memcached() [0x400000+0x2de456]\n #14 /opt/couchbase/bin/memcached() [0x400000+0x2dee10]\n #15 /opt/couchbase/bin/memcached() [0x400000+0x6b48c2]\n #16 /opt/couchbase/bin/memcached() [0x400000+0x6b19c5]\n #17 /opt/couchbase/bin/memcached() [0x400000+0x808e00]\n #18 /opt/couchbase/bin/memcached() [0x400000+0x7f0bba]\n #19 /opt/couchbase/bin/memcached() [0x400000+0x80bdb9]\n #20 /opt/couchbase/bin/memcached() [0x400000+0x6b1654]\n #21 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8140001000+0xcdd40]\n #22 /lib64/libpthread.so.0() [0x7f8141ecf000+0x7ea5]\n #23 /lib64/libc.so.6(clone+0x6d) [0x7f813f719000+0xfe8dd]"} 2022-02-08 14:33:14,003 | test | ERROR | pool-3-thread-17 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644359591058L, u'shortText': u'message', u'serverTime': u'2022-02-08T14:33:11.058Z', u'text': u"Control connection to memcached on 'ns_1@172.23.120.170' disconnected. Check logs for details."} 2022-02-08 14:33:14,007 | test | ERROR | pool-3-thread-17 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.121.123', u'tstamp': 1644359587249L, u'shortText': u'message', u'serverTime': u'2022-02-08T14:33:07.249Z', u'text': u"Service 'memcached' exited with status 134. Restarting. Messages:\n #8 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe6243f3000+0xa48f6]\n #9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe6243f3000+0xa4961]\n #10 /opt/couchbase/bin/../lib/libstdc++.so.6(__cxa_rethrow+0x46) [0x7fe6243f3000+0xa4c46]\n #11 /opt/couchbase/bin/memcached() [0x400000+0xc7683]\n #12 /opt/couchbase/bin/memcached() [0x400000+0x3ffdec]\n #13 /opt/couchbase/bin/memcached() [0x400000+0x4065e3]\n #14 /opt/couchbase/bin/memcached() [0x400000+0x328c04]\n #15 /opt/couchbase/bin/memcached() [0x400000+0x449ad8]\n #16 /opt/couchbase/bin/memcached() [0x400000+0x6b48c2]\n #17 /opt/couchbase/bin/memcached() [0x400000+0x6b19c5]\n #18 /opt/couchbase/bin/memcached() [0x400000+0x808e00]\n #19 /opt/couchbase/bin/memcached() [0x400000+0x7f0bba]\n #20 /opt/couchbase/bin/memcached() [0x400000+0x80bdb9]\n #21 /opt/couchbase/bin/memcached() [0x400000+0x6b1654]\n #22 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe6243f3000+0xcdd40]\n #23 /lib64/libpthread.so.0() [0x7fe6262c1000+0x7ea5]\n #24 /lib64/libc.so.6(clone+0x6d) [0x7fe623b0b000+0xfe8dd]"} 2022-02-08 14:33:14,010 | test | ERROR | pool-3-thread-17 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.123', u'tstamp': 1644359587206L, u'shortText': u'message', u'serverTime': u'2022-02-08T14:33:07.206Z', u'text': u"Control connection to memcached on 'ns_1@172.23.121.123' disconnected. Check logs for details."} 2022-02-08 14:33:14,013 | test | ERROR | pool-3-thread-17 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644359585192L, u'shortText': u'message', u'serverTime': u'2022-02-08T14:33:05.192Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.5749.203>,\n {{wait_seqno_persisted_failed,\n "GleamBookUsers0",222,835796,\n [{\'ns_1@172.23.120.170\',\n {\'EXIT\',\n {socket_closed,\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.120.170\'},\n {if_rebalance,<0.27034.195>,\n {wait_seqno_persisted,222,835796}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,474}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,211}]}]}}}}.\nRebalance Operation Id = e7d3904b45e6057d0048f0bc643a72d7'} 2022-02-08 14:33:14,016 | test | ERROR | pool-3-thread-17 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644359585157L, u'shortText': u'message', u'serverTime': u'2022-02-08T14:33:05.157Z', u'text': u'Worker <0.3738.203> (for action {move,{222,\n [\'ns_1@172.23.121.115\',\n \'ns_1@172.23.121.129\'],\n [\'ns_1@172.23.121.129\',\n \'ns_1@172.23.120.170\',\n \'ns_1@172.23.121.115\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.5749.203>,\n {{wait_seqno_persisted_failed,\n "GleamBookUsers0",\n 222,\n 835796,\n [{\'ns_1@172.23.120.170\',\n {\'EXIT\',\n {socket_closed,\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.120.170\'},\n {if_rebalance,\n <0.27034.195>,\n {wait_seqno_persisted,\n 222,\n 835796}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 474}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 211}]}]}}}'} 2022-02-08 14:33:14,019 | test | ERROR | pool-3-thread-17 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644357198801L, u'shortText': u'message', u'serverTime': u'2022-02-08T13:53:18.801Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2022-02-08 14:33:14,022 | test | ERROR | pool-3-thread-17 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.128', u'tstamp': 1644357196913L, u'shortText': u'message', u'serverTime': u'2022-02-08T13:53:16.913Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.121.128\' in 0 seconds.'} 2022-02-08 14:33:14,023 | test | ERROR | pool-3-thread-17 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1644357196671L, u'shortText': u'message', u'serverTime': u'2022-02-08T13:53:16.671Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2022-02-08 14:33:14,023 | test | ERROR | pool-3-thread-17 | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_storage_conf', u'type': u'info', u'node': u'ns_1@172.23.121.128', u'tstamp': 1644357191566L, u'shortText': u'message', u'serverTime': u'2022-02-08T13:53:11.566Z', u'text': u'Deleting old data files of bucket "GleamBookUsers0"'} 2022-02-08 14:33:14,112 | test | ERROR | pool-3-thread-17 | [task:call:325] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'0a77f2dd3724bebc7a78e3292f578067', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=83f0a7cc64006783d9bbc3b0f2d55d4b', u'status': u'notRunning'} - rebalance failed 2022-02-08 14:33:15,911 | infra | INFO | MainThread | [task_manager:shutdown:64] Running TaskManager shutdown FAIL 2022-02-08 14:33:17,029 | infra | INFO | MainThread | [task_manager:shutdown:64] Running TaskManager shutdown 2022-02-08 14:33:17,312 | infra | WARNING | MainThread | [cluster:shutdown:707] Cluster instance shutdown with force 2022-02-08 14:33:17,414 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2022-02-08 14:33:18,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:33:18,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 2022-02-08 14:33:19,219 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:33:19,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 14:33:19,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 14:33:19,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 14:33:19,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 14:33:19,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 14:33:19,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 14:33:19,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d 2022-02-08 14:33:19,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls -lt /opt/couchbase/var/lib/couchbase/crash/ ############################## 172.23.120.170: 1 core dump seen 2022-02-08 14:33:19,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: rm -rf /opt/couchbase/var/lib/couchbase/crash/8cc04116-872e-46a8-158948a9-186223b3.core running: //opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/8cc04116-872e-46a8-158948a9-186223b3.dmp > /opt/couchbase/var/lib/couchbase/crash/8cc04116-872e-46a8-158948a9-186223b3.core 2022-02-08 14:33:19,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: //opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/8cc04116-872e-46a8-158948a9-186223b3.dmp > /opt/couchbase/var/lib/couchbase/crash/8cc04116-872e-46a8-158948a9-186223b3.core running: gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/8cc04116-872e-46a8-158948a9-186223b3.core -ex "bt full" -ex quit 2022-02-08 14:33:19,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/8cc04116-872e-46a8-158948a9-186223b3.core -ex "bt full" -ex quit 172.23.120.170: Stack Trace of first crash - 8cc04116-872e-46a8-158948a9-186223b3.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007f813f74f387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55 55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); #0 0x00007f813f74f387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55 resultvar = 0 pid = 62455 selftid = 62504 #1 0x00007f813f750a78 in __GI_abort () at abort.c:90 save_stage = 2 act = { __sigaction_handler = { sa_handler = 0x7f813fae11c0 <_IO_2_1_stderr_>, sa_sigaction = 0x7f813fae11c0 <_IO_2_1_stderr_> }, sa_mask = { __val = {140193092897321, 0, 140193092416099, 140189645883160, 140193095881152, 1, 140193095881283, 140193095865216, 140193092422270, 140193095881152, 10, 140191490640224, 0, 140191876426432, 140193092423539, 140193095881152} }, sa_flags = 1074416656, sa_restorer = 0x108f110 } sigs = { __val = {32, 0 } } #2 0x00007f814009a63c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95 terminating = false t = #3 0x0000000000b4184b in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88 No locals. #4 0x00007f81400a58f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48 No locals. #5 0x00007f81400a5961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58 No locals. #6 0x00007f81400a5bf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7f80e0000940, tinfo=0x108f6b0 , dest=0x445880 <_ZNSt11logic_errorD1Ev@plt>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95 globals = header = 0x7f80e00008c0 #7 0x00000000004cd933 in EPBucket::flushVBucket_UNLOCKED (this=0x7f81302c0000, vb=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/char_traits.h:357 vbid = { vbid = 778 } toFlush = { items = { >, std::allocator > > >> = { _M_impl = { > >> = { <__gnu_cxx::new_allocator > >> = {}, }, >, std::allocator > > >::_Vector_impl_data> = { _M_start = 0x7f81184b5760, _M_finish = 0x7f81184b5780, _M_end_of_storage = 0x7f81184b5780 }, } }, }, ranges = { >> = { _M_impl = { > = { <__gnu_cxx::new_allocator> = {}, }, >::_Vector_impl_data> = { _M_start = 0x7f807c169ba0, _M_finish = 0x7f807c169c00, _M_end_of_storage = 0x7f807c169c00 }, } }, }, moreAvailable = false, maxDeletedRevSeqno = { > = { >> = {}, members of std::_Optional_base: _M_payload = { > = { _M_payload = { _M_empty = {}, _M_value = 3 }, _M_engaged = true }, } }, >> = {}, }, checkpointType = Memory, flushHandle = { _M_t = { >> = { _M_t = { >> = { >> = { , true>> = { > = {}, }, }, > = { _M_head_impl = 0x7f807df98240 }, }, } }, } } } moreAvailable = wakeupCheckpointRemover = range = rwUnderlying = 0x7f80b474dc00 ctx = { _M_t = { >> = { _M_t = { >> = { >> = { , true>> = { > = {}, }, }, > = { _M_head_impl = 0x7f7f95b98dc0 }, }, } }, } } mustDedupe = prev = persistedVbState = vbstate = { static CurrentVersion = 4, maxDeletedSeqno = { counter = { _M_elems = "\003\000\000\000\000" } }, highSeqno = 826228, purgeSeqno = 0, lastSnapStart = 826228, lastSnapEnd = 826228, maxCas = 1644359583888048128, hlcCasEpochSeqno = 1, mightContainXattrs = false, supportsNamespaces = true, version = 4, persistedCompletedSeqno = 826227, persistedPreparedSeqno = 826227, highPreparedSeqno = 826227, maxVisibleSeqno = 826228, onDiskPrepares = 0, onDiskPrepareBytes = 0, checkpointType = Memory, transition = { failovers = { static npos = 18446744073709551615, _M_dataplus = { > = { <__gnu_cxx::new_allocator> = {}, }, members of std::__cxx11::basic_string, std::allocator >::_Alloc_hider: _M_p = 0x7f801ef30300
}, _M_string_length = 892, { _M_local_buf = "|\003\000\000\000\000\000\000`\000\000\000\000\000\000", _M_allocated_capacity = 892 } }, replicationTopology = { m_type = nlohmann::detail::null, m_value = { object = 0x0, array = 0x0, string = 0x0, binary = 0x0, boolean = false, number_integer = 0, number_unsigned = 0, number_float = 0 } }, state = vbucket_state_replica } } logged = false callback = { __this = 0x7f81302c0000, __logged = @0x7f80f6fead65, __vbid = { vbid = } } writeOp = commitData = { collections = { collections = { _M_h = { , std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Hashtable_traits >> = { , std::__detail::_Select1st, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, true>> = { > = { = {}, }, , true>> = { > = {}, }, > = { = {}, }, }, , true>> = { > = { > = {}, }, }, }, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, true>> = {}, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, false>> = { , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >> = {}, }, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, std::integral_constant >> = {}, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, true>> = {}, , true> > >> = { , true> >, true>> = { , true> >> = { <__gnu_cxx::new_allocator, true> >> = {}, }, }, }, members of std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >: _M_buckets = 0x7f80f6feb0a0, _M_bucket_count = 1, _M_before_begin = { _M_nxt = 0x0 }, _M_element_count = 0, _M_rehash_policy = { static _S_growth_factor = 2, _M_max_load_factor = 1, _M_next_resize = 0 }, _M_single_bucket = 0x0 } }, scopes = { _M_h = { , std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Hashtable_traits >> = { , std::__detail::_Select1st, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, true>> = { > = { = {}, }, , true>> = { > = {}, }, > = { = {}, }, }, , true>> = { > = { > = {}, }, }, }, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, true>> = {}, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, false>> = { , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >> = {}, }, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, std::integral_constant >> = {}, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, true>> = {}, , true> > >> = { , true> >, true>> = { , true> >> = { <__gnu_cxx::new_allocator, true> >> = {}, }, }, }, members of std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >: _M_buckets = 0x7f80f6feb0d8, _M_bucket_count = 1, _M_before_begin = { _M_nxt = 0x0 }, _M_element_count = 0, _M_rehash_policy = { static _S_growth_factor = 2, _M_max_load_factor = 1, _M_next_resize = 0 }, _M_single_bucket = 0x0 } }, droppedScopes = { _M_h = { , std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Hashtable_traits >> = { , std::__detail::_Select1st, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, true>> = { > = { = {}, }, , true>> = { > = {}, }, > = { = {}, }, }, , true>> = { > = { > = {}, }, }, }, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, true>> = {}, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, false>> = { , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >> = {}, }, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, std::integral_constant >> = {}, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, true>> = {}, , true> > >> = { , true> >, true>> = { , true> >> = { <__gnu_cxx::new_allocator, true> >> = {}, }, }, }, members of std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >: _M_buckets = 0x7f80f6feb110, _M_bucket_count = 1, _M_before_begin = { _M_nxt = 0x0 }, _M_element_count = 0, _M_rehash_policy = { static _S_growth_factor = 2, _M_max_load_factor = 1, _M_next_resize = 0 }, _M_single_bucket = 0x0 } }, flushAccounting = { stats = { _M_h = { , std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Hashtable_traits >> = { , std::__detail::_Select1st, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, true>> = { > = { = {}, }, , true>> = { > = {}, }, > = { = {}, }, }, , true>> = { > = { > = {}, }, }, }, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, true>> = {}, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, false>> = { , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >> = {}, }, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, std::integral_constant >> = {}, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, true>> = {}, , true> > >> = { , true> >, true>> = { , true> >> = { <__gnu_cxx::new_allocator, true> >> = {}, }, }, }, members of std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >: _M_buckets = 0x7f80f6feb148, _M_bucket_count = 1, _M_before_begin = { _M_nxt = 0x0 }, _M_element_count = 0, _M_rehash_policy = { static _S_growth_factor = 2, _M_max_load_factor = 1, _M_next_resize = 0 }, _M_single_bucket = 0x0 } }, droppedStats = { _M_h = { , std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Hashtable_traits >> = { , std::__detail::_Select1st, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, true>> = { > = { = {}, }, , true>> = { > = {}, }, > = { = {}, }, }, , true>> = { > = { > = {}, }, }, }, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, true>> = {}, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, false>> = { , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >> = {}, }, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, std::integral_constant >> = {}, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, true>> = {}, , true> > >> = { , true> >, true>> = { , true> >> = { <__gnu_cxx::new_allocator, true> >> = {}, }, }, }, members of std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >: _M_buckets = 0x7f80f6feb180, _M_bucket_count = 1, _M_before_begin = { _M_nxt = 0x0 }, _M_element_count = 0, _M_rehash_policy = { static _S_growth_factor = 2, _M_max_load_factor = 1, _M_next_resize = 0 }, _M_single_bucket = 0x0 } }, droppedCollections = { _M_h = { , std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Hashtable_traits >> = { , std::__detail::_Select1st, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, true>> = { > = { = {}, }, , true>> = { > = {}, }, > = { = {}, }, }, , true>> = { > = { > = {}, }, }, }, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, true>> = {}, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, false>> = { , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >> = {}, }, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, std::integral_constant >> = {}, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, true>> = {}, , true> > >> = { , true> >, true>> = { , true> >> = { <__gnu_cxx::new_allocator, true> >> = {}, }, }, }, members of std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >: _M_buckets = 0x7f80f6feb1b8, _M_bucket_count = 1, _M_before_begin = { _M_nxt = 0x0 }, _M_element_count = 0, _M_rehash_policy = { static _S_growth_factor = 2, _M_max_load_factor = 1, _M_next_resize = 0 }, _M_single_bucket = 0x0 } }, droppedInStore = { _M_h = { , std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Hashtable_traits >> = { , std::__detail::_Select1st, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, true>> = { > = { = {}, }, , true>> = { > = {}, }, > = { = {}, }, }, , true>> = { > = { > = {}, }, }, }, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, true>> = {}, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, false>> = { , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >> = {}, }, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, std::integral_constant >> = {}, , std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, true>> = {}, , true> > >> = { , true> >, true>> = { , true> >> = { <__gnu_cxx::new_allocator, true> >> = {}, }, }, }, members of std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >: _M_buckets = 0x7f80f6feb1f0, _M_bucket_count = 1, _M_before_begin = { _M_nxt = 0x0 }, _M_element_count = 0, _M_rehash_policy = { static _S_growth_factor = 2, _M_max_load_factor = 1, _M_next_resize = 0 }, _M_single_bucket = 0x0 } } }, manifestUid = { > = { label = { static npos = 18446744073709551615, _M_dataplus = { > = { <__gnu_cxx::new_allocator> = {}, }, members of std::__cxx11::basic_string, std::allocator >::_Alloc_hider: _M_p = 0x7f80f6feb208 "unlabelled" }, _M_string_length = 10, { _M_local_buf = "unlabelled\000\000\000\000\000", _M_allocated_capacity = 7812730926796664437 } } }, members of Monotonic: val = 0 }, manifest = { > = { > = { > = { > = { > = { > = {}, }, }, }, > = {}, > = {}, }, }, members of std::reference_wrapper: _M_data = 0x7f805268c800 }, nonEmptyDroppedCollections = 0, needsMetaCommit = false }, writeOp = Upsert, proposedVBState = { static CurrentVersion = 4, maxDeletedSeqno = { counter = { _M_elems = "\003\000\000\000\000" } }, highSeqno = 826228, purgeSeqno = 0, lastSnapStart = 826229, lastSnapEnd = 826229, maxCas = 1644359583888048128, hlcCasEpochSeqno = 1, mightContainXattrs = false, supportsNamespaces = true, version = 4, persistedCompletedSeqno = 826227, persistedPreparedSeqno = 826227, highPreparedSeqno = 826227, maxVisibleSeqno = 826228, onDiskPrepares = 0, onDiskPrepareBytes = 0, checkpointType = Memory, transition = { failovers = { static npos = 18446744073709551615, _M_dataplus = { > = { <__gnu_cxx::new_allocator> = {}, }, members of std::__cxx11::basic_string, std::allocator >::_Alloc_hider: _M_p = 0x7f7fd1ba3400
}, _M_string_length = 892, { _M_local_buf = "\300\003\000\000\000\000\000\000\300\263\376\366\200\177\000", _M_allocated_capacity = 960 } }, replicationTopology = { m_type = nlohmann::detail::null, m_value = { object = 0x0, array = 0x0, string = 0x0, binary = 0x0, boolean = false, number_integer = 0, number_unsigned = 0, number_float = 0 } }, state = vbucket_state_replica } }, sysErrorCallback = { > = { > = {}, }, = { static _M_max_size = 16, static _M_max_align = 8, _M_functor = { _M_unused = { _M_object = 0x7f80a1473e00, _M_const_object = 0x7f80a1473e00, _M_function_pointer = 0x7f80a1473e00, _M_member_pointer = (void (std::_Undefined_class::*)(std::_Undefined_class * const)) 0x7f80a1473e00, this adjustment 140192438438616 }, _M_pod_data = "\000>G\241\200\177\000\000\330J~\030\201\177\000" }, _M_manager = 0x81f110 >::_M_manager(std::_Any_data &, const std::_Any_data &, std::_Manager_operation)> }, members of std::function: _M_invoker = 0x81efb0 >::_M_invoke(const std::_Any_data &, const std::system_error &)> } } proposedVBState = @0x7f80f6feb238: { static CurrentVersion = 4, maxDeletedSeqno = { counter = { _M_elems = "\003\000\000\000\000" } }, highSeqno = 826228, purgeSeqno = 0, lastSnapStart = 826229, lastSnapEnd = 826229, maxCas = 1644359583888048128, hlcCasEpochSeqno = 1, mightContainXattrs = false, supportsNamespaces = true, version = 4, persistedCompletedSeqno = 826227, persistedPreparedSeqno = 826227, highPreparedSeqno = 826227, maxVisibleSeqno = 826228, onDiskPrepares = 0, onDiskPrepareBytes = 0, checkpointType = Memory, transition = { failovers = { static npos = 18446744073709551615, _M_dataplus = { > = { <__gnu_cxx::new_allocator> = {}, }, members of std::__cxx11::basic_string, std::allocator >::_Alloc_hider: _M_p = 0x7f7fd1ba3400
}, _M_string_length = 892, { _M_local_buf = "\300\003\000\000\000\000\000\000\300\263\376\366\200\177\000", _M_allocated_capacity = 960 } }, replicationTopology = { m_type = nlohmann::detail::null, m_value = { object = 0x0, array = 0x0, string = 0x0, binary = 0x0, boolean = false, number_integer = 0, number_unsigned = 0, number_float = 0 } }, state = vbucket_state_replica } } maxSeqno = maxVbStateOpCas = minSeqno = flushBatchSize = mustPersistVBState = hcs = hps = maxVisibleSeqno = { > = { label = { static npos = 18446744073709551615, _M_dataplus = { > = { <__gnu_cxx::new_allocator> = {}, }, members of std::__cxx11::basic_string, std::allocator >::_Alloc_hider: _M_p = 0x7f80f6feae20 "unlabelled" }, _M_string_length = 10, { _M_local_buf = "unlabelled\000_logg", _M_allocated_capacity = 7812730926796664437 } } }, members of Monotonic: val = 826229 } aggStats = { numItems = 2, totalBytes = 1224, totalAgeInMilliseconds = 29313079464 } highSeqno = #8 0x000000000082b48f in EPBucket::flushVBucket(Vbid) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:374 vb = { vb = { > = { > = {}, members of std::__shared_ptr: _M_ptr = 0x0, _M_refcount = { _M_pi = 0x0 } }, }, lock = { _M_device = 0x0, _M_owns = false } } #9 0x00000000006de456 in Flusher::flushVB (this=0x7f80537dd100) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:293 checkpointRemovalMode = Eager vbid = { vbid = 778 } #10 0x00000000006dee10 in Flusher::step(GlobalTask*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:203 more = currentState = Flusher::Running #11 0x0000000000ab48c2 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:68 guard = { previous = 0x0 } #12 0x0000000000ab19c5 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f80f6feb840) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:189 scheduleOverhead = start = { __d = { __r = 14656544479614302 } } runAgain = false proxy = @0x7f812841de70: #13 0x0000000000c08e00 in operator() (this=0x7f80f6feb840) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 fn = @0x7f80f6feb840: { > = {}, members of folly::Function: data_ = { big = 0x7f812841de70, tiny = { __data = "p\336A(\201\177\000\000\000\000\000\000\000\000\000\000\220ea\204\200\177\000\000(\364\000\000\000\000\000\000H\000\000\000\000\000\000\000\020\271\376\366\200\177\000", __align = {} } }, call_ = 0xab2080 ::callSmall(folly::detail::function::Data&)>, exec_ = 0xab07d0 (folly::detail::function::Op, folly::detail::function::Data*, folly::detail::function::Data)> } #14 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f813e12b400, thread=..., task=task@entry=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 rctx = { prev_ = { > = { > = {}, members of std::__shared_ptr: _M_ptr = 0x0, _M_refcount = { _M_pi = 0x0 } }, } } startTime = { __d = { __r = 14656544479609608 } } stats = { expired = false, waitTime = { __r = 579996 }, runTime = { __r = 0 }, enqueueTime = { __d = { __r = 14656544479029612 } }, requestId = 0 } #15 0x0000000000bf0bba in folly::CPUThreadPoolExecutor::threadRun (this=0x7f813e12b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 task = { storage_ = { { emptyState = 112 'p', value = { = { func_ = { > = {}, members of folly::Function: data_ = { big = 0x7f812841de70, tiny = { __data = "p\336A(\201\177\000\000\000\000\000\000\000\000\000\000\220ea\204\200\177\000\000(\364\000\000\000\000\000\000H\000\000\000\000\000\000\000\020\271\376\366\200\177\000", __align = {} } }, call_ = 0xab2080 ::callSmall(folly::detail::function::Data&)>, exec_ = 0xab07d0 (folly::detail::function::Op, folly::detail::function::Data*, folly::detail::function::Data)> }, enqueueTime_ = { __d = { __r = 14656544479029612 } }, expiration_ = { __r = 0 }, expireCallback_ = { > = {}, members of folly::Function: data_ = { big = 0xf428, tiny = { __data = "(\364\000\000\000\000\000\000\033\023\244", '\000' , "_~0B\201\177\000\000p\212|)\201\177\000\000@\026QB\201\177\000", __align = {} } }, call_ = 0x46703d ::uninitCall(folly::detail::function::Data&)>, exec_ = 0x0 }, context_ = { > = { > = {}, members of std::__shared_ptr: _M_ptr = 0x0, _M_refcount = { _M_pi = 0x0 } }, } }, members of folly::CPUThreadPoolExecutor::CPUTask: poison = false, priority_ = 0 '\000', queueObserverPayload_ = 0 } }, hasValue = true } } guard = { list_ = { forbid = true, prev = 0x0, curr = { name = { static npos = , b_ = 0xcdb04b "CPUThreadPoolExecutor", e_ = 0xcdb060 "" } } } } #16 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 No locals. #17 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 No locals. #18 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 No locals. #19 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 No locals. #20 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 fn = #21 0x0000000000ab1654 in operator() (this=0x7f813e5dbbc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 fn = @0x7f813e5dbbc0: #22 operator() (__closure=0x7f813e5dbbc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 threadNameOpt = { storage_ = { { emptyState = -96 '\240', value = { static npos = 18446744073709551615, _M_dataplus = { > = { <__gnu_cxx::new_allocator> = {}, }, members of std::__cxx11::basic_string, std::allocator >::_Alloc_hider: _M_p = 0x7f80f6feb9a0 "WriterPool6" }, _M_string_length = 11, { _M_local_buf = "WriterPool6\000\000\000\000", _M_allocated_capacity = 8021036716417184343 } } }, hasValue = true } } func = #23 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 fn = @0x7f813e5dbbc0: #24 0x00007f81400ced40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 No locals. #25 0x00007f8141ed6ea5 in start_thread (arg=0x7f80f6ffd700) at pthread_create.c:307 __res = pd = 0x7f80f6ffd700 now = unwind_buf = { cancel_jmp_buf = {{ jmp_buf = {140191876503296, 692391505224261732, 0, 8392704, 0, 140191876503296, -677172054660489116, -677407202365284252}, mask_was_saved = 0 }}, priv = { pad = {0x0, 0x0, 0x0, 0x0}, data = { prev = 0x0, cleanup = 0x0, canceltype = 0 } } } not_first_call = pagesize_m1 = sp = freesize = #26 0x00007f813f8178dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 No locals. ############################## running: gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit 2022-02-08 14:33:26,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/librt.so.1...Reading symbols from /usr/lib/debug/usr/lib64/librt-2.17.so.debug...done. done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/bin/../lib/liblz4.so.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/liblz4.so.1.9.2-7.1.0-2255.x86_64.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/liblz4.so.1 Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libsnappy.so.1.1.8-7.1.0-2255.x86_64.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1 Reading symbols from /opt/couchbase/bin/../lib/libjemalloc.so.2...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libjemalloc.so.2-7.1.0-2255.x86_64.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libjemalloc.so.2 Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_core-2.1.so.7.0.0-7.1.0-2255.x86_64.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_extra-2.1.so.7.0.0-7.1.0-2255.x86_64.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_pthreads-2.1.so.7.0.0-7.1.0-2255.x86_64.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_openssl-2.1.so.7.0.0-7.1.0-2255.x86_64.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libssl.so.1.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libssl.so.1.1-7.1.0-2255.x86_64.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libssl.so.1.1 Reading symbols from /opt/couchbase/bin/../lib/libcrypto.so.1.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libcrypto.so.1.1-7.1.0-2255.x86_64.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libcrypto.so.1.1 Reading symbols from /opt/couchbase/bin/../lib/libstdc++.so.6...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libstdc++.so.6.0.28-7.1.0-2255.x86_64.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libstdc++.so.6 Reading symbols from /lib64/libm.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libm-2.17.so.debug...done. done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /opt/couchbase/bin/../lib/libgcc_s.so.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libgcc_s.so.1-7.1.0-2255.x86_64.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libgcc_s.so.1 Reading symbols from /lib64/libc.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libc-2.17.so.debug...done. done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/usr/lib64/ld-2.17.so.debug...done. done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 0x00007f1727fe4eb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) To enable execution of this file add add-auto-load-safe-path /opt/couchbase/lib/libstdc++.so.6.0.28-gdb.py line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" Thread 65 (Thread 0x7f17275ff700 (LWP 67078)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f1729bb15a7 in background_thread_sleep (tsdn=, interval=10000000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=0, info=, tsdn=) at src/background_thread.c:307 #3 background_thread0_work (tsd=0x7f17275ee220) at src/background_thread.c:452 #4 background_work (ind=, tsd=0x7f17275ee220) at src/background_thread.c:490 #5 background_thread_entry (ind_arg=) at src/background_thread.c:522 #6 0x00007f172a6a3ea5 in start_thread (arg=0x7f17275ff700) at pthread_create.c:307 #7 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 64 (Thread 0x7f17264fe700 (LWP 67079)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000b6466a in __gthread_cond_timedwait ( __abs_timeout=0x7f17264ec800, __mutex=, __cond=0x7f1726dc2138) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=0x7f1726dc2138) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > ( __atime=..., __lock=..., this=0x7f1726dc2138) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_until >, spdlog::details::mpmc_blocking_queue::dequeue_for(T&, std::chrono::milliseconds) [with T = spdlog::details::async_msg]:: > (__p=..., __atime=..., __lock=..., this=0x7f1726dc2138) at /usr/local/include/c++/7.3.0/condition_variable:129 #5 wait_for, spdlog::details::mpmc_blocking_queue::dequeue_for(T&, std::chrono::milliseconds) [with T = spdlog::details::async_msg]:: > (__p=..., __rtime=, __lock=..., this=0x7f1726dc2138) at /usr/local/include/c++/7.3.0/condition_variable:145 #6 dequeue_for (wait_duration=..., popped_item=..., this=0x7f1726dc2110) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/mpmc_blocking_q.h:58 #7 spdlog::details::thread_pool::process_next_msg_ ( this=this@entry=0x7f1726dc2110) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/thread_pool-inl.h:99 #8 0x0000000000b649b8 in worker_loop_ (this=0x7f1726dc2110) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/thread_pool-inl.h:90 #9 operator() (__closure=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/thread_pool-inl.h:28 #10 __invoke_impl):: > (__f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:60 #11 __invoke):: > (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #12 _M_invoke<0> (this=) at /usr/local/include/c++/7.3.0/thread:234 #13 operator() (this=) at /usr/local/include/c++/7.3.0/thread:243 #14 std::thread::_State_impl):: > > >::_M_run(void) (this=) at /usr/local/include/c++/7.3.0/thread:186 #15 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #16 0x00007f172a6a3ea5 in start_thread (arg=0x7f17264fe700) at pthread_create.c:307 #17 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 63 (Thread 0x7f1725cfd700 (LWP 67080)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000b6442a in __gthread_cond_timedwait ( __abs_timeout=0x7f1725ceb9b0, __mutex=, __cond=0x7f1726db8268) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=0x7f1726db8268) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > ( __atime=..., __lock=..., this=0x7f1726db8268) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_until >, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __atime=..., __lock=..., this=0x7f1726db8268) at /usr/local/include/c++/7.3.0/condition_variable:129 #5 wait_for, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __rtime=..., __lock=..., this=0x7f1726db8268) at /usr/local/include/c++/7.3.0/condition_variable:145 #6 operator() (__closure=0x7f1726dc3a48) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/periodic_worker-inl.h:25 #7 __invoke_impl&, std::chrono::seconds):: > ( __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:60 #8 __invoke&, std::chrono::seconds):: > (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #9 _M_invoke<0> (this=0x7f1726dc3a48) at /usr/local/include/c++/7.3.0/thread:234 #10 operator() (this=0x7f1726dc3a48) at /usr/local/include/c++/7.3.0/thread:243 #11 std::thread::_State_impl&, std::chrono::seconds):: > > >::_M_run(void) (this=0x7f1726dc3a40) at /usr/local/include/c++/7.3.0/thread:186 #12 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #13 0x00007f172a6a3ea5 in start_thread (arg=0x7f1725cfd700) at pthread_create.c:307 #14 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 62 (Thread 0x7f17254fc700 (LWP 67081)): #0 0x00007f1727fd9c3d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000005ac9c1 in get_command (buffsize=80, buffer=0x7f17254ea990 "@\346\315*\027\177") at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:57 #2 check_stdin_thread (arg=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:98 #3 0x0000000000b9d269 in run (this=0x7f1726db6890) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #4 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #5 0x00007f172a6a3ea5 in start_thread (arg=0x7f17254fc700) at pthread_create.c:307 #6 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 61 (Thread 0x7f17234f8700 (LWP 67085)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00000000005f4116 in __gthread_cond_timedwait ( __abs_timeout=0x7f17234e6980, __mutex=0x7f1726de6350, __cond=0x7f1726de6378) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=, __lock=, __atime=, this=0x7f1726de6378) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=, this=0x7f1726de6378) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=, __lock=, this=0x7f1726de6378) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 ExternalAuthManagerThread::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/external_auth_manager_thread.cc:91 #6 0x0000000000bb38a3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #7 0x0000000000b9d269 in run (this=0x7f1726db6c50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f172a6a3ea5 in start_thread (arg=0x7f17234f8700) at pthread_create.c:307 #10 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 60 (Thread 0x7f1722cf7700 (LWP 67086)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000067c9b7 in __gthread_cond_timedwait ( __abs_timeout=0x7f1722ce59b0, __mutex=, __cond=0x7f172691d3f8) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7f172691d3f8) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7f172691d3f8) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7f172691d3f8) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 AuditImpl::consume_events() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/auditd/src/audit.cc:411 #6 0x0000000000b9d269 in run (this=0x7f1726db6bf0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #7 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #8 0x00007f172a6a3ea5 in start_thread (arg=0x7f1722cf7700) at pthread_create.c:307 #9 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 59 (Thread 0x7f1723cf9700 (LWP 67087)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000ae385b in consume_socket (thread_index=, sp=0x29cfd60, ctx=0x29d15b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18846 #2 worker_thread_run (conn=0x29cf470) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18987 #3 worker_thread (thread_func_param=0x29cf470) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19145 #4 0x00007f172a6a3ea5 in start_thread (arg=0x7f1723cf9700) at pthread_create.c:307 #5 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 58 (Thread 0x7f17244fa700 (LWP 67088)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000ae385b in consume_socket (thread_index=, sp=0x29d0748, ctx=0x29d15b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18846 #2 worker_thread_run (conn=0x29cfe58) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18987 #3 worker_thread (thread_func_param=0x29cfe58) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19145 #4 0x00007f172a6a3ea5 in start_thread (arg=0x7f17244fa700) at pthread_create.c:307 #5 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 57 (Thread 0x7f1724cfb700 (LWP 67089)): #0 0x00007f1727fd9c3d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000ada7c8 in master_thread_run (ctx=0x29d15b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19291 #2 master_thread (thread_func_param=0x29d15b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19386 #3 0x00007f172a6a3ea5 in start_thread (arg=0x7f1724cfb700) at pthread_create.c:307 #4 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 56 (Thread 0x7f17224f6700 (LWP 67090)): #0 0x00007f1727fe4eb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f17299783c5 in epoll_dispatch (base=0x7f172688b900, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f172996f376 in event_base_loop (base=0x7f172688b900, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7f172692c010, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7f172692c010) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever ( this=this@entry=0x7f172692c010) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005b1be9 in worker_libevent (arg=0x7f172692c000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b9d269 in run (this=0x7f1726db6ef0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f172a6a3ea5 in start_thread (arg=0x7f17224f6700) at pthread_create.c:307 #10 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 55 (Thread 0x7f1721cf5700 (LWP 67091)): #0 0x00007f1727fe4eb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f17299783c5 in epoll_dispatch (base=0x7f172688bc00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f172996f376 in event_base_loop (base=0x7f172688bc00, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7f172692cbe0, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7f172692cbe0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever ( this=this@entry=0x7f172692cbe0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005b1be9 in worker_libevent (arg=0x7f172692cbd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b9d269 in run (this=0x7f1726db7010) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f172a6a3ea5 in start_thread (arg=0x7f1721cf5700) at pthread_create.c:307 #10 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 54 (Thread 0x7f17214f4700 (LWP 67092)): #0 0x00007f1727fe4eb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f17299783c5 in epoll_dispatch (base=0x7f172688bf00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f172996f376 in event_base_loop (base=0x7f172688bf00, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7f172692d7b0, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7f172692d7b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever ( this=this@entry=0x7f172692d7b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005b1be9 in worker_libevent (arg=0x7f172692d7a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b9d269 in run (this=0x7f1726db7130) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f172a6a3ea5 in start_thread (arg=0x7f17214f4700) at pthread_create.c:307 #10 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 53 (Thread 0x7f1720cf3700 (LWP 67093)): #0 0x00007f1727fe4eb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f17299783c5 in epoll_dispatch (base=0x7f172688c200, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f172996f376 in event_base_loop (base=0x7f172688c200, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7f172692e380, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7f172692e380) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever ( this=this@entry=0x7f172692e380) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005b1be9 in worker_libevent (arg=0x7f172692e370) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b9d269 in run (this=0x7f1726db7220) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f172a6a3ea5 in start_thread (arg=0x7f1720cf3700) at pthread_create.c:307 #10 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 52 (Thread 0x7f171bfff700 (LWP 67094)): #0 0x00007f1727fe4eb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f17299783c5 in epoll_dispatch (base=0x7f172688c500, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f172996f376 in event_base_loop (base=0x7f172688c500, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7f172692ef50, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7f172692ef50) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever ( this=this@entry=0x7f172692ef50) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005b1be9 in worker_libevent (arg=0x7f172692ef40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b9d269 in run (this=0x7f1726db71f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f172a6a3ea5 in start_thread (arg=0x7f171bfff700) at pthread_create.c:307 #10 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 51 (Thread 0x7f171b7fe700 (LWP 67095)): #0 0x00007f1727fe4eb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f17299783c5 in epoll_dispatch (base=0x7f172688c800, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f172996f376 in event_base_loop (base=0x7f172688c800, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7f172692fb20, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7f172692fb20) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever ( this=this@entry=0x7f172692fb20) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005b1be9 in worker_libevent (arg=0x7f172692fb10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b9d269 in run (this=0x7f1726db7190) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f172a6a3ea5 in start_thread (arg=0x7f171b7fe700) at pthread_create.c:307 #10 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 50 (Thread 0x7f171affd700 (LWP 67096)): #0 0x00007f1727fe4eb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f17299783c5 in epoll_dispatch (base=0x7f172688cb00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f172996f376 in event_base_loop (base=0x7f172688cb00, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7f17269306f0, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7f17269306f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever ( this=this@entry=0x7f17269306f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005b1be9 in worker_libevent (arg=0x7f17269306e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b9d269 in run (this=0x7f1726db71c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f172a6a3ea5 in start_thread (arg=0x7f171affd700) at pthread_create.c:307 #10 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 49 (Thread 0x7f17193fc700 (LWP 67097)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=423536152, absSteadyTime=absSteadyTime@entry=0x7f17193ea5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda018) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda018, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f17193ea618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda018) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda018, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda018, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d6ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d6ac0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d6a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f172692a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f17268f2b00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f17268f2b00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f17193fc700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 48 (Thread 0x7f1718bfb700 (LWP 67098)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=415143448, absSteadyTime=absSteadyTime@entry=0x7f1718be95f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda030) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda030, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f1718be9618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda030) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda030, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda030, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d6ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d6ac0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d6a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f172692a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f17268f2c00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f17268f2c00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f1718bfb700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 47 (Thread 0x7f1713fff700 (LWP 67099)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=335468056, absSteadyTime=absSteadyTime@entry=0x7f1713fed5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda048) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda048, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f1713fed618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda048) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda048, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda048, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d6ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d6ac0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d6a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f172692a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f17268f2c40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f17268f2c40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f1713fff700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 46 (Thread 0x7f1711fd9700 (LWP 67100)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=301757976, absSteadyTime=absSteadyTime@entry=0x7f1711fc75f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda060) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda060, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f1711fc7618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda060) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda060, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda060, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d6ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d6ac0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d6a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f172692a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f17268f2cc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f17268f2cc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f1711fd9700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 45 (Thread 0x7f17117d8700 (LWP 67101)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=293365272, absSteadyTime=absSteadyTime@entry=0x7f17117c65f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda078) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda078, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f17117c6618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda078) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda078, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda078, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d6ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d6ac0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d6a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f172692a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f17268f2d00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f17268f2d00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f17117d8700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 44 (Thread 0x7f1710fd7700 (LWP 67102)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=284972568, absSteadyTime=absSteadyTime@entry=0x7f1710fc55f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda090) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda090, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f1710fc5618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda090) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda090, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda090, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d6ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d6ac0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d6a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f172692a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f17268f2c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f17268f2c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f1710fd7700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 43 (Thread 0x7f170bfff700 (LWP 67103)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=201250328, absSteadyTime=absSteadyTime@entry=0x7f170bfed5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda0a8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda0a8, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f170bfed618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda0a8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda0a8, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda0a8, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d6ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d6ac0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d6a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f172692a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f17268f2d40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f17268f2d40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f170bfff700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 42 (Thread 0x7f170b7fe700 (LWP 67104)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=192857624, absSteadyTime=absSteadyTime@entry=0x7f170b7ec5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda0c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda0c0, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f170b7ec618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda0c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda0c0, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda0c0, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d6ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d6ac0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d6a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f172692a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f17268f2dc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f17268f2dc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f170b7fe700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 41 (Thread 0x7f170affd700 (LWP 67105)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=184464920, absSteadyTime=absSteadyTime@entry=0x7f170afeb5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda0d8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda0d8, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f170afeb618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda0d8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda0d8, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda0d8, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d81c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d81c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d8180, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f172692b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f17268f2e80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f17268f2e80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f170affd700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 40 (Thread 0x7f170a7fc700 (LWP 67106)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=176072216, absSteadyTime=absSteadyTime@entry=0x7f170a7ea5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda0f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda0f0, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f170a7ea618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda0f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda0f0, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda0f0, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d81c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d81c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d8180, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f172692b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f17268f2ec0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f17268f2ec0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f170a7fc700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 39 (Thread 0x7f1709ffb700 (LWP 67107)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=167679512, absSteadyTime=absSteadyTime@entry=0x7f1709fe95f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda108) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda108, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f1709fe9618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda108) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda108, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda108, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d81c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d81c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d8180, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f172692b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f1726dd7ac0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f1726dd7ac0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f1709ffb700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 38 (Thread 0x7f17097fa700 (LWP 67108)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=159286808, absSteadyTime=absSteadyTime@entry=0x7f17097e85f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda120) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda120, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f17097e8618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda120) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda120, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda120, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d81c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d81c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d8180, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f172692b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f1726dd7b40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f1726dd7b40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f17097fa700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 37 (Thread 0x7f1708ff9700 (LWP 67109)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=150894104, absSteadyTime=absSteadyTime@entry=0x7f1708fe75f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda138) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda138, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f1708fe7618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda138) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda138, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda138, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d81c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d81c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d8180, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f172692b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f1726dd7b80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f1726dd7b80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f1708ff9700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 36 (Thread 0x7f16dbfff700 (LWP 67110)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3690911256, absSteadyTime=absSteadyTime@entry=0x7f16dbfed5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda150) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda150, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f16dbfed618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda150) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda150, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda150, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d81c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d81c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d8180, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f172692b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f1726dd7b00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f1726dd7b00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f16dbfff700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 35 (Thread 0x7f16db7fe700 (LWP 67111)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3682518552, absSteadyTime=absSteadyTime@entry=0x7f16db7ec5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda168) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda168, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f16db7ec618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda168) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda168, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda168, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d81c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d81c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d8180, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f172692b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f1726ddbbc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f1726ddbbc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f16db7fe700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 34 (Thread 0x7f16daffd700 (LWP 67112)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3674125848, absSteadyTime=absSteadyTime@entry=0x7f16dafeb5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda180) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda180, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f16dafeb618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda180) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda180, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda180, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d81c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d81c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d8180, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f172692b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f1726ddbc00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f1726ddbc00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f16daffd700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 33 (Thread 0x7f16da7fc700 (LWP 67113)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3665733144, absSteadyTime=absSteadyTime@entry=0x7f16da7ea5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda198) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda198, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f16da7ea618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda198) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda198, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda198, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d98c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d98c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d9880, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f1726951000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f1726ddbd00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f1726ddbd00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f16da7fc700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 32 (Thread 0x7f16d9ffb700 (LWP 67114)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3657340440, absSteadyTime=absSteadyTime@entry=0x7f16d9fe95f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda1b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda1b0, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f16d9fe9618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda1b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda1b0, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda1b0, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d98c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d98c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d9880, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f1726951000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f1726ddbd40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f1726ddbd40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f16d9ffb700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 31 (Thread 0x7f16d97fa700 (LWP 67115)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3648947736, absSteadyTime=absSteadyTime@entry=0x7f16d97e85f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda1c8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda1c8, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f16d97e8618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda1c8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda1c8, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda1c8, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d98c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d98c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d9880, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f1726951000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f1726ddbd80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f1726ddbd80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f16d97fa700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 30 (Thread 0x7f16d8ff9700 (LWP 67116)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3640555032, absSteadyTime=absSteadyTime@entry=0x7f16d8fe75f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda1e0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda1e0, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f16d8fe7618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda1e0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda1e0, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda1e0, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d98c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d98c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d9880, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f1726951000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f1726ddbe00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f1726ddbe00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f16d8ff9700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 29 (Thread 0x7f16bbfff700 (LWP 67117)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3154040344, absSteadyTime=absSteadyTime@entry=0x7f16bbfed5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda1f8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda1f8, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f16bbfed618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda1f8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda1f8, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda1f8, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d98c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d98c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d9880, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f1726951000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f1726ddbe40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f1726ddbe40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f16bbfff700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 28 (Thread 0x7f16bb7fe700 (LWP 67118)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3145647640, absSteadyTime=absSteadyTime@entry=0x7f16bb7ec5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda210) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda210, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f16bb7ec618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda210) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda210, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda210, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d98c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d98c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d9880, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f1726951000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f1726ddbdc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f1726ddbdc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f16bb7fe700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 27 (Thread 0x7f16baffd700 (LWP 67119)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3137254936, absSteadyTime=absSteadyTime@entry=0x7f16bafeb5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda228) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda228, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f16bafeb618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda228) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda228, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda228, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d98c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d98c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d9880, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f1726951000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f1726ddbe80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f1726ddbe80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f16baffd700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 26 (Thread 0x7f16ba7fc700 (LWP 67120)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3128862232, absSteadyTime=absSteadyTime@entry=0x7f16ba7ea5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda240) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda240, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f16ba7ea618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda240) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda240, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda240, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29d98c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29d98c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x29d9880, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f1726951000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7f1726de3ec0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7f1726de3ec0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f172a6a3ea5 in start_thread (arg=0x7f16ba7fc700) at pthread_create.c:307 #22 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 25 (Thread 0x7f16b9ffb700 (LWP 67121)): #0 0x00007f172a6aafc3 in pread64 () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000009ac488 in magma::BasicFile::Read(long, unsigned long, char*, unsigned long*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_linux.cc:146 #2 0x0000000000a13afa in magma::FileWithStats::Read(long, unsigned long, char*, unsigned long*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_stats.cc:39 #3 0x00000000009b07e6 in magma::BufferedFileReader::Read(magma::File*, long, unsigned long, magma::Slice&, unsigned long&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/rw_file.cc:262 #4 0x00000000009bd8c4 in magma::ReadAndTransform(magma::File*, long, unsigned long, std::function >&, unsigned long&)>, std::unique_ptr >&, unsigned long&, unsigned long&, magma::BufferedFileReader*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/block_cache.cc:39 #5 0x00000000009bdab7 in magma::Cache::makeObject(magma::BlockCache::BlockCtor&) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #6 0x00000000009bce2e in magma::Cache::AcquireObject (this=0x7f17083760c0, ctor=..., doCaching=doCaching@entry=true) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:779 #7 0x00000000009b4925 in AcquireObject (doCaching=, ctor=..., this=0x7f166a165760) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/block_cache.h:61 #8 Read (buffer=, transform=..., doCaching=, readBytes=, outData=..., size=, offset=, fileID=, file=, this=0x7f166a165760) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/block_cache.h:61 #9 magma::ManagedFile::Read(std::shared_ptr const&, long, unsigned long, std::shared_ptr&, unsigned long&, magma::CacheReadMode, std::function >&, unsigned long&)>, magma::BufferedFileReader*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file_manager.cc:110 #10 0x0000000000a19469 in magma::SSTable::readBlock(std::shared_ptr const&, magma::BlockPointer const&, magma::Block*, magma::ReaderStats&, bool, magma::BufferedFileReader*) () at /opt/gcc-10.2.0/include/c++/10.2.0/new:175 #11 0x0000000000a1b7f1 in magma::SSTableIterator::initIterators(magma::Slice const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:762 #12 0x0000000000a1bece in magma::SSTableIterator::Seek (this=0x7f16635dda80, key=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:808 #13 0x00000000009efed9 in operator() (table=..., __closure=0x7f165bbb6e00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:1156 #14 __invoke_impl&, magma::LookupBuffer&, magma::Slice&, bool&, bool, magma::ReaderStats*)::&, magma::Table&> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:60 #15 __invoke_r&, magma::LookupBuffer&, magma::Slice&, bool&, bool, magma::ReaderStats*)::&, magma::Table&> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:113 #16 std::_Function_handler&, magma::LookupBuffer&, magma::Slice&, bool&, bool, magma::ReaderStats*)::{lambda(magma::Table&)#1}>::_M_invoke(std::_Any_data const&, magma::Table&) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:291 #17 0x00000000009dd79a in operator() (__args#0=..., this=0x7f16b9fe6f80) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #18 magma::TreeSnapshot::LookupTableByKey(magma::Slice const&, std::function, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/tree_snapshot.cc:365 #19 0x00000000009f716e in magma::LSMTree::Get(magma::Slice const&, std::shared_ptr&, magma::LookupBuffer&, magma::Slice&, bool&, bool, magma::ReaderStats*) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:87 #20 0x000000000098dee4 in magma::KVStore::KeyExists(magma::Slice const&, magma::LookupBuffer&, magma::ReaderStats*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:610 #21 0x000000000099b09c in operator() (buffer=..., index=21, __closure=0x7f16b9fe71b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/write.cc:61 #22 magma::KVStore::WriteDocs(magma::WAL*, std::vector > const&, std::function, std::function > const*> ()>) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/write.cc:103 #23 0x0000000000996096 in magma::KVStore::ReplayWrites(std::vector > const&, magma::WALOffset) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:746 #24 0x000000000099df83 in magma::Magma::Impl::recovery() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db_recovery.cc:180 #25 0x0000000000936485 in magma::Magma::Impl::Open() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:168 #26 0x000000000093667d in magma::Magma::Open (this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:187 #27 0x0000000000899671 in MagmaMemoryTrackingProxy::Open() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:337 #28 0x0000000000886672 in MagmaKVStore::MagmaKVStore(MagmaKVStoreConfig&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:651 #29 0x0000000000839b4b in make_unique () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/kvstore.cc:204 #30 KVStoreFactory::create(KVStoreConfig&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/kvstore.cc:205 #31 0x000000000070b507 in KVShard::KVShard(Configuration&, unsigned short, unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvshard.cc:44 #32 0x00000000007407f9 in make_unique () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.h:692 #33 VBucketMap::VBucketMap(KVBucket&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucketmap.cc:28 #34 0x00000000006f9c6b in KVBucket::KVBucket(EventuallyPersistentEngine&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.h:657 #35 0x0000000000821216 in EPBucket::EPBucket(EventuallyPersistentEngine&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:241 #36 0x00000000006a75f8 in make_unique () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:6924 #37 EventuallyPersistentEngine::makeBucket(Configuration&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:6924 #38 0x00000000006b8a12 in EventuallyPersistentEngine::initialize(std::__cxx11::basic_string, std::allocator > const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:2164 #39 0x00000000005c41cc in BucketManager::create(Cookie&, std::__cxx11::basic_string, std::allocator >, std::__cxx11::basic_string, std::allocator >, BucketType) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/buckets.cc:227 #40 0x000000000062de5e in operator() (__closure=0x7f1726a582e0) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/char_traits.h:322 #41 __invoke_impl&> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:60 #42 __invoke_r&> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:110 #43 std::_Function_handler::_M_invoke(std::_Any_data const&) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:291 #44 0x000000000062c6d4 in operator() (this=0x7f1710484a08) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #45 OneShotTask::run (this=0x7f1710484990) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/one_shot_task.h:50 #46 0x0000000000ab48c2 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:68 #47 0x0000000000ab19c5 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f16b9fe9840) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:189 #48 0x0000000000c08e00 in operator() (this=0x7f16b9fe9840) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #49 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) ( this=this@entry=0x7f1726951c00, thread=..., task=task@entry=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #50 0x0000000000bf0bba in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f1726951c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #51 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #52 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #53 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #54 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #55 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #56 0x0000000000ab1654 in operator() (this=0x7f1726de3fc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #57 operator() (__closure=0x7f1726de3fc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #58 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #59 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #60 0x00007f172a6a3ea5 in start_thread (arg=0x7f16b9ffb700) at pthread_create.c:307 #61 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 24 (Thread 0x7f16b97fa700 (LWP 67122)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3112076824, absSteadyTime=absSteadyTime@entry=0x7f16b97e85f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f1711fda270) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f1711fda270, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f16b97e8618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f1711fda270) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7f1711fda270, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf8947 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7f1711fda270, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x29dafc0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x29dafc0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x29daf80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7f1726951c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000ab1654 in operator() (this=0x7f1726ded000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 operator() (__closure=0x7f1726ded000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007f172a6a3ea5 in start_thread (arg=0x7f16b97fa700) at pthread_create.c:307 #23 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 23 (Thread 0x7f16b8ff9700 (LWP 67123)): #0 0x00007f1727fe4eb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f17299783c5 in epoll_dispatch (base=0x7f172688ce00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f172996f376 in event_base_loop (base=0x7f172688ce00, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7f1726de6f00, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7f1726de6f00) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever (this=0x7f1726de6f00) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x0000000000c00629 in folly::IOThreadPoolExecutor::threadRun ( this=0x7f1726de6780, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/IOThreadPoolExecutor.cpp:195 #7 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #8 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #9 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #10 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #11 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #12 0x00007f172889bd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #13 0x00007f172a6a3ea5 in start_thread (arg=0x7f16b8ff9700) at pthread_create.c:307 #14 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 22 (Thread 0x7f168bfff700 (LWP 67124)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f1729bb0e51 in background_thread_sleep (tsdn=, interval=7525000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=, info=, tsdn=) at src/background_thread.c:307 #3 background_work (ind=, tsd=0x7f168bfee220) at src/background_thread.c:497 #4 background_thread_entry (ind_arg=) at src/background_thread.c:522 #5 0x00007f172a6a3ea5 in start_thread (arg=0x7f168bfff700) at pthread_create.c:307 #6 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 21 (Thread 0x7f168b7fe700 (LWP 67125)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00000000009bf8ff in __gthread_cond_timedwait ( __abs_timeout=0x7f168b7ec960, __mutex=, __cond=0x7f1726a84998) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7f1726a84998) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7f1726a84998) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7f1726a84998) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:103 #6 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #7 0x0000000000b9d269 in run (this=0x7f17105387e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f172a6a3ea5 in start_thread (arg=0x7f168b7fe700) at pthread_create.c:307 #10 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 20 (Thread 0x7f168affd700 (LWP 67126)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00000000009bf8ff in __gthread_cond_timedwait ( __abs_timeout=0x7f168afeb960, __mutex=, __cond=0x7f1726a84998) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7f1726a84998) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7f1726a84998) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7f1726a84998) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:103 #6 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #7 0x0000000000b9d269 in run (this=0x7f1710538810) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f172a6a3ea5 in start_thread (arg=0x7f168affd700) at pthread_create.c:307 #10 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 19 (Thread 0x7f168a7fc700 (LWP 67127)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00000000009bf8ff in __gthread_cond_timedwait ( __abs_timeout=0x7f168a7ea960, __mutex=, __cond=0x7f1726a84998) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7f1726a84998) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7f1726a84998) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7f1726a84998) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:103 #6 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #7 0x0000000000b9d269 in run (this=0x7f1710538840) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f172a6a3ea5 in start_thread (arg=0x7f168a7fc700) at pthread_create.c:307 #10 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 18 (Thread 0x7f1689ffb700 (LWP 67128)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00000000009bf8ff in __gthread_cond_timedwait ( __abs_timeout=0x7f1689fe9960, __mutex=, __cond=0x7f1726a84998) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7f1726a84998) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7f1726a84998) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7f1726a84998) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:103 #6 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #7 0x0000000000b9d269 in run (this=0x7f1710538870) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f172a6a3ea5 in start_thread (arg=0x7f1689ffb700) at pthread_create.c:307 #10 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 17 (Thread 0x7f16897fa700 (LWP 67129)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f17288968bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7f17105388a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f172a6a3ea5 in start_thread (arg=0x7f16897fa700) at pthread_create.c:307 #8 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 16 (Thread 0x7f1688ff9700 (LWP 67130)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f17288968bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7f17105388d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f172a6a3ea5 in start_thread (arg=0x7f1688ff9700) at pthread_create.c:307 #8 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 15 (Thread 0x7f1673fff700 (LWP 67131)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f17288968bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7f1710538900) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f172a6a3ea5 in start_thread (arg=0x7f1673fff700) at pthread_create.c:307 #8 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 14 (Thread 0x7f16737fe700 (LWP 67132)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f17288968bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7f1710538930) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f172a6a3ea5 in start_thread (arg=0x7f16737fe700) at pthread_create.c:307 #8 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 13 (Thread 0x7f16702ff700 (LWP 67227)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f17288968bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7f16d84901d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f172a6a3ea5 in start_thread (arg=0x7f16702ff700) at pthread_create.c:307 #8 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 12 (Thread 0x7f166fafe700 (LWP 67228)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f17288968bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7f16d8457ac0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f172a6a3ea5 in start_thread (arg=0x7f166fafe700) at pthread_create.c:307 #8 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 11 (Thread 0x7f166f2fd700 (LWP 67229)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f17288968bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7f16d848ac60) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f172a6a3ea5 in start_thread (arg=0x7f166f2fd700) at pthread_create.c:307 #8 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 10 (Thread 0x7f166eafc700 (LWP 67230)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f17288968bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7f16d8456740) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f172a6a3ea5 in start_thread (arg=0x7f166eafc700) at pthread_create.c:307 #8 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 9 (Thread 0x7f166e2fb700 (LWP 67231)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f17288968bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7f16d84b7210) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f172a6a3ea5 in start_thread (arg=0x7f166e2fb700) at pthread_create.c:307 #8 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 8 (Thread 0x7f166dafa700 (LWP 67232)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f17288968bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7f16d84901a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f172a6a3ea5 in start_thread (arg=0x7f166dafa700) at pthread_create.c:307 #8 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 7 (Thread 0x7f166d2f9700 (LWP 67233)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f17288968bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7f16d848ac30) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f172a6a3ea5 in start_thread (arg=0x7f166d2f9700) at pthread_create.c:307 #8 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 6 (Thread 0x7f166caf8700 (LWP 67234)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f17288968bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7f16d848cf10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f172a6a3ea5 in start_thread (arg=0x7f166caf8700) at pthread_create.c:307 #8 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 5 (Thread 0x7f166c2f7700 (LWP 67235)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f17288968bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7f16d8490170) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f172a6a3ea5 in start_thread (arg=0x7f166c2f7700) at pthread_create.c:307 #8 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 4 (Thread 0x7f166baf6700 (LWP 67236)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f17288968bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7f16d84b71e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f172a6a3ea5 in start_thread (arg=0x7f166baf6700) at pthread_create.c:307 #8 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 3 (Thread 0x7f166b2f5700 (LWP 67237)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f17288968bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7f16d848ac00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f172a6a3ea5 in start_thread (arg=0x7f166b2f5700) at pthread_create.c:307 #8 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 2 (Thread 0x7f166aaf4700 (LWP 67238)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f17288968bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7f16d8456710) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f172a6a3ea5 in start_thread (arg=0x7f166aaf4700) at pthread_create.c:307 #8 0x00007f1727fe48dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 1 (Thread 0x7f172b7d9c40 (LWP 67077)): #0 0x00007f1727fe4eb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f17299783c5 in epoll_dispatch (base=0x7f172688b300, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f172996f376 in event_base_loop (base=0x7f172688b300, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7f1726de6500, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7f1726de6500) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever (this=0x7f1726de6500) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x000000000056a48b in memcached_main(int, char**) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/memcached.cc:1033 #7 0x00007f1727f08555 in __libc_start_main (main= 0x53fcb0 , argc=3, argv=0x7fff4136e478, init=, fini=, rtld_fini=, stack_end=0x7fff4136e468) at ../csu/libc-start.c:266 #8 0x0000000000562727 in _start () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/char_traits.h:322 Detaching from program: /opt/couchbase/bin/memcached, process 67077 [Inferior 1 (process 67077) detached] 2022-02-08 14:33:37,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 14:33:37,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:33:43,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 14:33:44,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:33:45,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:33:45,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 14:33:45,831 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'sanitizers.log.*' files found 2022-02-08 14:33:45,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:33:45,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:33:50,038 | test | INFO | MainThread | [basetestcase:check_logs:858] Comparing timestamps: Log's latest timestamp: 2022-02-08 14:33:11, Test's start timestamp is 2022-02-07 21:15:50.193000 2022-02-08 14:33:50,049 | test | INFO | MainThread | [basetestcase:check_logs:858] Comparing timestamps: Log's latest timestamp: 2022-02-08 14:33:11, Test's start timestamp is 2022-02-07 21:15:50.193000 2022-02-08 14:33:50,052 | test | INFO | MainThread | [basetestcase:check_coredump_exist:967] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:33:50,055 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:969] 172.23.120.170: Found ' CRITICAL ' logs - ['memcached<0.31072.53>: 2022-02-08T14:33:09.151653-08:00 CRITICAL *** Fatal error encountered during exception handling ***\n', 'memcached<0.31072.53>: 2022-02-08T14:33:09.151735-08:00 CRITICAL Caught unhandled std::exception-derived exception. what(): EPBucket::flushVBucket_UNLOCKED: vb:778 Trying to set PCS to 788145 but the current value is 826227 and the PCS must be monotonic. The current checkpoint type is Memory. Flush\'s seqno range:[826229,826229], proposedVBState:\'{"checkpoint_type":"Memory","completed_seqno":"826227","failover_table":[{"id":55948201611177,"seq":798368},{"id":101508313219041,"seq":797119},{"id":115941502795314,"seq":795620},{"id":166355663268749,"seq":794810},{"id":164709248212853,"seq":792892},{"id":188584913467217,"seq":791465},{"id":137848230237368,"seq":789531},{"id":107619573697285,"seq":787872},{"id":74664069466173,"seq":785783},{"id":231319981373807,"seq":784934},{"id":237032548753976,"seq":782492},{"id":13847788369127,"seq":779966},{"id":88376350451870,"seq":777938},{"id":90310469247942,"seq":775590},{"id":165237524817605,"seq":774160},{"id":254185266935297,"seq":773545},{"id":139909669828063,"seq":771678},{"id":23804278280015,"seq":771036},{"id":178116978319042,"seq":769867},{"id":64396326574943,"seq":768840},{"id":31089164292718,"seq":767704},{"id":221367105370637,"seq":737978},{"id":89373404760058,"seq":736179},{"id":222907968097276,"seq":733562},{"id":242979571432083,"seq":732153}],"high_prepared_seqno":"826227","high_seqno":"826228","hlc_epoch":"1","max_cas":"1644359583888048128","max_deleted_seqno":"3","max_visible_seqno":"826228","might_contain_xattrs":false,"namespaces_supported":true,"on_disk_prepare_bytes":"0","on_disk_prepares":"0","prepared_seqno":"826227","purge_seqno":"0","snap_end":"826229","snap_start":"826229","state":"replica","version":4}\'.\n', '[ns_server:info,2022-02-08T14:33:10.414-08:00,babysitter_of_ns_1@cb.local:<0.31072.53>:ns_port_server:log:226]memcached<0.31072.53>: CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-2255). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/8cc04116-872e-46a8-158948a9-186223b3.dmp before terminating.\n', '[ns_server:info,2022-02-08T14:33:11.367-08:00,babysitter_of_ns_1@cb.local:<0.12725.55>:ns_port_server:log:226]memcached<0.12725.55>: 2022-02-08T14:33:11.166527-08:00 CRITICAL Detected previous crash\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166588-08:00 CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-2255). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/8cc04116-872e-46a8-158948a9-186223b3.dmp before terminating.\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166591-08:00 CRITICAL Stack backtrace of crashed thread:\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166593-08:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x737378]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166594-08:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x788b4a]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166596-08:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x788e88]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166597-08:00 CRITICAL #3 /lib64/libpthread.so.0() [0x7f8141ecf000+0xf630]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166599-08:00 CRITICAL #4 /lib64/libc.so.6(gsignal+0x37) [0x7f813f719000+0x36387]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166600-08:00 CRITICAL #5 /lib64/libc.so.6(abort+0x148) [0x7f813f719000+0x37a78]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166601-08:00 CRITICAL #6 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8140001000+0x9963c]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166602-08:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x74184b]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166603-08:00 CRITICAL #8 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8140001000+0xa48f6]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166604-08:00 CRITICAL #9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8140001000+0xa4961]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166606-08:00 CRITICAL #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8140001000+0xa4bf4]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166607-08:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0xcd933]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166639-08:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0x42b48f]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166663-08:00 CRITICAL #13 /opt/couchbase/bin/memcached() [0x400000+0x2de456]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166665-08:00 CRITICAL #14 /opt/couchbase/bin/memcached() [0x400000+0x2dee10]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166666-08:00 CRITICAL #15 /opt/couchbase/bin/memcached() [0x400000+0x6b48c2]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166689-08:00 CRITICAL #16 /opt/couchbase/bin/memcached() [0x400000+0x6b19c5]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166690-08:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x808e00]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166713-08:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x7f0bba]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166753-08:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x80bdb9]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166756-08:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x6b1654]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166757-08:00 CRITICAL #21 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8140001000+0xcdd40]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166788-08:00 CRITICAL #22 /lib64/libpthread.so.0() [0x7f8141ecf000+0x7ea5]\n', 'memcached<0.12725.55>: 2022-02-08T14:33:11.166790-08:00 CRITICAL #23 /lib64/libc.so.6(clone+0x6d) [0x7f813f719000+0xfe8dd]\n'] 2022-02-08 14:33:50,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 14:33:50,085 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.120.170: No 'indexer.log*' files found 2022-02-08 14:33:50,088 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 14:33:50,094 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 5 queries submitted, 0 failed, 18687 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:33:50,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:33:51,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 14:33:51,772 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:33:51,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 14:33:51,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 14:33:51,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 14:33:51,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 14:33:51,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 14:33:51,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 14:33:51,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d 2022-02-08 14:33:51,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 14:33:52,002 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.115: No crash files found 2022-02-08 14:33:52,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 14:33:52,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:33:59,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 14:34:00,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:34:01,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:34:01,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 14:34:01,763 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'sanitizers.log.*' files found 2022-02-08 14:34:01,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:34:01,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:34:01,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 14:34:01,944 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.115: No 'indexer.log*' files found 2022-02-08 14:34:01,947 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 14:34:01,956 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 14:34:02,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:34:03,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 14:34:03,602 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:34:03,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 14:34:03,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 14:34:03,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 14:34:03,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 14:34:03,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 14:34:03,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 14:34:03,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 14:34:03,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 14:34:03,815 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.116: No crash files found 2022-02-08 14:34:03,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 14:34:03,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:34:12,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 14:34:19,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:34:20,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:34:20,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 14:34:20,681 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'sanitizers.log.*' files found 2022-02-08 14:34:20,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:34:20,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:34:20,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 14:34:20,783 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.116: No 'indexer.log*' files found 2022-02-08 14:34:20,786 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 14:34:20,793 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 14:34:21,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:34:22,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 14:34:22,494 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:34:22,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 14:34:22,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 14:34:22,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 14:34:22,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 14:34:22,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 14:34:22,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 14:34:22,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d 2022-02-08 14:34:22,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls -lt /opt/couchbase/var/lib/couchbase/crash/ ############################## 172.23.121.123: 1 core dump seen 2022-02-08 14:34:22,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: rm -rf /opt/couchbase/var/lib/couchbase/crash/e5b35aef-362e-4b1c-5981dd95-b5876bd1.core running: //opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/e5b35aef-362e-4b1c-5981dd95-b5876bd1.dmp > /opt/couchbase/var/lib/couchbase/crash/e5b35aef-362e-4b1c-5981dd95-b5876bd1.core 2022-02-08 14:34:22,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: //opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/e5b35aef-362e-4b1c-5981dd95-b5876bd1.dmp > /opt/couchbase/var/lib/couchbase/crash/e5b35aef-362e-4b1c-5981dd95-b5876bd1.core running: gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/e5b35aef-362e-4b1c-5981dd95-b5876bd1.core -ex "bt full" -ex quit 2022-02-08 14:34:22,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/e5b35aef-362e-4b1c-5981dd95-b5876bd1.core -ex "bt full" -ex quit 172.23.121.123: Stack Trace of first crash - e5b35aef-362e-4b1c-5981dd95-b5876bd1.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007fe623b41387 in raise () from /lib64/libc.so.6 #0 0x00007fe623b41387 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00007fe623b42a78 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x00007fe62448c63c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95 terminating = false t = #3 0x0000000000b4184b in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88 No locals. #4 0x00007fe6244978f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48 No locals. #5 0x00007fe624497961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58 No locals. #6 0x00007fe624497c46 in __cxxabiv1::__cxa_rethrow () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:133 globals = header = #7 0x00000000004c7683 in ActiveDurabilityMonitor::State::updateHighCompletedSeqno (this=0x7fe5718d1d80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/durability/active_durability_monitor.cc:1703 e = @0x7fe5a0002e80: #8 0x00000000007ffdec in ActiveDurabilityMonitor::commit(DurabilityMonitor::ActiveSyncWrite const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/durability/active_durability_monitor.cc:1001 s = {holder_ = { >, folly::SharedMutexImpl >, folly::SharedMutexImpl, folly::LockPolicyExclusive>> = {parent_ = 0x7fe47a9a3b48}, static AllowsConcurrentAccess = false}} key = cHandle = { = {readLock = {lock_ = 0x7fe558abce98, token_ = {type_ = folly::SharedMutexToken::DEFERRED_SHARED, slot_ = 0}}, manifest = 0x7fe558abce00}, itr = { const*, std::pair*>> = {}, underlying_ = {itemPtr_ = 0x7fe52689c850, index_ = 8}}, key = {> = {}, buffer = {static npos = , buf = 0x7fe526ae7160
, len = 19}, encoding = Yes}} prepareEnd = {__d = {__r = 14656541420114874}} cookie = result = __func__ = "commit" index = commitDuration = #9 0x00000000008065e3 in ActiveDurabilityMonitor::processCompletedSyncWriteQueue() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/durability/active_durability_monitor.cc:548 sw = {storage_ = {{emptyState = 0 '\000', value = { = {item = {value = 0x7fe5586c4900}, startTime = {__d = {__r = 14656502957926074}}, status = ToCommit}, cookie = 0x0, firstChain = {ackCount = {> = {label = {static npos = 18446744073709551615, _M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x7fe5b97e82e0 "unlabelled"}, _M_string_length = 10, {_M_local_buf = "unlabelled\000&\345\177\000", _M_allocated_capacity = 7812730926796664437}}}, val = 0 '\000'}, chainPtr = 0x0}, secondChain = {ackCount = {> = {label = {static npos = 18446744073709551615, _M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x7fe5b97e8310 "unlabelled"}, _M_string_length = 10, {_M_local_buf = "unlabelled\000\024\346\177\000", _M_allocated_capacity = 7812730926796664437}}}, val = 0 '\000'}, chainPtr = 0x0}, expiryTime = { > >, true, true>> = { > >, std::_Optional_base > >, true, true> >> = {}, _M_payload = { > > >> = {_M_payload = {_M_empty = {}, _M_value = {__d = {__r = 0}}}, _M_engaged = false}, }}, > > > >> = {}, }}}, hasValue = true}} lock = #10 0x0000000000728c04 in VBucket::processResolvedSyncWrites() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:696 rlh = {lock_ = 0x7fe4c47295e8, token_ = {type_ = folly::SharedMutexToken::INLINE_SHARED, slot_ = 115}} #11 0x0000000000849ad8 in VBNotifiableTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vb_notifiable_task.cc:53 vb = {> = {> = {}, _M_ptr = 0x7fe4c4729400, _M_refcount = {_M_pi = 0x7fe57170fa00}}, } runtime = startTime = {__d = {__r = 14656541420108366}} pendingVb = {vbid = 778} #12 0x0000000000ab48c2 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:68 guard = {previous = 0x0} #13 0x0000000000ab19c5 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7fe5b97e8840) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:189 scheduleOverhead = start = {__d = {__r = 14656541420106620}} runAgain = false proxy = @0x7fe6229beeb0: #14 0x0000000000c08e00 in operator() (this=0x7fe5b97e8840) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 fn = @0x7fe5b97e8840: {> = {}, data_ = {big = 0x7fe6229beeb0, tiny = {__data = "\260\356\233\"\346\177\000\000\000\000\000\000\000\000\000\000p+Pd\345\177\000\000\006$\000\000\000\000\000\000\220\000\000\000\000\000\000\000\020\211~\271\345\177\000", __align = {}}}, call_ = 0xab2080 ::callSmall(folly::detail::function::Data&)>, exec_ = 0xab07d0 (folly::detail::function::Op, folly::detail::function::Data*, folly::detail::function::Data)>} #15 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7fe622551c00, thread=..., task=task@entry=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 rctx = {prev_ = {> = {> = {}, _M_ptr = 0x0, _M_refcount = {_M_pi = 0x0}}, }} startTime = {__d = {__r = 14656541420102461}} stats = {expired = false, waitTime = {__r = 32979}, runTime = {__r = 0}, enqueueTime = {__d = {__r = 14656541420069482}}, requestId = 0} #16 0x0000000000bf0bba in folly::CPUThreadPoolExecutor::threadRun (this=0x7fe622551c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 task = {storage_ = {{emptyState = -80 '\260', value = { = {func_ = {> = {}, data_ = {big = 0x7fe6229beeb0, tiny = {__data = "\260\356\233\"\346\177\000\000\000\000\000\000\000\000\000\000p+Pd\345\177\000\000\006$\000\000\000\000\000\000\220\000\000\000\000\000\000\000\020\211~\271\345\177\000", __align = {}}}, call_ = 0xab2080 ::callSmall(folly::detail::function::Data&)>, exec_ = 0xab07d0 (folly::detail::function::Op, folly::detail::function::Data*, folly::detail::function::Data)>}, enqueueTime_ = {__d = {__r = 14656541420069482}}, expiration_ = {__r = 0}, expireCallback_ = {> = {}, data_ = {big = 0x2406, tiny = {__data = "\006$\000\000\000\000\000\000\033\023\244", '\000' , "_\236o&\346\177\000\000\240\211~\271\345\177\000\000@6\220&\346\177\000", __align = {}}}, call_ = 0x46703d ::uninitCall(folly::detail::function::Data&)>, exec_ = 0x0}, context_ = {> = {> = {}, _M_ptr = 0x0, _M_refcount = {_M_pi = 0x0}}, }}, poison = false, priority_ = 0 '\000', queueObserverPayload_ = 140626415404160}}, hasValue = true}} guard = {list_ = {forbid = true, prev = 0x0, curr = {name = {static npos = , b_ = 0xcdb04b "CPUThreadPoolExecutor", e_ = 0xcdb060 ""}}}} #17 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 No locals. #18 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 No locals. #19 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 No locals. #20 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 No locals. #21 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 fn = #22 0x0000000000ab1654 in operator() (this=0x7fe6229e3fc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 fn = @0x7fe6229e3fc0: #23 operator() (__closure=0x7fe6229e3fc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 threadNameOpt = {storage_ = {{emptyState = -96 '\240', value = {static npos = 18446744073709551615, _M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x7fe5b97e89a0 "NonIoPool0"}, _M_string_length = 10, {_M_local_buf = "NonIoPool0\000\000\000\000\000", _M_allocated_capacity = 8029725099528449870}}}, hasValue = true}} func = #24 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 fn = @0x7fe6229e3fc0: #25 0x00007fe6244c0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 No locals. #26 0x00007fe6262c8ea5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #27 0x00007fe623c098dd in clone () from /lib64/libc.so.6 No symbol table info available. ############################## running: gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit 2022-02-08 14:34:32,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/bin/../lib/liblz4.so.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/liblz4.so.1.9.2-7.1.0-2255.x86_64.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/liblz4.so.1 Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libsnappy.so.1.1.8-7.1.0-2255.x86_64.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1 Reading symbols from /opt/couchbase/bin/../lib/libjemalloc.so.2...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libjemalloc.so.2-7.1.0-2255.x86_64.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libjemalloc.so.2 Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_core-2.1.so.7.0.0-7.1.0-2255.x86_64.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_extra-2.1.so.7.0.0-7.1.0-2255.x86_64.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_pthreads-2.1.so.7.0.0-7.1.0-2255.x86_64.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_openssl-2.1.so.7.0.0-7.1.0-2255.x86_64.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libssl.so.1.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libssl.so.1.1-7.1.0-2255.x86_64.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libssl.so.1.1 Reading symbols from /opt/couchbase/bin/../lib/libcrypto.so.1.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libcrypto.so.1.1-7.1.0-2255.x86_64.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libcrypto.so.1.1 Reading symbols from /opt/couchbase/bin/../lib/libstdc++.so.6...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libstdc++.so.6.0.28-7.1.0-2255.x86_64.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libstdc++.so.6 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /opt/couchbase/bin/../lib/libgcc_s.so.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libgcc_s.so.1-7.1.0-2255.x86_64.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libgcc_s.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 0x00007fd913073eb3 in epoll_wait () from /lib64/libc.so.6 To enable execution of this file add add-auto-load-safe-path /opt/couchbase/lib/libstdc++.so.6.0.28-gdb.py line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" Thread 85 (Thread 0x7fd9125ff700 (LWP 13817)): #0 0x00007fd915736de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd914c405a7 in background_thread_sleep (tsdn=, interval=10000000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=0, info=, tsdn=) at src/background_thread.c:307 #3 background_thread0_work (tsd=0x7fd9125ee220) at src/background_thread.c:452 #4 background_work (ind=, tsd=0x7fd9125ee220) at src/background_thread.c:490 #5 background_thread_entry (ind_arg=) at src/background_thread.c:522 #6 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 84 (Thread 0x7fd9114fe700 (LWP 13818)): #0 0x00007fd915736de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000b6466a in __gthread_cond_timedwait ( __abs_timeout=0x7fd9114ec800, __mutex=, __cond=0x7fd911dc2138) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=0x7fd911dc2138) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > ( __atime=..., __lock=..., this=0x7fd911dc2138) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_until >, spdlog::details::mpmc_blocking_queue::dequeue_for(T&, std::chrono::milliseconds) [with T = spdlog::details::async_msg]:: > (__p=..., __atime=..., __lock=..., this=0x7fd911dc2138) at /usr/local/include/c++/7.3.0/condition_variable:129 #5 wait_for, spdlog::details::mpmc_blocking_queue::dequeue_for(T&, std::chrono::milliseconds) [with T = spdlog::details::async_msg]:: > (__p=..., __rtime=, __lock=..., this=0x7fd911dc2138) at /usr/local/include/c++/7.3.0/condition_variable:145 #6 dequeue_for (wait_duration=..., popped_item=..., this=0x7fd911dc2110) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/mpmc_blocking_q.h:58 #7 spdlog::details::thread_pool::process_next_msg_ ( this=this@entry=0x7fd911dc2110) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/thread_pool-inl.h:99 #8 0x0000000000b649b8 in worker_loop_ (this=0x7fd911dc2110) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/thread_pool-inl.h:90 #9 operator() (__closure=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/thread_pool-inl.h:28 #10 __invoke_impl):: > (__f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:60 #11 __invoke):: > (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #12 _M_invoke<0> (this=) at /usr/local/include/c++/7.3.0/thread:234 #13 operator() (this=) at /usr/local/include/c++/7.3.0/thread:243 #14 std::thread::_State_impl):: > > >::_M_run(void) (this=) at /usr/local/include/c++/7.3.0/thread:186 #15 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #16 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 83 (Thread 0x7fd910cfd700 (LWP 13819)): #0 0x00007fd915736de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000b6442a in __gthread_cond_timedwait ( __abs_timeout=0x7fd910ceb9b0, __mutex=, __cond=0x7fd911db8268) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=0x7fd911db8268) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > ( __atime=..., __lock=..., this=0x7fd911db8268) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_until >, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __atime=..., __lock=..., this=0x7fd911db8268) at /usr/local/include/c++/7.3.0/condition_variable:129 #5 wait_for, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __rtime=..., __lock=..., this=0x7fd911db8268) at /usr/local/include/c++/7.3.0/condition_variable:145 #6 operator() (__closure=0x7fd911dc3a48) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/periodic_worker-inl.h:25 #7 __invoke_impl&, std::chrono::seconds):: > ( __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:60 #8 __invoke&, std::chrono::seconds):: > (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #9 _M_invoke<0> (this=0x7fd911dc3a48) at /usr/local/include/c++/7.3.0/thread:234 #10 operator() (this=0x7fd911dc3a48) at /usr/local/include/c++/7.3.0/thread:243 #11 std::thread::_State_impl&, std::chrono::seconds):: > > >::_M_run(void) (this=0x7fd911dc3a40) at /usr/local/include/c++/7.3.0/thread:186 #12 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #13 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 82 (Thread 0x7fd9104fc700 (LWP 13820)): #0 0x00007fd913068c3d in poll () from /lib64/libc.so.6 #1 0x00000000005ac9c1 in get_command (buffsize=80, buffer=0x7fd9104ea990 "@\326\326\025\331\177") at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:57 #2 check_stdin_thread (arg=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:98 #3 0x0000000000b9d269 in run (this=0x7fd911db6890) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #4 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #5 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 81 (Thread 0x7fd90e4f8700 (LWP 13824)): #0 0x00007fd915736de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000005f4116 in __gthread_cond_timedwait ( __abs_timeout=0x7fd90e4e6980, __mutex=0x7fd911de6350, __cond=0x7fd911de6378) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=, __lock=, __atime=, this=0x7fd911de6378) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=, this=0x7fd911de6378) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=, __lock=, this=0x7fd911de6378) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 ExternalAuthManagerThread::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/external_auth_manager_thread.cc:91 #6 0x0000000000bb38a3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #7 0x0000000000b9d269 in run (this=0x7fd911db6c50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 80 (Thread 0x7fd90dcf7700 (LWP 13825)): #0 0x00007fd915736de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000067c9b7 in __gthread_cond_timedwait ( __abs_timeout=0x7fd90dce59b0, __mutex=, __cond=0x7fd91191d3f8) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fd91191d3f8) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fd91191d3f8) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7fd91191d3f8) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 AuditImpl::consume_events() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/auditd/src/audit.cc:411 #6 0x0000000000b9d269 in run (this=0x7fd911db6bf0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #7 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #8 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 79 (Thread 0x7fd90ecf9700 (LWP 13827)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000ae385b in consume_socket (thread_index=, sp=0x159ad60, ctx=0x159c5b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18846 #2 worker_thread_run (conn=0x159a470) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18987 #3 worker_thread (thread_func_param=0x159a470) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19145 #4 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 78 (Thread 0x7fd90f4fa700 (LWP 13828)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000ae385b in consume_socket (thread_index=, sp=0x159b748, ctx=0x159c5b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18846 #2 worker_thread_run (conn=0x159ae58) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18987 #3 worker_thread (thread_func_param=0x159ae58) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19145 #4 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 77 (Thread 0x7fd90fcfb700 (LWP 13829)): #0 0x00007fd913068c3d in poll () from /lib64/libc.so.6 #1 0x0000000000ada7c8 in master_thread_run (ctx=0x159c5b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19291 #2 master_thread (thread_func_param=0x159c5b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19386 #3 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 76 (Thread 0x7fd90d4f6700 (LWP 13830)): #0 0x00007fd913073eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd914a073c5 in epoll_dispatch (base=0x7fd91188b900, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fd9149fe376 in event_base_loop (base=0x7fd91188b900, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7fd91192c010, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7fd91192c010) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever ( this=this@entry=0x7fd91192c010) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005b1be9 in worker_libevent (arg=0x7fd91192c000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b9d269 in run (this=0x7fd911db6ef0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 75 (Thread 0x7fd90ccf5700 (LWP 13831)): #0 0x00007fd913073eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd914a073c5 in epoll_dispatch (base=0x7fd91188bc00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fd9149fe376 in event_base_loop (base=0x7fd91188bc00, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7fd91192cbe0, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7fd91192cbe0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever ( this=this@entry=0x7fd91192cbe0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005b1be9 in worker_libevent (arg=0x7fd91192cbd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b9d269 in run (this=0x7fd911db7010) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 74 (Thread 0x7fd907fff700 (LWP 13832)): #0 0x00007fd913073eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd914a073c5 in epoll_dispatch (base=0x7fd91188bf00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fd9149fe376 in event_base_loop (base=0x7fd91188bf00, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7fd91192d7b0, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7fd91192d7b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever ( this=this@entry=0x7fd91192d7b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005b1be9 in worker_libevent (arg=0x7fd91192d7a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b9d269 in run (this=0x7fd911db7130) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 73 (Thread 0x7fd9077fe700 (LWP 13833)): #0 0x00007fd913073eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd914a073c5 in epoll_dispatch (base=0x7fd91188c200, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fd9149fe376 in event_base_loop (base=0x7fd91188c200, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7fd91192e380, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7fd91192e380) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever ( this=this@entry=0x7fd91192e380) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005b1be9 in worker_libevent (arg=0x7fd91192e370) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b9d269 in run (this=0x7fd911db7220) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 72 (Thread 0x7fd906ffd700 (LWP 13834)): #0 0x00007fd913073eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd914a073c5 in epoll_dispatch (base=0x7fd91188c500, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fd9149fe376 in event_base_loop (base=0x7fd91188c500, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7fd91192ef50, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7fd91192ef50) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever ( this=this@entry=0x7fd91192ef50) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005b1be9 in worker_libevent (arg=0x7fd91192ef40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b9d269 in run (this=0x7fd911db71f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 71 (Thread 0x7fd9067fc700 (LWP 13835)): #0 0x00007fd913073eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd914a073c5 in epoll_dispatch (base=0x7fd91188c800, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fd9149fe376 in event_base_loop (base=0x7fd91188c800, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7fd91192fb20, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7fd91192fb20) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever ( this=this@entry=0x7fd91192fb20) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005b1be9 in worker_libevent (arg=0x7fd91192fb10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b9d269 in run (this=0x7fd911db7190) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 70 (Thread 0x7fd905ffb700 (LWP 13836)): #0 0x00007fd913073eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd914a073c5 in epoll_dispatch (base=0x7fd91188cb00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fd9149fe376 in event_base_loop (base=0x7fd91188cb00, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7fd9119306f0, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7fd9119306f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever ( this=this@entry=0x7fd9119306f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005b1be9 in worker_libevent (arg=0x7fd9119306e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b9d269 in run (this=0x7fd911db71c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 69 (Thread 0x7fd8fffff700 (LWP 13837)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=4294891032, absSteadyTime=absSteadyTime@entry=0x7fd8fffed5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd80d8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd80d8, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8fffed618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd80d8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd80d8, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd80d8, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a1ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a1ac0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x15a1a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd91192a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7fd9118f2b00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7fd9118f2b00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 68 (Thread 0x7fd8ff7fe700 (LWP 13838)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=4286498328, absSteadyTime=absSteadyTime@entry=0x7fd8ff7ec5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd80f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd80f0, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8ff7ec618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd80f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd80f0, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd80f0, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a1ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a1ac0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x15a1a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd91192a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7fd9118f2c00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7fd9118f2c00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 67 (Thread 0x7fd8feffd700 (LWP 13839)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=4278105624, absSteadyTime=absSteadyTime@entry=0x7fd8fefeb5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8348) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8348, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8fefeb618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8348) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8348, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf8947 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8348, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a1ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a1ac0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x15a1a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd91192a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000ab1654 in operator() (this=0x7fd9118f2c40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 operator() (__closure=0x7fd9118f2c40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 66 (Thread 0x7fd8fcfd7700 (LWP 13840)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=4244395544, absSteadyTime=absSteadyTime@entry=0x7fd8fcfc55f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8600) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8600, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8fcfc5618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8600) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8600, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf8947 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8600, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a1ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a1ac0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x15a1a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd91192a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000ab1654 in operator() (this=0x7fd9118f2cc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 operator() (__closure=0x7fd9118f2cc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 65 (Thread 0x7fd8effff700 (LWP 13841)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=4026455576, absSteadyTime=absSteadyTime@entry=0x7fd8effed5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8288) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8288, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8effed618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8288) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8288, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8288, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a1ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a1ac0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x15a1a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd91192a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7fd9118f2d00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7fd9118f2d00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 64 (Thread 0x7fd8ef7fe700 (LWP 13842)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=4018062872, absSteadyTime=absSteadyTime@entry=0x7fd8ef7ec5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8228) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8228, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8ef7ec618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8228) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8228, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8228, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a1ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a1ac0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x15a1a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd91192a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7fd9118f2c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7fd9118f2c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 63 (Thread 0x7fd8eeffd700 (LWP 13843)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=4009670168, absSteadyTime=absSteadyTime@entry=0x7fd8eefeb5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8690) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8690, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8eefeb618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8690) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8690, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf8947 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8690, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a1ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a1ac0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x15a1a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd91192a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000ab1654 in operator() (this=0x7fd9118f2d40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 operator() (__closure=0x7fd9118f2d40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 62 (Thread 0x7fd8ee7fc700 (LWP 13844)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=4001277464, absSteadyTime=absSteadyTime@entry=0x7fd8ee7ea5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd82b8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd82b8, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8ee7ea618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd82b8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd82b8, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf8947 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd82b8, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a1ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a1ac0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x15a1a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd91192a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000ab1654 in operator() (this=0x7fd9118f2dc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 operator() (__closure=0x7fd9118f2dc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 61 (Thread 0x7fd8edffb700 (LWP 13845)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3992884760, absSteadyTime=absSteadyTime@entry=0x7fd8edfe95f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8030) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8030, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8edfe9618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8030) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8030, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8030, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a31c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a31c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x15a3180, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd91192b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7fd9118f2e80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7fd9118f2e80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 60 (Thread 0x7fd8ed7fa700 (LWP 13846)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3984492056, absSteadyTime=absSteadyTime@entry=0x7fd8ed7e85f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd83f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd83f0, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8ed7e8618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd83f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd83f0, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd83f0, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a31c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a31c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x15a3180, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd91192b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7fd9118f2ec0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7fd9118f2ec0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 59 (Thread 0x7fd8ecff9700 (LWP 13847)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3976099352, absSteadyTime=absSteadyTime@entry=0x7fd8ecfe75f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8300) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8300, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8ecfe7618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8300) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8300, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8300, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a31c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a31c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x15a3180, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd91192b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7fd911dd7ac0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7fd911dd7ac0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 58 (Thread 0x7fd8cffff700 (LWP 13848)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3489584664, absSteadyTime=absSteadyTime@entry=0x7fd8cffed5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8108) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8108, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8cffed618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8108) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8108, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8108, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a31c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a31c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x15a3180, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd91192b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7fd911dd7b40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7fd911dd7b40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 57 (Thread 0x7fd8cf7fe700 (LWP 13849)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3481191960, absSteadyTime=absSteadyTime@entry=0x7fd8cf7ec5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8438) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8438, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8cf7ec618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8438) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8438, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8438, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a31c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a31c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x15a3180, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd91192b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7fd911dd7b80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7fd911dd7b80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 56 (Thread 0x7fd8ceffd700 (LWP 13850)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3472799256, absSteadyTime=absSteadyTime@entry=0x7fd8cefeb5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8150) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8150, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8cefeb618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8150) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8150, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8150, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a31c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a31c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x15a3180, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd91192b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7fd911dd7b00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7fd911dd7b00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 55 (Thread 0x7fd8ce7fc700 (LWP 13851)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3464406552, absSteadyTime=absSteadyTime@entry=0x7fd8ce7ea5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8168) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8168, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8ce7ea618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8168) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8168, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8168, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a31c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a31c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x15a3180, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd91192b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7fd911ddbbc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7fd911ddbbc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 54 (Thread 0x7fd8cdffb700 (LWP 13852)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3456013848, absSteadyTime=absSteadyTime@entry=0x7fd8cdfe95f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8138) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8138, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8cdfe9618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8138) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8138, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8138, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a31c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a31c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x15a3180, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd91192b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7fd911ddbc00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7fd911ddbc00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 53 (Thread 0x7fd8cd7fa700 (LWP 13853)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3447621144, absSteadyTime=absSteadyTime@entry=0x7fd8cd7e85f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd83a8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd83a8, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8cd7e8618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd83a8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd83a8, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf8947 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd83a8, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a48c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a48c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x15a4880, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd911951000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000ab1654 in operator() (this=0x7fd911ddbd00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 operator() (__closure=0x7fd911ddbd00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 52 (Thread 0x7fd8ccff9700 (LWP 13854)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3439228440, absSteadyTime=absSteadyTime@entry=0x7fd8ccfe75f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd83d8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd83d8, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8ccfe7618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd83d8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd83d8, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd83d8, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a48c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a48c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x15a4880, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd911951000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7fd911ddbd40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7fd911ddbd40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 51 (Thread 0x7fd8affff700 (LWP 13855)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=2952713752, absSteadyTime=absSteadyTime@entry=0x7fd8affed5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8558) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8558, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8affed618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8558) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8558, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8558, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a48c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a48c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x15a4880, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd911951000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7fd911ddbd80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7fd911ddbd80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 50 (Thread 0x7fd8af7fe700 (LWP 13856)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=2944321048, absSteadyTime=absSteadyTime@entry=0x7fd8af7ec5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd80c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd80c0, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8af7ec618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd80c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd80c0, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf8947 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd80c0, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a48c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a48c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x15a4880, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd911951000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000ab1654 in operator() (this=0x7fd911ddbe00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 operator() (__closure=0x7fd911ddbe00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 49 (Thread 0x7fd8ae7ff700 (LWP 13857)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=2927547928, absSteadyTime=absSteadyTime@entry=0x7fd8ae7ed5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8090) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8090, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8ae7ed618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8090) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8090, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8090, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a48c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a48c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x15a4880, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd911951000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7fd911ddbe40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7fd911ddbe40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 48 (Thread 0x7fd8adffe700 (LWP 13858)): #0 0x00007fd9130ca56b in __memcpy_ssse3_back () from /lib64/libc.so.6 #1 0x00000000007bfe39 in Blob (len=867, start=0x7fd7864f4040 "{\"name\":\"sJohn\",\"age\":45,\"animals\":[\"Cultured\",\"Cyan\"],\"attributes\":{\"hair\":\"Barbie Pink\",\"dimensions\":{\"height\":21,\"weight\":35},\"hobbies\":[{\"type\":\"Home\",\"name\":\"Sports\",\"details\":{\"location\":{\"lat\":"..., this=0x7fd699fd8e00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/blob.cc:39 #2 Blob::New ( start=start@entry=0x7fd7864f4040 "{\"name\":\"sJohn\",\"age\":45,\"animals\":[\"Cultured\",\"Cyan\"],\"attributes\":{\"hair\":\"Barbie Pink\",\"dimensions\":{\"height\":21,\"weight\":35},\"hobbies\":[{\"type\":\"Home\",\"name\":\"Sports\",\"details\":{\"location\":{\"lat\":"..., len=len@entry=867) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/blob.cc:20 #3 0x00000000006eb41f in setData (nb=867, dta=0x7fd7864f4040 "{\"name\":\"sJohn\",\"age\":45,\"animals\":[\"Cultured\",\"Cyan\"],\"attributes\":{\"hair\":\"Barbie Pink\",\"dimensions\":{\"height\":21,\"weight\":35},\"hobbies\":[{\"type\":\"Home\",\"name\":\"Sports\",\"details\":{\"location\":{\"lat\":"..., this=0x7fd783bec780) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/item.h:608 #4 Item::Item(DocKey const&, unsigned int, long, void const*, unsigned long, unsigned char, unsigned long, long, Vbid, unsigned long, unsigned char) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/item.cc:80 #5 0x0000000000878f4f in make_unique > () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/kv_magma_common/magma-kvstore_metadata.h:102 #6 MagmaKVStore::makeItem(Vbid, magma::Slice const&, magma::Slice const&, magma::Slice const&, ValueFilter) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1232 #7 0x000000000087a5d0 in MagmaKVStore::scan(BySeqnoScanContext&) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1822 #8 0x00000000008d3d9d in DCPBackfillBySeqnoDisk::scan() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_by_seqno_disk.cc:173 #9 0x00000000008d7792 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:131 #10 0x00000000008d91b1 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:344 #11 0x00000000008d9954 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:78 #12 0x0000000000ab48c2 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:68 #13 0x0000000000ab19c5 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7fd8adfec840) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:189 #14 0x0000000000c08e00 in operator() (this=0x7fd8adfec840) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #15 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) ( this=this@entry=0x7fd911951000, thread=..., task=task@entry=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #16 0x0000000000bf0bba in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd911951000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #17 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #18 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #19 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #20 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #21 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #22 0x0000000000ab1654 in operator() (this=0x7fd911ddbdc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #23 operator() (__closure=0x7fd911ddbdc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #24 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #25 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #26 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #27 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 47 (Thread 0x7fd8ad7fd700 (LWP 13859)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=2910762520, absSteadyTime=absSteadyTime@entry=0x7fd8ad7eb5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8060) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8060, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8ad7eb618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8060) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8060, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf880d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8060, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a48c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a48c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x15a4880, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd911951000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000ab1654 in operator() (this=0x7fd911ddbe80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #18 operator() (__closure=0x7fd911ddbe80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 46 (Thread 0x7fd8acffc700 (LWP 13860)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=2902369816, absSteadyTime=absSteadyTime@entry=0x7fd8acfea5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8618) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8618, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd8acfea618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8618) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8618, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf8947 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8618, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a48c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a48c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x15a4880, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd911951000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000ab1654 in operator() (this=0x7fd911de3ec0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 operator() (__closure=0x7fd911de3ec0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 45 (Thread 0x7fd88ffff700 (LWP 13861)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=2415842840, absSteadyTime=absSteadyTime@entry=0x7fd88ffed5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8258) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8258, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd88ffed618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8258) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8258, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf8947 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8258, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a5fc0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a5fc0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x15a5f80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd911951c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000ab1654 in operator() (this=0x7fd911de3fc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 operator() (__closure=0x7fd911de3fc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 44 (Thread 0x7fd88f7fe700 (LWP 13862)): #0 0x00007fd91306dbf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000be88dd in nativeFutexWaitImpl ( waitMask=waitMask@entry=2407450136, absSteadyTime=absSteadyTime@entry=0x7fd88f7ec5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fd8fcfd8330) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fd8fcfd8330, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fd88f7ec618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000bf29aa in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fd8fcfd8330) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fd8fcfd8330, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000bf8947 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fd8fcfd8330, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000bf8e76 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x15a5fc0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000bfc5d0 in try_wait_for > ( timeout=, this=0x15a5fc0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x15a5f80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000bf0b47 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fd911951c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000ab1654 in operator() (this=0x7fd911ded000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #19 operator() (__closure=0x7fd911ded000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7fd88effd700 (LWP 13863)): #0 0x00007fd913073eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd914a073c5 in epoll_dispatch (base=0x7fd91188ce00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fd9149fe376 in event_base_loop (base=0x7fd91188ce00, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7fd911de6f00, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7fd911de6f00) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever (this=0x7fd911de6f00) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x0000000000c00629 in folly::IOThreadPoolExecutor::threadRun ( this=0x7fd911de6780, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/IOThreadPoolExecutor.cpp:195 #7 0x0000000000c0bdb9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #8 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #9 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #10 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #11 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #12 0x00007fd91392ad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #13 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7fd88e7fc700 (LWP 13864)): #0 0x00007fd915736de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd914c3fe51 in background_thread_sleep (tsdn=, interval=1925000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=, info=, tsdn=) at src/background_thread.c:307 #3 background_work (ind=, tsd=0x7fd88e7eb220) at src/background_thread.c:497 #4 background_thread_entry (ind_arg=) at src/background_thread.c:522 #5 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7fd88dffb700 (LWP 13865)): #0 0x00007fd915736de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000009bf8ff in __gthread_cond_timedwait ( __abs_timeout=0x7fd88dfe9960, __mutex=, __cond=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:103 #6 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #7 0x0000000000b9d269 in run (this=0x7fd8fc51e8c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7fd88d7fa700 (LWP 13866)): #0 0x00007fd915736de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000009bf8ff in __gthread_cond_timedwait ( __abs_timeout=0x7fd88d7e8960, __mutex=, __cond=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:103 #6 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #7 0x0000000000b9d269 in run (this=0x7fd8fc51e8f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7fd88cff9700 (LWP 13867)): #0 0x00007fd915736de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000009bf8ff in __gthread_cond_timedwait ( __abs_timeout=0x7fd88cfe7960, __mutex=, __cond=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:103 #6 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #7 0x0000000000b9d269 in run (this=0x7fd8fc51e920) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7fd85ffff700 (LWP 13868)): #0 0x00007fd915736de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000009bf8ff in __gthread_cond_timedwait ( __abs_timeout=0x7fd85ffed960, __mutex=, __cond=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:103 #6 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #7 0x0000000000b9d269 in run (this=0x7fd8fc51e950) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7fd85f7fe700 (LWP 13869)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd8fc51e980) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7fd85effd700 (LWP 13870)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd8fc51e9b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7fd85e7fc700 (LWP 13871)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd8fc51e9e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7fd85dffb700 (LWP 13872)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd8fc51ea10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fd85d7fa700 (LWP 13884)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd8cc45d170) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fd85aeff700 (LWP 13885)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd8fc4d6a90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fd85a6fe700 (LWP 13886)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd8fc5081c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fd859efd700 (LWP 13887)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd8fc58d650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fd8596fc700 (LWP 13888)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd8cc4f5f80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fd858efb700 (LWP 13889)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd8cc4f5fb0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fd853fff700 (LWP 13890)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd8cc4f5fe0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fd8537fe700 (LWP 13891)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd8cc4f6010) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fd852ffd700 (LWP 13892)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd8cc4f6040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fd8527fc700 (LWP 13893)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd8cc4f6070) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fd851ffb700 (LWP 13894)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd8cc52e120) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fd8517fa700 (LWP 13895)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd8cc52e150) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fd848a91700 (LWP 13911)): #0 0x00007fd915736de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000009bf8ff in __gthread_cond_timedwait ( __abs_timeout=0x7fd848a7f960, __mutex=, __cond=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:103 #6 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #7 0x0000000000b9d269 in run (this=0x7fd84a21aa40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fd844e9b700 (LWP 13912)): #0 0x00007fd915736de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000009bf8ff in __gthread_cond_timedwait ( __abs_timeout=0x7fd844e89960, __mutex=, __cond=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:103 #6 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #7 0x0000000000b9d269 in run (this=0x7fd84a21b970) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fd8157ff700 (LWP 13913)): #0 0x00007fd915736de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000009bf8ff in __gthread_cond_timedwait ( __abs_timeout=0x7fd8157ed960, __mutex=, __cond=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:103 #6 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #7 0x0000000000b9d269 in run (this=0x7fd8581772e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fd814ffe700 (LWP 13914)): #0 0x00007fd915736de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000009bf8ff in __gthread_cond_timedwait ( __abs_timeout=0x7fd814fec960, __mutex=, __cond=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7fd8fc48cd18) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:103 #6 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #7 0x0000000000b9d269 in run (this=0x7fd84ba6b1a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fd80ffff700 (LWP 13915)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd84ba864a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fd80f7fe700 (LWP 13916)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd84ba5dcb0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fd80effd700 (LWP 13917)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd84ba8e8f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fd80e7fc700 (LWP 13918)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd84ba85240) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fd80dffb700 (LWP 13919)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd84ba23f10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fd80d7fa700 (LWP 13920)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd84ba468b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fd80cff9700 (LWP 13921)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd84ba6af00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fd80c7f8700 (LWP 13922)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd84ba8e8c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fd80bff7700 (LWP 13923)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd84ba23ee0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fd80b7f6700 (LWP 13924)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd84ba85210) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fd80aff5700 (LWP 13925)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd84ba87610) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fd80a7f4700 (LWP 13926)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd84a3174a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fd809ff3700 (LWP 13927)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd84a318790) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fd8097f2700 (LWP 13928)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd84a32ea80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fd808ff1700 (LWP 13929)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd84a32fd70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fd8087f0700 (LWP 13930)): #0 0x00007fd915736a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9139258bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009bf668 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009bfa51 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000b9d269 in run (this=0x7fd84a33f060) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fd915732ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd9130738dd in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fd91686bc40 (LWP 13816)): #0 0x00007fd913073eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd914a073c5 in epoll_dispatch (base=0x7fd91188b300, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fd9149fe376 in event_base_loop (base=0x7fd91188b300, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c124e7 in folly::EventBase::loopBody ( this=this@entry=0x7fd911de6500, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c129b6 in folly::EventBase::loop ( this=this@entry=0x7fd911de6500) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c14376 in folly::EventBase::loopForever (this=0x7fd911de6500) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x000000000056a48b in memcached_main(int, char**) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/memcached.cc:1033 #7 0x00007fd912f97555 in __libc_start_main () from /lib64/libc.so.6 #8 0x0000000000562727 in _start () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/char_traits.h:322 Detaching from program: /opt/couchbase/bin/memcached, process 13816 [Inferior 1 (process 13816) detached] 2022-02-08 14:34:37,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 14:34:37,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:34:46,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 14:34:47,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:34:48,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:34:49,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 14:34:49,147 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'sanitizers.log.*' files found 2022-02-08 14:34:49,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:34:49,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:34:49,269 | test | INFO | MainThread | [basetestcase:check_logs:858] Comparing timestamps: Log's latest timestamp: 2022-02-08 14:33:07, Test's start timestamp is 2022-02-07 21:15:50.193000 2022-02-08 14:34:49,272 | test | INFO | MainThread | [basetestcase:check_logs:858] Comparing timestamps: Log's latest timestamp: 2022-02-08 14:33:07, Test's start timestamp is 2022-02-07 21:15:50.193000 2022-02-08 14:34:49,273 | test | INFO | MainThread | [basetestcase:check_coredump_exist:967] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:34:49,273 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:969] 172.23.121.123: Found ' CRITICAL ' logs - ['memcached<0.11793.59>: 2022-02-08T14:33:04.538265-08:00 CRITICAL *** Fatal error encountered during exception handling ***\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538297-08:00 CRITICAL Caught unhandled std::exception-derived exception. what(): Monotonic (ActiveDM(vb:778)::State::highCompletedSeqno) invariant failed: new value (788145) breaks invariant on current value (826227)\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538304-08:00 CRITICAL Exception thrown from:\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538358-08:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x13bec8]\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538382-08:00 CRITICAL #1 /opt/couchbase/bin/memcached() [0x400000+0x3ff5e1]\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538398-08:00 CRITICAL #2 /opt/couchbase/bin/memcached() [0x400000+0x3ff9e0]\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538412-08:00 CRITICAL #3 /opt/couchbase/bin/memcached() [0x400000+0x3ffdec]\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538424-08:00 CRITICAL #4 /opt/couchbase/bin/memcached() [0x400000+0x4065e3]\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538444-08:00 CRITICAL #5 /opt/couchbase/bin/memcached() [0x400000+0x328c04]\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538581-08:00 CRITICAL #6 /opt/couchbase/bin/memcached() [0x400000+0x449ad8]\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538606-08:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x6b48c2]\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538627-08:00 CRITICAL #8 /opt/couchbase/bin/memcached() [0x400000+0x6b19c5]\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538657-08:00 CRITICAL #9 /opt/couchbase/bin/memcached() [0x400000+0x808e00]\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538682-08:00 CRITICAL #10 /opt/couchbase/bin/memcached() [0x400000+0x7f0bba]\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538784-08:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0x80bdb9]\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538805-08:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0x6b1654]\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538879-08:00 CRITICAL #13 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe6243f3000+0xcdd40]\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538922-08:00 CRITICAL #14 /lib64/libpthread.so.0() [0x7fe6262c1000+0x7ea5]\n', 'memcached<0.11793.59>: 2022-02-08T14:33:04.538963-08:00 CRITICAL #15 /lib64/libc.so.6(clone+0x6d) [0x7fe623b0b000+0xfe8dd]\n', '[ns_server:info,2022-02-08T14:33:06.384-08:00,babysitter_of_ns_1@cb.local:<0.11793.59>:ns_port_server:log:226]memcached<0.11793.59>: CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-2255). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/e5b35aef-362e-4b1c-5981dd95-b5876bd1.dmp before terminating.\n', '[ns_server:info,2022-02-08T14:33:07.505-08:00,babysitter_of_ns_1@cb.local:<0.25782.60>:ns_port_server:log:226]memcached<0.25782.60>: 2022-02-08T14:33:07.304035-08:00 CRITICAL Detected previous crash\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304103-08:00 CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-2255). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/e5b35aef-362e-4b1c-5981dd95-b5876bd1.dmp before terminating.\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304115-08:00 CRITICAL Stack backtrace of crashed thread:\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304118-08:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x737378]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304120-08:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x788b4a]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304122-08:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x788e88]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304124-08:00 CRITICAL #3 /lib64/libpthread.so.0() [0x7fe6262c1000+0xf630]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304135-08:00 CRITICAL #4 /lib64/libc.so.6(gsignal+0x37) [0x7fe623b0b000+0x36387]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304137-08:00 CRITICAL #5 /lib64/libc.so.6(abort+0x148) [0x7fe623b0b000+0x37a78]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304139-08:00 CRITICAL #6 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe6243f3000+0x9963c]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304144-08:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x74184b]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304145-08:00 CRITICAL #8 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe6243f3000+0xa48f6]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304154-08:00 CRITICAL #9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe6243f3000+0xa4961]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304156-08:00 CRITICAL #10 /opt/couchbase/bin/../lib/libstdc++.so.6(__cxa_rethrow+0x46) [0x7fe6243f3000+0xa4c46]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304190-08:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0xc7683]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304192-08:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0x3ffdec]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304194-08:00 CRITICAL #13 /opt/couchbase/bin/memcached() [0x400000+0x4065e3]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304196-08:00 CRITICAL #14 /opt/couchbase/bin/memcached() [0x400000+0x328c04]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304197-08:00 CRITICAL #15 /opt/couchbase/bin/memcached() [0x400000+0x449ad8]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304199-08:00 CRITICAL #16 /opt/couchbase/bin/memcached() [0x400000+0x6b48c2]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304200-08:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x6b19c5]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304229-08:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x808e00]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304231-08:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x7f0bba]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304233-08:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x80bdb9]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304235-08:00 CRITICAL #21 /opt/couchbase/bin/memcached() [0x400000+0x6b1654]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304236-08:00 CRITICAL #22 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe6243f3000+0xcdd40]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304238-08:00 CRITICAL #23 /lib64/libpthread.so.0() [0x7fe6262c1000+0x7ea5]\n', 'memcached<0.25782.60>: 2022-02-08T14:33:07.304243-08:00 CRITICAL #24 /lib64/libc.so.6(clone+0x6d) [0x7fe623b0b000+0xfe8dd]\n'] 2022-02-08 14:34:49,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 14:34:49,299 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.123: No 'indexer.log*' files found 2022-02-08 14:34:49,302 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 14:34:49,309 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 5 queries submitted, 0 failed, 18709 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:34:50,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:34:51,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 14:34:51,490 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:34:51,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 14:34:51,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 14:34:51,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 14:34:51,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 14:34:51,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 14:34:51,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 14:34:51,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d 2022-02-08 14:34:51,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 14:34:51,848 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.124: No crash files found 2022-02-08 14:34:51,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 14:34:51,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:35:00,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 14:35:02,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:35:03,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:35:04,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 14:35:04,312 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'sanitizers.log.*' files found 2022-02-08 14:35:04,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:35:04,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:35:04,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 14:35:04,716 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.124: No 'indexer.log*' files found 2022-02-08 14:35:04,720 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 14:35:04,729 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 14:35:06,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:35:07,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 14:35:07,345 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:35:07,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 14:35:07,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 14:35:07,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 14:35:07,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 14:35:07,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 14:35:07,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 14:35:07,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d 2022-02-08 14:35:07,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 14:35:07,642 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.126: No crash files found 2022-02-08 14:35:07,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 14:35:07,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:35:16,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 14:35:17,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:35:18,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:35:18,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 14:35:19,095 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'sanitizers.log.*' files found 2022-02-08 14:35:19,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:35:19,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:35:19,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 14:35:19,470 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.126: No 'indexer.log*' files found 2022-02-08 14:35:19,473 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 14:35:19,490 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 14:35:20,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:35:21,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 14:35:22,161 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:35:22,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 14:35:22,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 14:35:22,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 14:35:22,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 14:35:22,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 14:35:22,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 14:35:22,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 2022-02-08 14:35:22,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 14:35:22,790 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.127: No crash files found 2022-02-08 14:35:22,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 14:35:22,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:35:30,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 14:35:32,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:35:32,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:35:33,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 14:35:33,832 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'sanitizers.log.*' files found 2022-02-08 14:35:33,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:35:33,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:35:34,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 14:35:34,131 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.127: No 'indexer.log*' files found 2022-02-08 14:35:34,137 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 14:35:34,154 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 14:35:35,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:35:35,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 14:35:35,882 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:35:35,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 14:35:35,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 14:35:35,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 14:35:35,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 14:35:36,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 14:35:36,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 14:35:36,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 14:35:36,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 14:35:36,249 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.128: No crash files found 2022-02-08 14:35:36,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 14:35:36,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:35:41,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 14:35:42,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:35:43,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:35:43,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 14:35:43,700 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'sanitizers.log.*' files found 2022-02-08 14:35:43,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:35:43,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:35:43,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 14:35:43,871 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.128: No 'indexer.log*' files found 2022-02-08 14:35:43,875 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 14:35:43,881 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 14:35:45,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:35:46,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 14:35:46,559 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:35:46,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 14:35:46,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 14:35:46,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 14:35:46,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 14:35:46,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 14:35:46,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 14:35:47,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d 2022-02-08 14:35:47,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 14:35:47,128 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.129: No crash files found 2022-02-08 14:35:47,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 14:35:47,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 5 queries submitted, 0 failed, 18733 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:35:56,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 14:35:57,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:35:58,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:35:58,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 14:35:58,812 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'sanitizers.log.*' files found 2022-02-08 14:35:58,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:35:58,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:35:59,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 14:35:59,118 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.129: No 'indexer.log*' files found 2022-02-08 14:35:59,121 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 14:35:59,148 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 14:35:59,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:35:59,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 14:35:59,730 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:35:59,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 14:35:59,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 14:35:59,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 14:35:59,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 14:35:59,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 14:35:59,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 14:35:59,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 14:35:59,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 14:35:59,993 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.130: No crash files found 2022-02-08 14:36:00,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 14:36:00,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:36:00,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 14:36:00,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:36:00,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:36:00,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 14:36:00,236 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.130: No 'sanitizers.log.*' files found 2022-02-08 14:36:00,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:36:00,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:36:00,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 14:36:00,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:00,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 14:36:00,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:00,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:00,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:00,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:00,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:00,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:00,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:00,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:00,749 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 14:36:00,766 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 14:36:00,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:36:01,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 14:36:01,219 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:36:01,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 14:36:01,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 14:36:01,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 14:36:01,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 14:36:01,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 14:36:01,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 14:36:01,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 14:36:01,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 14:36:01,403 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.131: No crash files found 2022-02-08 14:36:01,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 14:36:01,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:36:01,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 14:36:01,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:36:01,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:36:01,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 14:36:01,595 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.131: No 'sanitizers.log.*' files found 2022-02-08 14:36:01,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:36:01,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:36:01,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 14:36:01,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:01,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 14:36:01,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:01,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:01,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:01,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:01,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:01,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:01,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:01,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:01,984 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 14:36:01,990 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 14:36:02,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:36:02,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 14:36:02,418 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:36:02,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 14:36:02,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 14:36:02,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 14:36:02,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 14:36:02,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 14:36:02,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 14:36:02,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 14:36:02,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 14:36:02,602 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.132: No crash files found 2022-02-08 14:36:02,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 14:36:02,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:36:02,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 14:36:02,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:36:02,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:36:02,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 14:36:02,793 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.132: No 'sanitizers.log.*' files found 2022-02-08 14:36:02,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:36:02,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:36:02,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 14:36:02,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:02,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 14:36:02,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:02,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:02,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:03,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:03,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:03,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:03,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:03,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:03,187 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 14:36:03,197 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 14:36:03,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:36:03,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 14:36:03,627 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:36:03,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 14:36:03,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 14:36:03,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 14:36:03,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 14:36:03,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 14:36:03,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 14:36:03,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 14:36:03,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 14:36:03,809 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.133: No crash files found 2022-02-08 14:36:03,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 14:36:03,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:36:03,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 14:36:03,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:36:03,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:36:04,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 14:36:04,029 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.133: No 'sanitizers.log.*' files found 2022-02-08 14:36:04,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:36:04,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:36:04,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 14:36:04,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:04,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 14:36:04,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:04,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:04,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:04,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:04,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:04,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:04,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:04,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:04,430 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 14:36:04,450 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 14:36:04,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:36:04,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 14:36:04,891 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:36:04,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 14:36:04,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 14:36:04,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 14:36:04,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 14:36:04,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 14:36:05,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 14:36:05,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 14:36:05,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 14:36:05,085 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.134: No crash files found 2022-02-08 14:36:05,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 14:36:05,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:36:05,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 14:36:05,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:36:05,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:36:05,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 14:36:05,315 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.134: No 'sanitizers.log.*' files found 2022-02-08 14:36:05,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:36:05,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:36:05,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 14:36:05,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'panic' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:05,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'fatal' /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -v 'fatal remote' 2022-02-08 14:36:05,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Error parsing XATTR' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:05,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'protobuf.Error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:05,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Encounter planner error' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:05,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'corruption' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:05,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'processFlushAbort' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:05,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'found missing page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:05,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'invalid last page' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:05,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: grep -r 'Storage corrupted and unrecoverable' /opt/couchbase/var/lib/couchbase/logs/indexer.log 2022-02-08 14:36:05,732 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 14:36:05,742 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 14:36:06,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:36:07,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 14:36:07,846 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:36:07,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 14:36:07,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 14:36:07,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 14:36:07,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 14:36:07,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 14:36:07,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 14:36:07,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 14:36:08,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 14:36:08,055 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.135: No crash files found 2022-02-08 14:36:08,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 14:36:08,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:36:15,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 14:36:23,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:36:24,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:36:24,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 14:36:24,900 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'sanitizers.log.*' files found 2022-02-08 14:36:24,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:36:24,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:36:25,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 14:36:25,066 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.135: No 'indexer.log*' files found 2022-02-08 14:36:25,072 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 14:36:25,082 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.136 with username: root, password: couchbase 2022-02-08 14:36:25,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:36:26,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /etc/ 2022-02-08 14:36:26,328 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:36:26,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /cygdrive/c/Windows 2022-02-08 14:36:26,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: uname -m 2022-02-08 14:36:26,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/cpuinfo 2022-02-08 14:36:26,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: df -Thl 2022-02-08 14:36:26,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: cat /proc/meminfo 2022-02-08 14:36:26,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname 2022-02-08 14:36:26,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: hostname -d 2022-02-08 14:36:26,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2022-02-08 14:36:26,539 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:899] 172.23.121.136: No crash files found 2022-02-08 14:36:26,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2022-02-08 14:36:26,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:36:27,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2022-02-08 14:36:27,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:36:28,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2022-02-08 14:36:28,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2022-02-08 14:36:28,134 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'sanitizers.log.*' files found 2022-02-08 14:36:28,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2022-02-08 14:36:28,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2022-02-08 14:36:28,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.136: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2022-02-08 14:36:28,226 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:918] 172.23.121.136: No 'indexer.log*' files found 2022-02-08 14:36:28,227 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.136 2022-02-08 14:36:28,433 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1158] Running cbcollect on node ns_1@172.23.120.170,ns_1@172.23.121.115,ns_1@172.23.121.116,ns_1@172.23.121.123,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.127,ns_1@172.23.121.128,ns_1@172.23.121.129,ns_1@172.23.121.130,ns_1@172.23.121.131,ns_1@172.23.121.132,ns_1@172.23.121.133,ns_1@172.23.121.134,ns_1@172.23.121.135 2022-02-08 14:36:28,520 | test | DEBUG | MainThread | [rest_client:_http_request:814] {'date': 'Tue, 08 Feb 2022 22:36:27 GMT', 'content-length': '2', 'server': 'Couchbase Server', 'expires': 'Thu, 01 Jan 1970 00:00:00 GMT', 'x-frame-options': 'DENY', 'x-permitted-cross-domain-policies': 'none', 'pragma': 'no-cache', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'connection': 'close', 'content-type': 'application/json', 'cache-control': 'no-cache,no-store,must-revalidate', 'status': '200'} 2022-02-08 14:36:28,532 | test | DEBUG | MainThread | [rest_client:_http_request:815] [] 2022-02-08 14:36:28,561 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> trigger_cb_collect_on_cluster():L1160 2022-02-08 14:36:28,562 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: Wait for CB collect to start 2022-02-08 14:36:38,563 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1161] ns_1@172.23.120.170,ns_1@172.23.121.115,ns_1@172.23.121.116,ns_1@172.23.121.123,ns_1@172.23.121.124,ns_1@172.23.121.126,ns_1@172.23.121.127,ns_1@172.23.121.128,ns_1@172.23.121.129,ns_1@172.23.121.130,ns_1@172.23.121.131,ns_1@172.23.121.132,ns_1@172.23.121.133,ns_1@172.23.121.134,ns_1@172.23.121.135 - cbcollect status: True 2022-02-08 14:36:38,566 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1166] Polling active_tasks to check cbcollect status 2022-02-08 14:36:38,644 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 0 - running 2022-02-08 14:36:38,664 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:36:38,665 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 5 queries submitted, 0 failed, 18754 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:36:53,831 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 1 - running 2022-02-08 14:36:53,869 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:36:53,871 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:37:03,967 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 2 - running 2022-02-08 14:37:04,006 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:37:04,009 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:37:14,042 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 3 - running 2022-02-08 14:37:14,058 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:37:14,059 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:37:24,102 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 4 - running 2022-02-08 14:37:24,132 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:37:24,134 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:37:34,234 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 5 - running 2022-02-08 14:37:34,260 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:37:34,262 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:37:44,302 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 6 - running 2022-02-08 14:37:44,325 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:37:44,326 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 5 queries submitted, 0 failed, 18778 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:37:54,371 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 7 - running 2022-02-08 14:37:54,415 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:37:54,417 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:38:04,482 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 8 - running 2022-02-08 14:38:04,513 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:38:04,516 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:38:14,538 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 9 - running 2022-02-08 14:38:14,601 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:38:14,602 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:38:24,634 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 10 - running 2022-02-08 14:38:24,668 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:38:24,670 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:38:34,717 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 11 - running 2022-02-08 14:38:34,763 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:38:34,766 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:38:44,809 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 12 - running 2022-02-08 14:38:44,851 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:38:44,852 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 5 queries submitted, 0 failed, 18804 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:38:54,920 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 13 - running 2022-02-08 14:38:54,973 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:38:54,974 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:39:04,996 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 14 - running 2022-02-08 14:39:05,023 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:39:05,025 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:39:15,046 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 15 - running 2022-02-08 14:39:15,068 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:39:15,069 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:39:25,135 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 16 - running 2022-02-08 14:39:25,161 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:39:25,164 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:39:35,213 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 17 - running 2022-02-08 14:39:35,257 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:39:35,259 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:39:45,305 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 18 - running 2022-02-08 14:39:45,329 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:39:45,331 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 5 queries submitted, 0 failed, 18828 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:39:55,369 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 19 - running 2022-02-08 14:39:55,394 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:39:55,395 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:40:05,444 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 20 - running 2022-02-08 14:40:05,476 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:40:05,476 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:40:15,516 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 21 - running 2022-02-08 14:40:15,579 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:40:15,585 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:40:25,631 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 22 - running 2022-02-08 14:40:25,667 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:40:25,667 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:40:35,723 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 23 - running 2022-02-08 14:40:35,743 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:40:35,746 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:40:45,785 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 24 - running 2022-02-08 14:40:45,808 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:40:45,809 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 5 queries submitted, 0 failed, 18852 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:40:55,848 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 25 - running 2022-02-08 14:40:55,940 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:40:55,944 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:41:05,983 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 26 - running 2022-02-08 14:41:06,026 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:41:06,029 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:41:16,138 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 27 - running 2022-02-08 14:41:16,184 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:41:16,187 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:41:26,229 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 28 - running 2022-02-08 14:41:26,273 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:41:26,275 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:41:36,319 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 29 - running 2022-02-08 14:41:36,339 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:41:36,342 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:41:46,482 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 30 - running 2022-02-08 14:41:46,509 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:41:46,510 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 5 queries submitted, 0 failed, 18875 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:41:56,551 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 31 - running 2022-02-08 14:41:56,599 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:41:56,601 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:42:06,651 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 32 - running 2022-02-08 14:42:06,709 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:42:06,710 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:42:16,769 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 33 - running 2022-02-08 14:42:16,809 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:42:16,809 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:42:26,848 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 34 - running 2022-02-08 14:42:26,875 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:42:26,877 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:42:36,927 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 35 - running 2022-02-08 14:42:36,961 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:42:36,963 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:42:47,005 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 36 - running 2022-02-08 14:42:47,071 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:42:47,072 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 5 queries submitted, 0 failed, 18900 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:42:57,111 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 37 - running 2022-02-08 14:42:57,128 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:42:57,128 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:43:07,174 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 38 - running 2022-02-08 14:43:07,223 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:43:07,226 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:43:17,296 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 39 - running 2022-02-08 14:43:17,375 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:43:17,378 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:43:27,418 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 40 - running 2022-02-08 14:43:27,447 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:43:27,448 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:43:37,509 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 41 - running 2022-02-08 14:43:37,592 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:43:37,596 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:43:47,651 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 42 - running 2022-02-08 14:43:47,673 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:43:47,674 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 5 queries submitted, 0 failed, 18923 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:43:57,697 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 43 - running 2022-02-08 14:43:57,720 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:43:57,723 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:44:07,796 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 44 - running 2022-02-08 14:44:07,839 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:44:07,842 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:44:17,890 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 45 - running 2022-02-08 14:44:17,917 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:44:17,917 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:44:27,970 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 46 - running 2022-02-08 14:44:28,043 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:44:28,045 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:44:38,088 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 47 - running 2022-02-08 14:44:38,104 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:44:38,105 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:44:48,144 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 48 - running 2022-02-08 14:44:48,186 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:44:48,188 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 5 queries submitted, 0 failed, 18947 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:44:58,227 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 49 - running 2022-02-08 14:44:58,240 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:44:58,240 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:45:08,289 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 50 - running 2022-02-08 14:45:08,329 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:45:08,332 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:45:18,364 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 51 - running 2022-02-08 14:45:18,407 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:45:18,408 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:45:28,436 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 52 - running 2022-02-08 14:45:28,461 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:45:28,463 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:45:38,500 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 53 - running 2022-02-08 14:45:38,523 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:45:38,525 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:45:48,575 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 54 - running 2022-02-08 14:45:48,621 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:45:48,625 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 5 queries submitted, 0 failed, 18972 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:45:58,683 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 55 - running 2022-02-08 14:45:58,716 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:45:58,717 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:46:08,759 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 56 - running 2022-02-08 14:46:08,785 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:46:08,786 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:46:18,859 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 57 - running 2022-02-08 14:46:18,888 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:46:18,888 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:46:28,930 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 58 - running 2022-02-08 14:46:28,960 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:46:28,963 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:46:40,095 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1173] CBCollectInfo Iteration 59 - running 2022-02-08 14:46:40,161 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1183 2022-02-08 14:46:40,163 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2022-02-08 14:46:50,197 | test | DEBUG | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1189] {u'perNode': {u'ns_1@172.23.121.130': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.130.zip', u'status': u'collected'}, u'ns_1@172.23.121.131': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.131.zip', u'status': u'collected'}, u'ns_1@172.23.120.170': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.120.170.zip', u'status': u'started'}, u'ns_1@172.23.121.129': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.129.zip', u'status': u'started'}, u'ns_1@172.23.121.116': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.116.zip', u'status': u'collected'}, u'ns_1@172.23.121.127': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.127.zip', u'status': u'started'}, u'ns_1@172.23.121.128': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.128.zip', u'status': u'started'}, u'ns_1@172.23.121.115': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.115.zip', u'status': u'started'}, u'ns_1@172.23.121.126': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.126.zip', u'status': u'started'}, u'ns_1@172.23.121.123': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.123.zip', u'status': u'started'}, u'ns_1@172.23.121.134': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.134.zip', u'status': u'collected'}, u'ns_1@172.23.121.124': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.124.zip', u'status': u'started'}, u'ns_1@172.23.121.135': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.135.zip', u'status': u'collected'}, u'ns_1@172.23.121.132': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.132.zip', u'status': u'collected'}, u'ns_1@172.23.121.133': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.133.zip', u'status': u'collected'}}, u'type': u'clusterLogsCollection', u'node': u'ns_1@172.23.120.170', u'recommendedRefreshPeriod': 2, u'cancelURI': u'/controller/cancelLogsCollection', u'progress': 46, u'ts': u'2022-02-08 22:36:28', u'status': u'running'} 2022-02-08 14:46:50,201 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1193] ns_1@172.23.120.170: Copying cbcollect ZIP file to Client 2022-02-08 14:46:50,210 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 5 queries submitted, 0 failed, 18999 passed, 0 rejected, 0 cancelled, 0 timeout 5 queries submitted, 0 failed, 19022 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:48:18,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:48:40,187 | infra | ERROR | MainThread | [remote_util:execute_command_raw_jsch:3256] JSch exception on 172.23.120.170: com.jcraft.jsch.JSchException: channel is not opened. 2022-02-08 14:48:40,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /etc/ 5 queries submitted, 0 failed, 19047 passed, 0 rejected, 0 cancelled, 0 timeout 2022-02-08 14:49:03,740 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:49:03,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2022-02-08 14:49:03,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: uname -m 2022-02-08 14:49:03,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/cpuinfo 2022-02-08 14:49:03,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: df -Thl 2022-02-08 14:49:04,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: cat /proc/meminfo 2022-02-08 14:49:04,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname 2022-02-08 14:49:04,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.120.170: hostname -d cp: cannot stat ‘/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.120.170.zip’: No such file or directory 2022-02-08 14:49:04,875 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1223] ns_1@172.23.120.170 - Failed to copy cb collect zip file 2022-02-08 14:49:04,881 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 2022-02-08 14:49:04,884 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1193] ns_1@172.23.121.115: Copying cbcollect ZIP file to Client 2022-02-08 14:49:04,927 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.115 with username: root, password: couchbase 2022-02-08 14:49:08,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:49:10,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /etc/ 2022-02-08 14:49:12,036 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:49:12,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: ls /cygdrive/c/Windows 2022-02-08 14:49:12,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: uname -m 2022-02-08 14:49:12,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/cpuinfo 2022-02-08 14:49:12,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: df -Thl 2022-02-08 14:49:12,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: cat /proc/meminfo 2022-02-08 14:49:12,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname 2022-02-08 14:49:12,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.115: hostname -d cp: cannot stat ‘/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.115.zip’: No such file or directory 2022-02-08 14:49:13,219 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1223] ns_1@172.23.121.115 - Failed to copy cb collect zip file 2022-02-08 14:49:13,220 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.115 2022-02-08 14:49:13,221 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1193] ns_1@172.23.121.116: Copying cbcollect ZIP file to Client 2022-02-08 14:49:13,227 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.116 with username: root, password: couchbase 2022-02-08 14:49:14,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:49:14,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /etc/ 2022-02-08 14:49:15,331 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:49:15,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: ls /cygdrive/c/Windows 2022-02-08 14:49:15,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: uname -m 2022-02-08 14:49:15,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/cpuinfo 2022-02-08 14:49:15,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: df -Thl 2022-02-08 14:49:15,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: cat /proc/meminfo 2022-02-08 14:49:15,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname 2022-02-08 14:49:15,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: hostname -d 2022-02-08 14:49:20,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.116: rm -f /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.116.zip 2022-02-08 14:49:20,509 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.116 2022-02-08 14:49:20,512 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1193] ns_1@172.23.121.123: Copying cbcollect ZIP file to Client 2022-02-08 14:49:20,519 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.123 with username: root, password: couchbase 2022-02-08 14:49:29,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:49:31,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /etc/ 2022-02-08 14:49:33,667 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:49:33,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: ls /cygdrive/c/Windows 2022-02-08 14:49:33,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: uname -m 2022-02-08 14:49:33,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/cpuinfo 2022-02-08 14:49:34,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: df -Thl 2022-02-08 14:49:34,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: cat /proc/meminfo 2022-02-08 14:49:34,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname 2022-02-08 14:49:34,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.123: hostname -d cp: cannot stat ‘/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.123.zip’: No such file or directory 2022-02-08 14:49:34,904 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1223] ns_1@172.23.121.123 - Failed to copy cb collect zip file 2022-02-08 14:49:34,908 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.123 2022-02-08 14:49:34,930 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1193] ns_1@172.23.121.124: Copying cbcollect ZIP file to Client 2022-02-08 14:49:34,938 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.124 with username: root, password: couchbase 2022-02-08 14:49:37,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:49:38,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /etc/ 2022-02-08 14:49:39,719 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:49:39,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: ls /cygdrive/c/Windows 2022-02-08 14:49:39,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: uname -m 2022-02-08 14:49:39,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/cpuinfo 2022-02-08 14:49:40,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: df -Thl 2022-02-08 14:49:40,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: cat /proc/meminfo 2022-02-08 14:49:40,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname 2022-02-08 14:49:40,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.124: hostname -d cp: cannot stat ‘/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.124.zip’: No such file or directory 2022-02-08 14:49:40,983 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1223] ns_1@172.23.121.124 - Failed to copy cb collect zip file 2022-02-08 14:49:40,987 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.124 2022-02-08 14:49:40,993 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1193] ns_1@172.23.121.126: Copying cbcollect ZIP file to Client 2022-02-08 14:49:41,010 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.126 with username: root, password: couchbase 2022-02-08 14:49:43,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:49:45,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /etc/ 2022-02-08 14:49:47,378 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:49:47,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: ls /cygdrive/c/Windows 2022-02-08 14:49:47,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: uname -m 2022-02-08 14:49:47,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/cpuinfo 2022-02-08 14:49:47,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: df -Thl 2022-02-08 14:49:47,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: cat /proc/meminfo 2022-02-08 14:49:47,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname 2022-02-08 14:49:48,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.126: hostname -d cp: cannot stat ‘/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.126.zip’: No such file or directory 2022-02-08 14:49:48,210 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1223] ns_1@172.23.121.126 - Failed to copy cb collect zip file 2022-02-08 14:49:48,213 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.126 2022-02-08 14:49:48,213 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1193] ns_1@172.23.121.127: Copying cbcollect ZIP file to Client 2022-02-08 14:49:48,219 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.127 with username: root, password: couchbase 2022-02-08 14:49:50,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:49:52,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /etc/ 2022-02-08 14:49:53,145 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:49:53,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: ls /cygdrive/c/Windows 2022-02-08 14:49:53,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: uname -m 2022-02-08 14:49:53,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/cpuinfo 2022-02-08 14:49:53,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: df -Thl 2022-02-08 14:49:53,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: cat /proc/meminfo 2022-02-08 14:49:53,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname 2022-02-08 14:49:53,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.127: hostname -d 5 queries submitted, 0 failed, 19073 passed, 0 rejected, 0 cancelled, 0 timeout cp: cannot stat ‘/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.127.zip’: No such file or directory 2022-02-08 14:49:53,724 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1223] ns_1@172.23.121.127 - Failed to copy cb collect zip file 2022-02-08 14:49:53,726 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.127 2022-02-08 14:49:53,726 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1193] ns_1@172.23.121.128: Copying cbcollect ZIP file to Client 2022-02-08 14:49:53,734 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.128 with username: root, password: couchbase 2022-02-08 14:49:54,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:49:54,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /etc/ 2022-02-08 14:49:54,933 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:49:54,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: ls /cygdrive/c/Windows 2022-02-08 14:49:54,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: uname -m 2022-02-08 14:49:54,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/cpuinfo 2022-02-08 14:49:55,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: df -Thl 2022-02-08 14:49:55,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: cat /proc/meminfo 2022-02-08 14:49:55,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname 2022-02-08 14:49:55,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: hostname -d 2022-02-08 14:49:58,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.128: rm -f /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.128.zip 2022-02-08 14:49:58,515 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.128 2022-02-08 14:49:58,517 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1193] ns_1@172.23.121.129: Copying cbcollect ZIP file to Client 2022-02-08 14:49:58,529 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.129 with username: root, password: couchbase 2022-02-08 14:50:02,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:50:05,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /etc/ 2022-02-08 14:50:06,720 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:50:06,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: ls /cygdrive/c/Windows 2022-02-08 14:50:06,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: uname -m 2022-02-08 14:50:06,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/cpuinfo 2022-02-08 14:50:07,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: df -Thl 2022-02-08 14:50:07,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: cat /proc/meminfo 2022-02-08 14:50:07,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname 2022-02-08 14:50:07,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.129: hostname -d cp: cannot stat ‘/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.129.zip’: No such file or directory 2022-02-08 14:50:07,473 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1223] ns_1@172.23.121.129 - Failed to copy cb collect zip file 2022-02-08 14:50:07,476 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.129 2022-02-08 14:50:07,479 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1193] ns_1@172.23.121.130: Copying cbcollect ZIP file to Client 2022-02-08 14:50:07,489 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.130 with username: root, password: couchbase 2022-02-08 14:50:07,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:50:07,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /etc/ 2022-02-08 14:50:08,039 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:50:08,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: ls /cygdrive/c/Windows 2022-02-08 14:50:08,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: uname -m 2022-02-08 14:50:08,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/cpuinfo 2022-02-08 14:50:08,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: df -Thl 2022-02-08 14:50:08,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: cat /proc/meminfo 2022-02-08 14:50:08,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname 2022-02-08 14:50:08,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: hostname -d 2022-02-08 14:50:10,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.130: rm -f /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.130.zip 2022-02-08 14:50:10,591 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.130 2022-02-08 14:50:10,592 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1193] ns_1@172.23.121.131: Copying cbcollect ZIP file to Client 2022-02-08 14:50:10,599 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.131 with username: root, password: couchbase 2022-02-08 14:50:10,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:50:10,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /etc/ 2022-02-08 14:50:11,036 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:50:11,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: ls /cygdrive/c/Windows 2022-02-08 14:50:11,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: uname -m 2022-02-08 14:50:11,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/cpuinfo 2022-02-08 14:50:11,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: df -Thl 2022-02-08 14:50:11,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: cat /proc/meminfo 2022-02-08 14:50:11,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname 2022-02-08 14:50:11,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: hostname -d 2022-02-08 14:50:13,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.131: rm -f /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.131.zip 2022-02-08 14:50:13,288 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.131 2022-02-08 14:50:13,290 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1193] ns_1@172.23.121.132: Copying cbcollect ZIP file to Client 2022-02-08 14:50:13,301 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.132 with username: root, password: couchbase 2022-02-08 14:50:13,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:50:13,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /etc/ 2022-02-08 14:50:13,744 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:50:13,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: ls /cygdrive/c/Windows 2022-02-08 14:50:13,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: uname -m 2022-02-08 14:50:13,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/cpuinfo 2022-02-08 14:50:13,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: df -Thl 2022-02-08 14:50:13,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: cat /proc/meminfo 2022-02-08 14:50:13,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname 2022-02-08 14:50:13,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: hostname -d 2022-02-08 14:50:16,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.132: rm -f /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.132.zip 2022-02-08 14:50:16,141 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.132 2022-02-08 14:50:16,144 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1193] ns_1@172.23.121.133: Copying cbcollect ZIP file to Client 2022-02-08 14:50:16,151 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.133 with username: root, password: couchbase 2022-02-08 14:50:16,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:50:16,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /etc/ 2022-02-08 14:50:16,628 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:50:16,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: ls /cygdrive/c/Windows 2022-02-08 14:50:16,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: uname -m 2022-02-08 14:50:16,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/cpuinfo 2022-02-08 14:50:16,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: df -Thl 2022-02-08 14:50:16,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: cat /proc/meminfo 2022-02-08 14:50:16,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname 2022-02-08 14:50:16,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: hostname -d 2022-02-08 14:50:19,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.133: rm -f /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.133.zip 2022-02-08 14:50:19,242 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.133 2022-02-08 14:50:19,243 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1193] ns_1@172.23.121.134: Copying cbcollect ZIP file to Client 2022-02-08 14:50:19,249 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.134 with username: root, password: couchbase 2022-02-08 14:50:19,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:50:19,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /etc/ 2022-02-08 14:50:19,694 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:50:19,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: ls /cygdrive/c/Windows 2022-02-08 14:50:19,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: uname -m 2022-02-08 14:50:19,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/cpuinfo 2022-02-08 14:50:19,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: df -Thl 2022-02-08 14:50:19,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: cat /proc/meminfo 2022-02-08 14:50:19,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname 2022-02-08 14:50:19,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: hostname -d 2022-02-08 14:50:22,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.134: rm -f /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.134.zip 2022-02-08 14:50:22,118 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.134 2022-02-08 14:50:22,125 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1193] ns_1@172.23.121.135: Copying cbcollect ZIP file to Client 2022-02-08 14:50:22,134 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.121.135 with username: root, password: couchbase 2022-02-08 14:50:22,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: sw_vers | grep ProductVersion | awk '{ print $2 }' 2022-02-08 14:50:23,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /etc/ 2022-02-08 14:50:24,003 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3402] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-08 14:50:24,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: ls /cygdrive/c/Windows 2022-02-08 14:50:24,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: uname -m 2022-02-08 14:50:24,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/cpuinfo 2022-02-08 14:50:24,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: df -Thl 2022-02-08 14:50:24,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: cat /proc/meminfo 2022-02-08 14:50:24,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname 2022-02-08 14:50:24,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: hostname -d 2022-02-08 14:50:29,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3219] Running command on 172.23.121.135: rm -f /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2022-02-08T223628-ns_1@172.23.121.135.zip 2022-02-08 14:50:29,079 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.135 2022-02-08 14:50:29,085 | test | CRITICAL | MainThread | [basetestcase:tearDownEverything:624] Skipping get_trace !! 2022-02-08 14:50:29,092 | test | WARNING | MainThread | [basetestcase:tearDownEverything:630] CLEANUP WAS SKIPPED 2022-02-08 14:50:29,095 | infra | INFO | MainThread | [basetestcase:tearDownEverything:652] ========== tasks in thread pool ========== 2022-02-08 14:50:29,098 | infra | INFO | MainThread | [basetestcase:tearDownEverything:654] ========================================== 2022-02-08 14:50:29,101 | infra | INFO | MainThread | [task_manager:shutdown:64] Running TaskManager shutdown 2022-02-08 14:50:29,104 | infra | INFO | MainThread | [task_manager:shutdown:64] Running TaskManager shutdown 2022-02-08 14:50:29,107 | infra | WARNING | MainThread | [cluster:shutdown:707] Cluster instance shutdown with force ERROR ====================================================================== ERROR: SystemTestMagma (aGoodDoctor.Hospital.Murphy) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/aGoodDoctor/Hospital.py", line 210, in tearDown BaseTestCase.tearDown(self) File "pytests/basetestcase.py", line 586, in tearDown self.assertFalse(result, msg="Cb_log file validation failed") AssertionError: Cb_log file validation failed ====================================================================== FAIL: SystemTestMagma (aGoodDoctor.Hospital.Murphy) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/aGoodDoctor/Hospital.py", line 711, in SystemTestMagma self.assertTrue(rebalance_task.result, "Rebalance Failed") AssertionError: Rebalance Failed ---------------------------------------------------------------------- Ran 1 test in 63287.988s FAILED (failures=1, errors=1) During the test, Remote Connections: 5732, Disconnections: 5732 SDK Connections: 1, Disconnections: 0 !!!!!! CRITICAL :: SDK disconnection mismatch !!!!! summary so far suite aGoodDoctor.Hospital.Murphy , pass 0 , fail 1 failures so far... aGoodDoctor.Hospital.Murphy.SystemTestMagma testrunner logs, diags and results are available under /data/workspace/magma_temp_job4/logs/testrunner-22-Feb-07_21-15-37/test_1 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.9/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 17h 35m 16s 2 actionable tasks: 2 executed Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [description-setter] Description set: 7.1.0-2255 [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 8s Notifying upstream projects of job completion Email was triggered for: Unstable (Test Failures) Sending email for trigger: Unstable (Test Failures) An attempt to send an e-mail to empty list of recipients, ignored. Finished: UNSTABLE